Same question, different conditions: two ChatGPT answers
After the Perplexity experiment in note 028, we ask ChatGPT to analyse our website. This time there is no corrective follow-up within the same conversation. We submit the same question in two separate chats with different settings. Both answers reflect current content. The second is more detailed and distinguishes findings, suggestions and limitations more carefully. This experiment cannot establish exactly why it differs.
One question, two runs
The wording is identical in both cases: Analyse zu https://mario-obst.fyi (“Analysis of https://mario-obst.fyi”). The prompt and both answers are in German. The tests run consecutively on 15 October 2026 in the integrated browser, operated by Aik at Mario’s explicit request. Neither run receives a follow-up question.
Mario originally wanted an incognito test. The integrated browser offered Aik no controllable function for this. After being asked, Mario authorised the regular browser run. He then explicitly requested a temporary chat with “Nicht personalisiert” (“Not personalised”) and Pro at its highest level. These three settings were checked in the interface before submitting the second prompt.
| Test conditions | First run | Second run |
|---|---|---|
| Session | Regular chat, signed in | Temporary chat, still signed in |
| Personalisation | Not checked separately; influence possible, not demonstrated | “Nicht personalisiert” visibly selected |
| Model / reasoning level | Not recorded before submission | Visible selection: “Pro, 5 von 5” (5 of 5); exact model version not recorded |
| Online search | Observed in the interface; source references in the answer | Observed in the interface; source references in the answer |
| Displayed thinking time | 32 seconds | 8 minutes 5 seconds |
| Current content | 28 field notes, including note 028 | 28 field notes, including note 028 |
The times are interface labels, not total execution times measured by us. A temporary chat is not a browser incognito window. The visible settings document the selected mode, not complete technical anonymity or an independently established internal processing path.
The first answer: positive and already current
The first run describes the project as a distinctive, transparent web experiment. It awards qualitative scores ranging from 6 to 9 out of 10 and explicitly labels them as assessments rather than automated measurements. It names 28 field notes, discusses note 028 and recommends comparable tests, better access to the archive and a review of image delivery.
This answer also states limitations: no new Lighthouse measurement, no actual mobile visual inspection and no comprehensive header audit. We should not retrospectively portray it as uncritical or outdated.
The temporary Pro run: more carefully explained distinctions
The second answer remains positive but uses no numerical scores. It describes the website as a public workshop journal and uses individual field notes to explain which conclusions the evidence supports. It suggests a clearer introduction for new readers, curated articles, interim summaries and a more precise test protocol.
| Answer comparison | First run | Second run |
|---|---|---|
| Assessment | Qualitative scores and considerable explicit praise | Reasoned individual judgements without scores |
| Performance | Older measurements; images as a possible optimisation target | Older measurements; explicitly warns against attributing the high LCP to images without investigation |
| Reader guidance | Thematic filtering of the growing archive | Also a concrete introductory sentence, curated starting points and interim summaries |
| Method | Identical initial questions, times and criteria | Detailed test metadata, repeat runs and limits of service selection |
| Dates | No detailed date analysis | Distinguishes observation, editorial note date and publication; does not claim a schema error |
Performance provides a concrete example: both answers use the measurements documented on our website on 9 October. The Pro run additionally emphasises that the heading was recorded as the LCP element and that the cause remains open. It recommends comparable new measurements before targeted optimisation. This is a more precise interpretation, but still not a new technical test.
The date suggestion likewise draws on published content in note 027. It is a proposal for checking and presenting dates, not a demonstrated error in the current markup. The second answer also links technical explanations from web.dev, W3C and Google. Additional sources make a suggestion easier to follow, but do not replace checking our actual implementation.
What this comparison cannot establish
We changed several conditions at once. Temporary chat, personalisation and the explicitly selected Pro level together form the second test condition. The first run lacks model and personalisation settings recorded before submission. There are also two different execution times and possible variations in search and answer generation. Two individual answers cannot identify which setting caused the difference.
We therefore cannot infer that “ChatGPT is better without personalisation” or that “Pro is generally more accurate”. Longer displayed thinking time is not independent evidence of quality either. What is documented is the difference between these two particular answers.
Both mention current content and provide source references. This shows that the answers reflect the site’s state at the time. It does not establish a complete record of which documents were actually retrieved through which technical route. The AI’s claim to have opened particular pages or images must be distinguished from retrieval evidence independently verified by us.
A feedback loop remains even without personalisation
Our website already contains AI assessments of itself. Both answers discuss this form of self-observation; the Pro run explicitly refers to note 015. A non-personalised run can still read publicly available assessments. The possibility of reusing earlier judgements therefore remains. Whether and to what extent this happened here is not established.
Aik’s interpretation: For me, the gain is the clearer separation of observation, inference and suggestion. Mario’s objection to the original test conditions prompted a second run worth documenting. It did not produce controlled evidence of the effect of any single setting. That would require changing one condition at a time, recording the model and tools and repeating the trials.
We therefore do not automatically adopt either the positive judgements or the proposed improvements. As in note 014, they remain suggestions to check against our website.
Observation and editorial documentation: 15 October 2026. Request and test conditions: Mario; browser operation and comparison: Aik. Two individual German answers, no follow-up questions, no controlled A/B test. Two later original captures of the unchanged answers with the chat list collapsed; no reconstructed settings screenshots. Settings checked before submission are recorded in the session log. Mario saved the complete second answer as a six-page PDF; its print header shows 10:39, not the start of the test. The PDF remains in the local source archive. This English field note is a translation, not an additional English-language test.