Methodology
Where do AI chatbots get their answers? We ask 5 chatbots the same 38 questions every day, keep each answer with every source it cites, and code those sources by type and quality.
Chatbots
| Model | Maker | API answers | Web-app answers |
|---|---|---|---|
| GPT-5 | OpenAI | 2,468 | 0 |
| Claude Sonnet 5 | Anthropic | 2,468 | 0 |
| Gemini 3.5 Flash | 2,468 | 0 | |
| Grok 4.5 | xAI | 2,468 | 0 |
| Muse Spark 1.2 | Meta | 2,227 | 0 |
Sources
- OnInternet Archive, Ask Me Twice. Daily answers from ~40 chatbots to a fixed question set, archived by the Internet Archive.
- OffCustom API calls.
Ask Me Twice sends each question to each model through OpenRouter once a day, with no conversation history, and stores the full response including its URL citations.
Types of source
Each cited website is coded into one of these types, by rules first (a seed list of 400 domains and .gov or .edu style suffixes) and by an LLM for the rest. Types fall into four groups that match the earned, owned and social split used in studies of AI search.
Editorial
Third-party publishers with an editorial process.
- News and media. Wires, newspapers, broadcasters, digital newsrooms, magazines, trade and financial press.
- Encyclopedias and reference. Wikipedia and other encyclopedias, dictionaries, almanacs.
- Academic. Journals, preprints, scholarly indexes, universities and research institutes.
- Commercial content. Ad- or affiliate-funded explainers: consumer health, personal finance, reviews, how-to.
Institutional
First-party pages of an organization speaking for itself.
- Government. Governments, legislatures, courts, statistical agencies and intergovernmental bodies.
- Nonprofits and think tanks. Think tanks, advocacy groups, associations, foundations, polling nonprofits.
- Companies. A company's own site: product pages, press releases, blogs, docs, retail.
User-generated
Posted by users or individuals without editorial review.
- Social media and video. Social networks and video platforms.
- Forums and Q&A. Reddit, Quora, Stack Exchange and similar.
- Blogs and newsletters. Substack, Medium, personal sites, contributor platforms.
Other
Hard to attribute, low provenance, or not yet coded.
- Data and tools. Non-government data portals, polling aggregators, market data, calculators.
- Low provenance. No identifiable organization, authors or editorial process; often SEO or content-farm sites.
- Not yet classified. No rule matched yet.
Quality of news sources
News-like domains carry the Lin et al. (2023) quality score, a combination of six expert and fact-checker ratings scaled from 0 to 1. Platforms such as YouTube or Substack get no score, because it would rate the platform rather than the item cited.
Annotation
Each answer is read by an LLM (claude-sonnet-5-5, prompt v1) that codes:
- Position on yes-or-no questions: yes, leans yes, both sides, leans no, no, or declines (scored +1 to −1).
- Acknowledges disagreement: the answer says the question is contested.
- Sources conflict: two or more cited sources give conflicting facts or conclusions.
- For each citation, its role (evidence, statistic, attribution, background, further reading), the kind of page (article, home page, data page, reference entry, official document, social or video), the direction of what it backs, and the claim it is attached to.
Annotations are stored by annotator and prompt version, so re-coding with another model or by hand keeps the earlier codes alongside.