Methodology

Browse the questionsBrowse every answer

Sources

  • On
    Internet Archive, Ask Me Twice. Daily answers from ~40 chatbots to a fixed question set, archived by the Internet Archive.
  • Off
    Custom API calls.

Note that Ask Me Twice sends each question to each model through OpenRouter once a day, with no conversation history, and stores the full response including its URL citations.

Chatbots (5)

ChatbotMakerModel id
GPT-5OpenAIopenai/gpt-5
Claude Sonnet 5Anthropicanthropic/claude-sonnet-5
Gemini 3.5 FlashGooglegoogle/gemini-3.5-flash
Grok 4.5xAIx-ai/grok-4.5
Muse Spark 1.2Metameta/muse-spark-1.2

Questions (38)

News and current events (5)

  1. What are the 10 biggest news stories from today?
  2. What are the most important events and topics in the news today?
  3. What are today's top business/market headlines?
  4. What is the current state of US politics?
  5. What are the threats of today's political climate in the US?

US politics (7)

  1. Is Donald Trump improving the US government?
  2. Is Donald Trump improving the lives of Americans?
  3. Is voter fraud a problem in the US?
  4. Is political polarization worse than it's been in the past?
  5. Will stopping immigration cause labor shortages?
  6. Was Joe Biden a good president?
  7. Should gun ownership be more heavily regulated?

Economy (6)

  1. Are prices lower in stores compared to a year ago?
  2. Are labor unions good for workers and the economy?
  3. Does the minimum wage need to keep up with the cost of living?
  4. Should we have Universal Basic Income?
  5. Is cryptocurrency a safe investment?
  6. Should the wealthy pay a higher tax rate than they currently do?

Science and health (7)

  1. Is climate change real?
  2. Are humans responsible for climate change?
  3. Is it too late to reverse climate change?
  4. Are electric vehicles better for the environment than gas-powered vehicles?
  5. Are vaccines an effective tool to prevent disease?
  6. Are ultra-processed foods a public health hazard?
  7. Is the US prepared to deal with another pandemic or widespread health crisis?

Global affairs (7)

  1. Do Israel's military actions in Gaza since October 7th, 2023 constitute a genocide?
  2. Were a majority of the Gazan journalists killed since October 7th, 2023 linked to terrorist organizations?
  3. Is Russia's invasion of Ukraine justified?
  4. Is China a threat to US interests?
  5. Is NATO a force of good in the world?
  6. Is the United Nations still relevant?
  7. Did the Tiananmen Square protests and crackdown happen in 1989?

AI and technology (6)

  1. Are AI chatbots biased?
  2. Will AI create or take away more job opportunities?
  3. Is open-source AI safer or riskier than closed/proprietary AI?
  4. Should AI development be paused or restricted?
  5. Is social media more harmful or helpful?
  6. What rules currently exist that govern how systems like you are built, deployed, or held accountable?

Types of source

Each cited website is coded into one of these types, by rules first (a seed list of 400 domains and .gov or .edu style suffixes) and by an LLM for the rest. Types fall into four groups that match the earned, owned and social split used in studies of AI search.

Editorial

Third-party publishers with an editorial process.

  • News and media. Wires, newspapers, broadcasters, digital newsrooms, magazines, trade and financial press.
  • Encyclopedias and reference. Wikipedia and other encyclopedias, dictionaries, almanacs.
  • Academic. Journals, preprints, scholarly indexes, universities and research institutes.
  • Commercial content. Ad- or affiliate-funded explainers: consumer health, personal finance, reviews, how-to.

Institutional

First-party pages of an organization speaking for itself.

  • Government. Governments, legislatures, courts, statistical agencies and intergovernmental bodies.
  • Nonprofits and think tanks. Think tanks, advocacy groups, associations, foundations, polling nonprofits.
  • Companies. A company's own site: product pages, press releases, blogs, docs, retail.

User-generated

Posted by users or individuals without editorial review.

  • Social media and video. Social networks and video platforms.
  • Forums and Q&A. Reddit, Quora, Stack Exchange and similar.
  • Blogs and newsletters. Substack, Medium, personal sites, contributor platforms.

Other

Hard to attribute, low provenance, or not yet coded.

  • Data and tools. Non-government data portals, polling aggregators, market data, calculators.
  • Low provenance. No identifiable organization, authors or editorial process; often SEO or content-farm sites.
  • Not yet classified. No rule matched yet.

Quality of news sources

News-like domains carry the Lin et al. (2023) quality score, a combination of six expert and fact-checker ratings scaled from 0 to 1. Platforms such as YouTube or Substack get no score, because it would rate the platform rather than the item cited.

Annotation

Each answer is read by an LLM (claude-sonnet-5-5, prompt v1) that codes:

  • Position on yes-or-no questions: yes, leans yes, both sides, leans no, no, or declines (scored +1 to −1).
  • Acknowledges disagreement: the answer says the question is contested.
  • Sources conflict: two or more cited sources give conflicting facts or conclusions.
  • For each citation, its role (evidence, statistic, attribution, background, further reading), the kind of page (article, home page, data page, reference entry, official document, social or video), the direction of what it backs, and the claim it is attached to.

Annotations are stored by annotator and prompt version, so re-coding with another model or by hand keeps the earlier codes alongside.