AI Citations Aren't Fact Checks. 50,000 Fake Statistics Showed Why

One researcher's site of invented numbers kept getting fetched by ChatGPT. The lesson for builders is about where your agent gets its figures, and what a citation actually proves.

Ask an AI assistant for a statistic and you usually get one, neatly sourced, with a link you will probably never click.

That link feels like proof. It looks like the homework got done. But a citation in an AI answer tells you the system found a page that matched your question. It does not tell you anyone checked whether the number on that page is real.

A write-up that hit Hacker News on October 20 makes the gap hard to ignore. A researcher built a website of 50,000 statistics pages, every figure invented, and says ChatGPT kept fetching and citing it for months. If you build agents or retrieval systems that answer questions with numbers, your product has the same weak spot. The good news is that the fix is mostly plumbing: decide which sources your agent may treat as final, and check the source behind a number, not only whether the number appears there. You can start this week.

What happened with the 50,000 fake statistics

Metehan Yesilyurt, a researcher at the AI-search analytics company Peec AI, published the account on Peec's blog, dated October 7, 2026. It reached the Hacker News front page on October 20, where it passed 260 points and 140 comments by mid-afternoon Eastern.

Here is what he describes. He used GPT-4.1 with a minimal prompt to generate 50,000 statistics pages in a single day, covering SEO, content marketing, social media and e-commerce across 200 countries, on a site called stateglobe.com. He says that cost about $30, and roughly $50 once he added 15,000 listicles later. In his words: "Every one of them was fake."

About seven months after launch, he checked the site's traffic. According to his Cloudflare data, the site received more than 600,000 bot requests in the last 30 days, and around 72,000 of them came from ChatGPT-User.

That bot name matters. OpenAI's crawler documentation says ChatGPT-User handles user-initiated actions and is "not used for crawling the web in an automatic fashion." So those requests are tied to people using ChatGPT, not a training crawler sweeping the web. Yesilyurt draws the same conclusion: "So these are not training crawls." A request count still cannot tell you how many answers actually showed one of his numbers.

The numbers spread from there. He reports 330+ referring domains (other sites linking to his pages, counted with the SEO tools Ahrefs and Semrush), including pages on Shopify, DHL and Bluehost. He also lists an academic paper, investment research from what he calls Qatar's largest financial group, an Indian economics publication and a Canadian influencer community report, and notes these are only the citations that linked back.

Meanwhile Google, by his account, deindexed the pages (dropped them from its search results) after two updates. The site still runs, now with a disclaimer.

One caveat before going further. These are one person's figures, read from his own Cloudflare, GA4 and SEO-tool dashboards. Nobody outside has audited them, and the Hacker News thread argues about the method as much as the result. Treat the exact counts as his. The mechanism he describes is the part worth your attention, because it does not depend on any single number being right.

Why a fake number wins a citation

Yesilyurt's explanation is short: "LLMs use rerankers that score relevance, but not truth." It is his reading of how these systems behave, not a published spec, and it fits what his logs show.

A reranker is the step in a search pipeline that takes a pile of candidate pages and sorts them by how well each one answers the question. Think of a librarian who pulls twenty books off the shelf and hands you the three whose chapter titles best match what you asked. That librarian is good at matching. Fact-checking the chapters is not part of the job.

Now picture what a statistics page looks like to that librarian. The title says "E-commerce statistics 2026." The headings repeat the year, which reads as fresh. Every paragraph is one clean claim with one clean number. For a question like "what share of shoppers abandon carts in Germany," a page like that is close to a perfect match. Yesilyurt puts it bluntly: "Fact-checking doesn't happen anywhere in the process."

Then the loop kicks in. Once a few real sites copy the number (with a link), the next retrieval finds the number in more places, each one adding a little apparent authority. He calls this citation laundering: "every citation made it look more trustworthy."

Google caught it. A retrieval pipeline built for live answers, by his logs, did not.

The part that should worry builders

This is not only an OpenAI story. Any system that answers a question by searching the web, picking passages and writing a reply has the same shape. That includes the research agent you wired up with a search tool, the support bot that pulls from public docs, and the RAG pipeline (retrieval-augmented generation, where the model looks documents up before it answers) that indexes "industry reports" a teammate dropped into a folder.

Here's the uncomfortable insight. The obvious safety check, "make the model cite its sources," does nothing against this. A citation check that confirms the number appears on the linked page passes stateglobe.com every time. The fake page contains its own fake number. The citation is real. The fact is not.

So the question a builder has to answer is not "did the agent cite something?" It is "what kind of source is the agent allowed to treat as the final word on a number?"

Put this into practice

None of this needs a new framework. It is a handful of rules, a small piece of code and one log query. Start with the step that matches where your agent gets its numbers.

1. Find out how often your agent repeats a figure. Pull a day of outputs from your logs and search for numbers with a percent sign, a currency symbol or the word "million." A rough count tells you whether this is a corner case or half your traffic. If you run a research or reporting agent, expect it to be a lot.

2. Give the agent a source rule, in writing. Add something like this to the system prompt or tool instructions:

When you state a number, put the source URL and the year next to it.
Prefer primary sources: government statistics offices, company filings,
the original study. If the only source is a statistics roundup, an
aggregator or a blog, say so in the answer and mark the figure "unverified".
Never combine numbers from different sources into a new statistic.

This will not stop a model from finding a bad page. It does force the answer to show its provenance, so a reader (or the next step in your pipeline) can judge it.

3. Route common statistics to primary data first. For figures that have an official home, skip open web search. The US Bureau of Labor Statistics, for example, offers a public data API (version 1.0 needs no sign-up; version 2.0 requires registering), and many national statistics offices offer similar access. A Hacker News commenter in the thread suggested exactly this: have AI tools query primary statistical sources directly instead of searching for numbers. An agent that calls a data tool for "unemployment rate" never meets a stats farm.

4. Check the source as well as the quote. If you already verify that a cited number appears on the cited page, keep that check and add a second one about the page itself. Here is a minimal version in Python you can adapt:

from urllib.parse import urlparse

PRIMARY = {"bls.gov", "census.gov", "sec.gov", "ec.europa.eu", "who.int"}

def source_tier(url: str) -> str:
    host = urlparse(url).hostname or ""
    if any(host == d or host.endswith("." + d) for d in PRIMARY):
        return "primary"
    return "unverified"

def label_claim(number: str, url: str, page_text: str) -> str:
    if number not in page_text:
        return "drop"            # the page does not say it
    return source_tier(url)      # the page says it, but who is the page?

Run it on each figure before the answer goes out. Anything labeled "unverified" either gets the warning from step 2 or gets sent to a primary-source lookup. The domain list is yours to grow; the point is that "the page says so" and "a trustworthy source says so" become two different checks.

5. If you publish data, make it easy to trust. Yesilyurt's own advice for statistics pages is sound for honest publishers: "Make one claim per sentence," "Put the source and year next to every number," and "Show the update date (and actually update it)." If retrieval systems are going to quote you, give them the provenance to quote too.

6. Look at who is reading your site. Search your server logs, or the logs of your CDN (the caching network in front of your site), for the ChatGPT-User, OAI-SearchBot and PerplexityBot user agents, which Yesilyurt recommends tracking. It shows you which of your pages AI assistants are actually opening on behalf of users. User-agent strings can be faked, so if the numbers matter, check the requesting IPs against the ranges OpenAI publishes for its bots. OpenAI notes that robots.txt rules "may not apply" to ChatGPT-User because a person started the action, so blocking is not a reliable lever here; knowing is.

What to expect when you do this: the first log search tells you how big the problem is for your product, and the source rule makes answers a little longer and easier to check. Some answers will come back marked "unverified." That is the system working.

Honest limitations

The evidence is thin in one specific way. This is a single site, run by someone whose company sells AI-search visibility tracking, with numbers read from his own dashboards. The 72,000 ChatGPT-User requests tell you pages were fetched, not how many answers showed a fake number to a person.

It may not last. Yesilyurt himself says the tactic may stop working as models adapt, and Google already filtered the site. OpenAI could add quality signals to its retrieval tomorrow, and nothing here tells you how its system ranks pages today.

A domain allowlist is crude. Government sites get numbers wrong too, and plenty of excellent data lives on independent research blogs. The tier check in step 4 is a starting point that trades some coverage for some safety, not a truth detector.

And none of this fixes the model making up a number on its own with no retrieval at all. That is a different failure, and the source rule only helps because it forces the model to attach a URL that someone can check.

Where this leaves you

The useful shift here is small and permanent. A citation means "found," not "checked." Once your agent repeats numbers to real people, someone in your stack has to own the second meaning.

You do not have to solve web spam. You have to decide which sources your agent may treat as final, put the source beside every figure, and send anything else to a primary lookup or a visible warning. Pick one agent this week, run the log search from step 1, and see how many of its numbers you could defend if someone asked where they came from.

Sources: Peec AI, "Publish stats, get cited" (Oct 7, 2026) · Hacker News discussion (Oct 20, 2026) · OpenAI crawler documentation · BLS Public Data API