Independent AI intelligence Two editions daily · ET
Fervor AI

Analysis · November 21, 2026 · concept

A90 same-advice experimentModel defaultsfrontier-modelsfine-tuningagent-security

AI Models All Give the Same Answer. Two Prompts Make It Yours.

Sixty-five models from thirty companies were asked the same open questions 127,918 times. The answers barely moved. Here is what that means for anything you build on top of them, and the prompts that actually fix it.

Ask an AI model to invent a random person and there is a good chance she is 34, works as a marine biologist and lives in Portland or Seattle. Ask it where to put $10,000 and you will almost certainly hear "a broad index fund." Ask for a startup name and it will probably start with "Nex."

That is not a quirk of one chatbot. It is what happens across nearly every model on the market, according to an experiment published on November 20 by Cody Collier of A90. He sent the same plain questions to 65 models from 30 companies through OpenRouter (a service that routes one API call to many model providers) and collected 127,918 answers.

The result is a problem for builders, not just for people asking for book recommendations. If your feature, your test data or your review step leans on what a model says by default, it says the same thing for your competitor, your users' other apps and the model you brought in as a second opinion. A model's default answer is a shared resource: everyone who asks the same way gets it.

The good news is that the same experiment tested fixes, and two of them work very well.

What the experiment found

Collier kept the setup plain on purpose: default settings, no system prompt, thinking turned off or set to its lowest level, and each model asked the same question between 20 and 200 times. Every answer is published in a CSV file, which is more than most AI studies give you.

A few numbers from the write-up carry the story:

  • 98% of answers to "$10,000 to invest" named a broad index fund.
  • 73% of book recommendations were "Man's Search for Meaning."
  • 76% of business ideas were a subscription box.
  • 69% of 3,151 invented people were exactly 34 years old, and 81% were women.
  • 69% of suggested startup names began with "Nex."

Some of those defaults are fine. Collier says so himself: an index fund may simply be the right answer for $10,000. The trouble starts when the default is supposed to be personal, creative or random, and it is none of those things.

The defaults also leak into the world. The name "Elena Vance," which Collier calls Gemini's signature invented person, appears in 684 new GitHub READMEs in 2026 and in none during the six years before. Claude has its own, "Marisol Okafor." If you have ever wondered why demo data across a dozen open-source projects looks oddly familiar, now you know.

Where the sameness comes from

The obvious guess is that every model learned from the same internet. Collier tested that and found something more specific.

He measured a small open model, OLMo 2 1B, after each stage of its training on one question: "pick a random number between 1 and 100." Randomness is measured in bits here; a perfectly fair pick scores 6.64 and a model that always says the same number scores 0. The freshly pretrained model scored 6.51, close to fair. After supervised fine-tuning (the stage where a model learns from example conversations showing how an assistant should answer), it dropped to 4.95. The later tuning stages barely moved it.

In Collier's words, "the favorite arrives in a single step, supervised fine-tuning." He also read all 866,138 examples in that model's fine-tuning data. The favorite number and the age 34 are not in there. Names are: of 573 mentions of "Vance," 556 were written by ChatGPT.

That is one small open model, so it can't settle the question for every lab. But it points somewhere specific: in this case the favorite answer was added during training rather than soaked up from the internet, and some of the recurring names arrived in training examples written by another company's model. If labs train on each other's outputs, directly or through shared datasets, defaults can travel with them.

Why your second opinion may not be one

Here is the part builders should sit with.

A common safety pattern in agent workflows is to have a second model check the first: one model writes the code or the answer, a model from a different vendor reviews it. The logic is that a different model makes different mistakes.

Collier turned each model's answers into a fingerprint and measured how alike every pair was, without telling the analysis who made what. 59% of models sat closest to a sibling from their own lab, against 4% by chance, and a blind test named the right parent family for 22 of 23 open fine-tunes (open models someone trained further from a known starting model). Some of the closest pairs crossed company lines: Grok's nearest relative was an OpenAI model, and Qwen4 Max answered most like Claude Opus 5.5.

Read that carefully. It does not prove that reviewers from two vendors share the same blind spots on your code. Collier warns that "likeness is not ancestry," and his questions were about books and cities, not bugs. But it does say that models from different companies drift toward the same answers more than their logos suggest. When the checker shares training habits with the writer, their agreement is weaker evidence than it looks.

Compare that with the most trusted machine check in the news this week. When the communities behind Lean (a programming language for writing machine-checked math proofs) and Mathlib (its shared math library) certified a 17-million-line Riemann Hypothesis proof on November 21, the confidence came from Lean's own kernel (the small core program that checks every proof step) plus several independently written checkers in different languages, two of which (con-leche and con-ron) come with formal proofs that they cannot accept a false proof. That is what an independent check looks like. A second chatbot is a weaker version of it.

The fixes that work, and the ones that don't

Collier measured one number for each fix: the chance that a stranger who asks the same way gets your exact answer, averaged over six questions (a book, a business idea, a trip, a city, a stock and a gift). A plain question scores 34%.

What barely helps. Telling the model "don't give me the most common answer" got it down to 15%, but mostly by moving everyone to the same second choice. Book recommendations shifted from one title at 73% to another at 17%; trips moved from Lisbon to Ljubljana, which then took 58% of answers. Asking it to "be original" doesn't work, according to Collier. Thinking mode made answers more common, not less: on a "which US city" question, Austin went from 47% with thinking off to 57% with it on. Switching languages didn't help either. And "another one, please" only works inside one chat; open a new chat and Claude Opus 5.5 went straight back to the same book.

What helps somewhat. Putting a random seed in the prompt ("Random seed: 483920. Use it to choose.") cut the overlap to 19%.

What works. Two prompts stand out:

  1. Ask for ten, then pick one yourself. "List 10 options, then pick number 7" dropped the overlap to 7%. The trick is that you choose the number, not the model.
  2. Say who you are. One sentence of real context ("I'm 43, a dental hygienist in Leeds...") dropped it to 4%. The model stops answering an average stranger and starts answering you.

Collier's summary is the useful rule: what worked was "giving it something nobody else gives it."

There is a catch with the second fix. A one-line description moves you into a new lane, and the lane can be crowded too. A retired teacher in New Jersey asking where to move got Portugal 92% of the time. Context makes the answer yours only if the context is specific.

The security version of this problem

Sameness is a style issue in a book recommendation. It is a security issue in anything that should be secret.

Collier asked models to write passwords. 25% of AI-written passwords contained the two characters "vL," against 0.21% by chance, and one model repeated a single password 19 times out of 30. When the same models wrote session-token code that called a proper secure generator, 384 of 390 attempts were correct.

The lesson is blunt: a model is a bad source of randomness and a fine writer of code that calls a real source of randomness. Given a bash tool, 54 of 63 models reached for a real random number generator on their own.

Put this into practice

None of this needs new tools. It needs you to stop trusting the default in four places.

1. In product features, pass real context. If your app asks a model for a recommendation, a name or an idea, put the user's own details in the prompt: their stated goals, history, location or past choices. Generic prompts produce the same answer your competitor's app gives.

2. When you need variety, make the choice outside the model. Ask for a numbered list of options and pick one in code:

import secrets

prompt = (
    "Suggest names for a budgeting app. List 10 different options, "
    "one per line, numbered 1 to 10. No other text."
)
options = [line.split(". ", 1)[-1] for line in call_model(prompt).splitlines() if line.strip()]
if len(options) < 2:
    raise ValueError("model did not return a usable list")
choice = options[secrets.randbelow(len(options))]

Here call_model stands in for whatever client you already use, and the length check catches the times a model ignores the format. The model brings range; your code brings the randomness.

3. Never let a model invent secrets, test identities or sample data you will treat as random. Passwords, tokens, IDs and seeds come from secrets (Python), crypto.randomUUID() (JavaScript) or openssl rand. For fake people in test data, a library like Faker beats asking a model to "make up some users," which will hand you a lot of 34-year-old marine biologists.

4. Don't treat a second model as an independent reviewer. Keep it if it catches things, but put at least one check in front of every merge that shares nothing with any model: a type checker, a test suite the agent cannot edit, a linter, a policy rule that blocks by default. Then test the reviewer itself. Plant a few known bugs in a branch and see whether it flags them. If it agrees with the writer on the planted bugs too, you have learned what its approval is worth.

To see the problem in your own stack, try this today: send your product's most common prompt to two or three different models twenty times each and count how often the top answer repeats. It takes ten minutes and the number is usually higher than anyone on the team expects.

Honest limitations

This is one person's experiment, run on one day through one provider per model. Collier says plainly that 50 asks per question is "enough to see a family, not to rank small differences between two models," and models may answer differently elsewhere or after an update. Thinking was off or at its lowest in the main runs.

The fingerprint study measures likeness, not cause. Two models that answer alike may share training data rather than a parent, and nothing here measures whether two reviewers miss the same bugs in real code. That is a reasonable worry the data points toward, not a finding.

The training-stage result comes from one small open model, because the large labs don't publish their fine-tuning data. And sameness is not always wrong. For some questions, the common answer is the right one.

The default is a shared resource

Every model ships with a set of answers it gives when nobody tells it anything. You share those answers with every other developer who didn't tell it anything either.

That is fine for a demo. For a product, a test suite or a security control, decide where your output should come from. Give the model your user, your random number or your tool, and it stops sounding like everyone else.

Sources: A90, "AI is giving everyone the same advice" (Cody Collier, Nov 20 2026) and its answers CSV; Lean and Mathlib communities, "Lean verification of the Riemann Hypothesis" (Nov 21 2026); Python secrets module docs.