Asking an AI whether your startup idea is good has become the default first move for thousands of founders. It is fast, articulate and free. This week, a research report showed exactly what that articulate answer can be made of.
On September 2, 2026, Trellner Research traced the sources behind AI software recommendations and found three websites that had generated 215,128 pages ranking the best software across 380 categories. The pages were not written for readers. They were written to be retrieved and quoted by AI answer engines. It worked: engines like Perplexity cite them as sources. The same report found that nearly 60 percent of the sources behind AI recommendations sit outside the 100,000 most visited websites.
In plain terms: when an AI tells you what the market thinks, a growing share of what it read was never a market at all.
What does AI actually check when it evaluates your idea?
Nothing about your customers. A language model has never watched a person hesitate at your price, close the tab, or quietly go back to the spreadsheet they already use. What it does is compare your description against the text it has read: pitch essays, listicles, forum threads, roundup pages. Its verdict is a judgment about how your idea sounds next to that corpus.
That corpus was always biased toward survivors and self reporting. The new problem is that part of it is now synthetic on purpose.
Why is AI sourced opinion getting less reliable?
Because manufacturing citable content is now nearly free, and being cited by an answer engine is worth money. Publishing 215,128 pages is what three sites did. There is an emerging industry doing the same thing across every category, usually under the name answer engine optimization. The pages are structured, confident and plausible, which is exactly what retrieval systems reward.
None of this makes AI useless. It makes AI a mirror of published text, at the precise moment publishing text became free. A mirror like that can still be handy. You just should not ask it to predict what strangers will pay for.
What is AI feedback genuinely good for?
Everything that happens before the verdict. It can sharpen a fuzzy problem statement in minutes. It can list the assumptions hiding inside your pitch faster than you would alone. It can surface competitors you have not heard of, which you then verify yourself. It can turn a vague plan to talk to users into a concrete list of non leading questions. Used this way, AI compresses the preparation phase of validation from days to hours.
What can only real people tell you?
Whether they pay. Whether they come back. What they actually compared you to before saying no. Payment is behavior, and there is no behavior data about an idea that has not met the market yet. The only way to create that data is to put the idea in front of real people under conditions where saying yes costs them something: money, time or reputation.
This is also why the sequence matters. Founders who ask AI for a verdict tend to stop there, because the answer feels like research. Founders who ask AI for a sharper test tend to go run it.
How do you combine both without fooling yourself?
One rule covers it: AI shapes the question, people settle the answer. Draft your riskiest assumption with a model, then test it on humans who have no reason to be kind. And weigh every yes by what it cost the person saying it. A yes that cost nothing, whether it came from a friend, a survey or a model that read 215,128 manufactured pages, rounds to zero.
Key takeaways
- AI feedback summarizes published text, and a growing share of that text is manufactured to be cited by AI systems, not read by people.
- Trellner Research found three sites with 215,128 machine targeted software recommendation pages that answer engines cite as sources.
- Use AI before the verdict: sharpening assumptions, mapping competitors, preparing non leading questions.
- Only costly behavior from real people validates demand: payment, committed time, repeat usage.
- If a yes cost the person nothing, treat it as noise, whatever produced it.
FAQ
Can ChatGPT validate my startup idea?
It can help you refine the idea, list assumptions and find competitors you missed. It cannot validate demand, because validation is evidence that real people will pay or commit time, and a model has no access to the behavior of your future customers.
What is answer engine optimization and why should founders care?
It is the practice of publishing content designed to be retrieved and cited by AI systems rather than read by people. It means the corpus behind AI opinions is increasingly manufactured, so an AI judgment about your market may rest on pages nobody ever read.
Is AI feedback worse than asking friends and family?
It fails the same way at a different scale. A friend tells you what keeps the relationship comfortable, an AI tells you what its corpus sounds like. Neither one pays you.
What counts as real validation evidence?
Behavior with a cost attached. Someone pays, pre-orders, commits time to a trial, introduces you to their team, or cancels a competitor. Words, likes and scores are inputs for better tests, not evidence.
How many real people should I hear from before building?
There is no magic number. Stop when the answers start repeating and at least a few strangers have taken an action that cost them something. If nobody has, the repeating answers are politeness, not a pattern.