This week an agency owner who has spent eight years building MVPs for founders wrote something most agencies will not say out loud. Of the roughly sixty things clients asked him to build under that name, maybe fifteen were actually MVPs. The rest were three-month roadmaps with the word MVP stapled to the front, because MVP sounds cheaper and the founder already knew the full quote would hurt.
He also named the tell, and it is a good one. The settings page. A real test does not have one. A product does.
What is the difference between a test and a product?
A test exists to answer one question you cannot answer today. Will a stranger sign up for this. Will anyone pay this price. Does the problem exist outside your head. Everything in a test is subordinate to the question, which is why real ones are small, ugly and fast.
A product exists to serve users you already have. User roles, billing tiers, email notifications, a settings page. All of that is sensible once demand is proven. Before demand is proven, it is scaffolding around an unasked question.
The two get confused because building a product feels better. Nobody can reject a thing that is still being built. The agency owner called the settings page procrastination with an invoice attached, and that is the mechanism exactly. Building is the most respectable available way to not find out.
How do you know which one you are building?
One question does most of the work: what will you know at the end of this that you do not know today?
Ask it about a twelve-week build and the honest answer is usually: that the thing can be built. Which you already knew, because almost everything can be built now. The question your runway actually depends on, whether anyone wants it, is still scheduled for day 91, after the sunk cost has grown big enough to argue back.
That is the quiet damage of a long fake MVP. It is not just slow learning. By the time evidence arrives, you have three months of your own work sitting on the other side of the scale, and no dataset feels heavy enough to outweigh it.
Why does this keep happening in the AI era?
Because building got cheap and the confusion got cheaper with it. The same week as the agency confession, a builder pointed an AI at 1,575 posts in his own maker community and published the split: about 15 percent of people described a real problem, 2 percent asked for a new tool, and 6.5 percent were promoting something they had built. Three sellers for every person asking to buy. Faster building did not produce more demand. It produced more supply wearing demand's clothes.
When shipping took six months, the cost forced some discipline. Now the weekend build has removed the cost without adding the question. You can reach the wrong conclusion faster than ever, in production, with a settings page.
What does a real test look like?
It fits on an index card, another of the agency owner's rules that survives contact with reality. One question, one audience, one way to answer it, one date. The best one he ever ran took nine days and four off-the-shelf tools. Not because small is virtuous, but because the question was allowed to stay in charge of the scope.
And the answer has to come from someone with no stake in your feelings. Your cofounder cannot answer it. Your friends will answer a different question, usually about the relationship. The only vote that counts belongs to a stranger who was free to say no, and the whole art of testing is arranging to hear that vote as early and as cheaply as possible.
Key takeaways
- A test answers one question you cannot answer today. A product serves users you have not proven exist yet.
- The diagnostic is: what will you know at the end that you do not know now? "That it can be built" does not count.
- Long builds do not just delay evidence, they grow a sunk cost that argues against whatever the evidence says.
- AI made building cheap, not knowing. The wrong conclusion just ships faster now.
- If the plan has a settings page before it has a stranger's verdict, it is a product wearing a test's name.
Sixty builds, fifteen tests. The ratio is not an agency problem. It is the honest census of how founders spend their first year. The settings page is a confession, and the market reads it before you do.