A study on AI homework just explained why your dashboard looks great while your revenue does not.
What is a vanity metric?
A vanity metric is any number that can improve without the underlying business improving. Signups from disposable emails. Sessions that never return. Upvotes from people who will never buy. The number is real, the meaning is not.
This week a study made the pattern unusually clean. Students using AI saw homework scores rise. Then exam scores dropped. The homework was the proxy, the exam was the truth, and the gap stayed invisible until the crutch was removed. Startup dashboards work exactly like this. The proxy inflates, the underlying capability quietly falls, and nobody notices until the market runs the exam.
Why do vanity metrics feel so convincing?
Because they are real numbers, honestly collected, moving in the right direction. Nobody fakes them on purpose at first.
A founder posted this week that his B2B freemium product hit 5,000 trial users. His server logs told a second story: 60% were disposable email addresses scraping his free tier and vanishing. The 5,000 was true. It just measured abuse, not demand. The uncomfortable part is that the inflated version goes in the deck, because the audience for the deck rewards the inflated version.
Vanity metrics survive because three groups benefit from them: the founder feels progress, the team feels momentum, and investors feel velocity. The only party with no vote is the customer, who expresses their opinion later, by not paying.
How do you tell a real signal from a fake one?
Ask one question: what did the person give up?
Real signals cost the giver something. Money is the strongest. Time is next, a stranger who spends forty minutes in your product on a Tuesday is telling you something no survey can. Data has weight, someone importing their real records is planting a flag. Reputation counts too, a person who recommends you to a colleague is spending their credibility.
Compare that to another story from this week. A founder watched two Reddit posts with zero upvotes produce his only four paying customers, while Instagram sent 111 visitors and nothing. The upvote was never the signal. The invoice was. Applause is free, and anything free is unlimited, and anything unlimited measures nothing.
Can a metric be real for one company and fake for another?
Yes, and this is where most advice fails. Signups are a real signal for a product that charges at signup. They are noise for a freemium tool with an open door. Daily active users mean everything for a habit product and nothing for an annual tax tool. The metric is not vain by nature, it is vain by context.
The test stays the same in every context. Could this number rise while real demand stays flat? If someone was paid to inflate it, could they? If yes, treat it as decoration.
Key takeaways
- A vanity metric is any number that can rise without the business improving.
- The homework-vs-exam gap stays invisible until the market forces the exam.
- Real signals cost the giver something: money, time, data, or reputation.
- Free actions like upvotes, likes, and disposable signups measure enthusiasm for clicking, not intent to buy.
- Every metric should pass one test: could this inflate while demand stays flat?
FAQ
What is the difference between a vanity metric and an actionable metric?
A vanity metric describes attention, an actionable metric predicts behavior. If a number cannot change a decision you would make this week, it is decoration.
Are signups a vanity metric?
It depends on the cost of signing up. A signup that requires a card or real data carries signal. A free signup with a disposable email carries close to none.
Why do investors accept vanity metrics in pitch decks?
Many do not, but the incentive structure rewards top-line velocity in the short term. The correction arrives at the next round, when growth has to reconcile with revenue.
What should an early-stage founder measure instead?
Count the things people gave up: payments, deposits, sustained usage, real data imported, unprompted referrals. Five of those beat five thousand hollow signups.
Can AI tools create vanity metrics?
Yes, faster than ever. AI inflates output metrics like content shipped and tickets closed while the quality signal lags behind. The homework study is the cleanest demonstration so far.