This morning a founder posted something on r/ProductManagement that almost nobody does in public. Before running a small paid test in early November, 20 people offered five lessons at the full price of $14.99, he wrote down his pass line: at least 3 of the 20 pay, and at least 2 of those are strangers. His reason for the strangers rule is the most honest sentence I have read this week. His only paying customer so far is someone he knows, and he does not want to fool himself twice.
The number he picked is debatable. The order in which he picked it is not. He decided what failure looks like while failure was still possible.
Why does the pass mark have to come first?
Because a bar set after the results arrive is not a bar, it is a caption. Teams that skip this step do not consciously cheat. They look at whatever came in, two sales, forty signups, a warm email, and ask whether it feels like progress. Almost anything feels like progress when months of work are waiting on a yes. The result gets interpreted instead of judged, and the project continues by default.
Writing the line first costs nothing and removes the single biggest source of self-deception in early validation: the quiet renegotiation of what counts as success, conducted by the person with the most to lose from a no.
What makes a good pass line?
Three parts. A number: how many people must act, not approve, within a fixed window. An independence rule: who the evidence must come from. Friends buy out of loyalty, strangers buy out of need, so a pass built on people who know you measures your relationships, not your market. And a consequence: what you will do on a miss, written down while you still mean it. Change the offer, change the segment, or stop.
Details sharpen it further. Full price beats a discount, because a discount tests generosity, not demand. If there is a money-back guarantee, refunds count against the pass, since the real decision happens when the buyer can still change their mind.
What if the result lands exactly on the line?
It usually will. Small tests love ambiguous outcomes: two strangers and one friend, three sales but one refund request pending. Decide the edge cases now, because the ambiguous result is where hindsight argues loudest. A practical rule: a result on the line is a fail that earned one more test, with a sharper bar and a fresh set of strangers. It is not a pass.
Is 3 out of 20 the right number?
Fifteen percent conversion on a cold full-price offer would be a strong result, and on 20 people the margin of error is enormous. But arguing about the number misses what the number is for. At this scale a test does not measure a market, it disciplines a decision. The founder who wrote 3 of 20 before the test will learn something real from 1 of 20. The founder who wrote nothing will learn that 1 of 20 is actually quite promising for a first try.
Key takeaways
- Write the pass line before the test: a number, a deadline, an action for a miss.
- Add an independence rule. Evidence from strangers outranks evidence from friends, every time.
- Test at full price and count refunds inside the result, not outside it.
- Decide the ambiguous case in advance. A result on the line is a fail that earned a sharper retest.
A test you cannot fail is a ritual. The pass mark, written first, is what turns it back into a question.