Validating
Log in

Every Question in an Experiment Wants One of Three Things

Every question in a Validating experiment asks you to pick, to measure, or to say it yourself. Here is what each collects and what a good answer looks like.

Pick, measure, or say it yourself. Knowing which one you are in is most of the job.

An experiment is a short sequence of questions. They look different from each other on purpose, and the differences are not styling. Underneath, every question belongs to one of three families, and the family is a statement about what kind of answer is being collected.

What are the three families?

Pick. The question puts a fixed set of options in front of you and you choose. One of them, or several, depending on the question. Two versions side by side. A yes or a no. A set of options to tap.

Measure. The question asks you to place something on a range. You are not choosing between options here, you are saying how much.

Say it yourself. The question opens up and asks you to produce something. Your own words, your own ordering, your own reaction to a concept. Nothing is pre-written for you.

Every question you will ever answer is one of those three. There is no fourth kind.

Why does the shape matter to me?

Because a good answer in one family is a wasted answer in another.

In a pick question, the useful thing you can do is choose the option you would actually take, not the one that sounds most reasonable. The options are fixed, so the only information you add is which one. The moment you pick the diplomatic option instead of the real one, that question has collected nothing.

In a measure question, the middle is where answers go to die. If the thing barely registers, say so at the bottom of the range. If it is the worst part of your week, say so at the top. A safe middle reads as no signal at all.

In a say-it-yourself question, specificity is the entire value. Anyone can write "useful". Nobody else can write the sentence about the client who calls at seven on a Friday.

Are these questions answered by real people?

Yes. Every question type in the set a validator sees is built to be answered by a human, and there is no version of them that a machine fills in on your behalf.

That is the whole point of the arrangement. Anyone can get a plausible answer out of a model in nine seconds for free. What they cannot get that way is you.

Can my answer change which question comes next?

Sometimes, and only from one direction.

A question can carry a rule that routes you, so answering one way sends you to a different question rather than the next one in the sequence. Those rules can only read answers that have a fixed value, which means picks and numbers.

What you write in your own words never routes you anywhere. It is read, it is not used to steer you. A short answer will not get you out of an experiment faster, and a long one will not add questions.

That is why two people can start the same experiment and answer a different number of questions.

What happens while I am answering?

Your answers are saved one question at a time as you go, not as one submission at the end. A counter at the top shows you where you are in the sequence.

When a question asks you to write, there is a character counter, and the room is deliberately not generous. That is a design decision rather than a limit on your thinking. Short and specific beats long and general in every one of these.

Is there a right answer?

No, and it is worth being blunt about why.

Nobody is scoring you on agreement. The person on the other end has usually been in love with their idea for months and has been surrounded by people being nice about it. The reason you are being paid is that you have no reason to be nice about it.

The only bad answer is a vague one.

Key takeaways

  • Every question belongs to one of three families: pick from a fixed set, place something on a range, or produce something yourself.
  • The family tells you what kind of answer is being collected and what a useful one looks like.
  • In a pick question, choose the option you would really take, not the reasonable sounding one.
  • In a measure question, avoid the middle unless the middle is true.
  • Only fixed answers can route you to a different next question. What you write in your own words never changes your path.
  • Answers save as you go, and a counter shows you where you are.

FAQ

Will I be told which family a question belongs to?

No, and you do not need to be. It is obvious from the question itself. Either the options are in front of you, or there is a range to move, or there is space to produce something.

Why do two people get a different number of questions?

Routing rules. A pick or a number can send you down a shorter or simply a different path.

Does writing more get me more?

No. The room for writing is deliberately small, and specific beats long every time.

Does answering faster or slower change anything?

No. What is collected is the answer, not the speed.

What if I do not have a real opinion on a question?

Give the least flattering true thing you have. A weak but honest answer is data. A confident invented one is noise.