Product Experiments
Once you know which assumption to test, this is how: ten experiments, what each one is for, how to run it, and — the part teams skip — what counts as a clear signal.
Match the test to the doubt
Every experiment answers one of the four assumption types. Name your doubt, and the shortlist picks itself.
Desirability — Do they want it?
The riskiest layer for most bets, and the one with the richest toolkit — because "will anyone care?" is answered by watching behaviour, not by shipping. These run from a few hours to a couple of weeks, and none of them require the real product.
Fake door / painted door
A button, menu item, or CTA for a feature that doesn’t exist yet; clicking it records intent and shows a short survey or a "coming soon".
When: You need to know whether anyone wants a specific feature before you build it — the fastest desirability test there is.
How: Add the entry point in the real product (or a prototype), instrument the click, and route it to a holding message. Run until you have enough traffic to judge.
Clear signal: Click-through above a threshold you set first — ideally with a follow-up survey naming a real need, not idle curiosity.
Hours
Counting curiosity clicks as demand. A click is interest; capture an email or a reason to separate "ooh, what’s that" from "I need this".
Landing-page smoke test
A standalone page that sells the value proposition with a single call to action — sign up, join the waitlist, pre-order — driven by a small ad or email spend.
When: You’re testing demand for a whole product or a big bet, not one in-app feature.
How: Write the page as if the thing exists, send a defined slice of traffic to it, and measure conversion on the one CTA.
Clear signal: Conversion above the rate the business would need — a number you calculate before launch, not admire after.
Days
Vanity traffic. Sending it to your own followers tests their politeness, not the market.
Story-based interviews
One-on-one conversations that dig into a specific past event — what someone actually did last time they hit the problem, not what they say they’d do.
When: Early, when you’re unsure the problem is even real or worth solving.
How: Recruit people in the segment and ask for the story of the last time it happened ("tell me about the last time…", never "would you use…"). Follow the facts.
Clear signal: Consistent, unprompted mentions of the pain — and a current workaround they already pay for in time or money.
Days·also tests sources the assumptions in the first place
Leading questions and hypotheticals. "Would you use a weekly digest?" gets a polite yes from everyone and signal from no one.
Concierge
You deliver the outcome your product would automate — by hand, visibly, for a few real customers.
When: You want to test whether the service is wanted (and worth paying for) before building software to deliver it.
How: Recruit a handful of customers and do the job for them manually — spreadsheets, emails, phone calls — learning what "good" actually means.
Clear signal: They come back, and ideally pay — plus you learn the real steps the eventual product must automate.
A week+·also tests viability
Automating too early. Stay manual long enough to hear the edge cases; the moment you script it, you stop learning.
Wizard of Oz
The interface looks fully automated to the user, but a human is doing the work behind the curtain.
When: You want a realistic usage test of an experience whose backend — often AI or algorithmic — is expensive to build.
How: Ship a real-looking front end; a human fulfils each request in real time; the user never knows.
Clear signal: Real engagement at a level that would justify building the true backend — plus a spec, because the human’s logic is the algorithm you now need.
A week·also tests feasibility
Confusing it with concierge. Wizard of Oz hides the human to test the experience; concierge shows the human to test the value.
Usability — Can they use it?
Rarely the riskiest assumption at the idea stage — usability is usually fixable by iterating. It becomes the linchpin when the whole value depends on unaided self-serve: onboarding, imports, anything where a human stepping in defeats the point.
Prototype & usability test
A clickable mockup put in front of target users who attempt real tasks while thinking aloud.
When: The value is clear but the interaction is risky — onboarding, complex flows, anything where confusion kills adoption.
How: Build the flow in a prototyping tool, give five users a concrete task, watch silently, and note where they hesitate or go wrong.
Clear signal: Most users finish unaided within a target time, and the stumbles cluster on fixable spots rather than the core concept.
Days
Helping. The moment you explain, you’ve contaminated the test — a real user won’t have you sitting beside them.
First-click / tree test
A lightweight findability check: where do you click to do X on this screen (first-click), or navigate a text-only menu to locate X (tree test)?
When: You suspect a discoverability or navigation problem, not a comprehension one.
How: Show a static screen or a menu structure to 10+ users, ask where they’d go for a task, and measure first-click success.
Clear signal: A strong majority first-click the intended path. If they scatter, the label or the placement is wrong.
Hours
Testing only the happy path. Include the tasks users get wrong today, not just the ones you designed for.
Feasibility — Can we build it?
The type engineers surface naturally — and the one where the test is often itself a build. The discipline is the timebox: a spike is a throwaway experiment with an end date, not the first sprint of the feature wearing a lab coat.
Technical spike
A timeboxed, throwaway piece of code written to answer one feasibility question — not to ship.
When: The risk is "can we even build or run this?" — a new API, a performance ceiling, an integration, an AI accuracy bar.
How: Fix the question and a time budget (a day or two), build the smallest thing that answers it, then delete it.
Clear signal: A concrete yes/no against a pre-set bar — p95 latency < 300 ms, error rate < 2% — measured, not estimated.
Days
The spike quietly becoming the feature. If you’re keeping the code, you’ve stopped experimenting and started building — usually on a shaky base.
Viability — Does the business work?
The type founders test last and regret first. Price, margin, and cost-to-serve feel like "later problems" — until a desirable, usable, buildable product turns out to be a bad business. These tests are mostly conversations and spreadsheets, which is exactly why skipping them is inexcusable.
Pricing / paywall test
Putting a real price in front of real prospects — a pricing page, a paywall, a quoted number in a sales call — and measuring willingness to pay.
When: The idea is desirable and buildable, but you’re unsure the economics work: price, packaging, or willingness to pay.
How: Quote the price for real, or gate a feature behind a "buy" that leads to a waitlist, and count who proceeds vs. balks.
Clear signal: Enough prospects clear the price without negotiating toward free — measured against the conversion the model needs.
Days
Asking instead of charging. "How much would you pay?" is fiction; a checkout button — even a fake one — is data.
Data mining
Answering the assumption from data you already have — product analytics, support tickets, sales notes, search logs — before running any new test.
When: Always first. The cheapest experiment is the one you don’t run because the answer is already in your logs.
How: Frame the assumption as a query against existing behaviour and look for the pattern that would confirm or refute it.
Clear signal: A clear, quantified pattern in real historical behaviour — strong enough that a fresh test wouldn’t change the decision.
Hours·also tests any type
Seeing what you want. Existing data is riddled with survivorship and selection bias — state what would refute you before you go looking.
Choosing between them
Four rules for picking the right experiment — and not over-testing.
Start with the data you already have
Before you run anything, check whether analytics, tickets, or past interviews already answer it. The best experiment is often the one you don’t need.
Match the test to the doubt
A desirability doubt needs customers; a feasibility doubt needs a spike; a viability doubt needs a price. Running the wrong type is how you get a confident answer to a question you weren’t asking.
Pick the cheapest that still isolates one belief
Smaller is better only until the test stops giving a clean signal. If two beliefs ride on one experiment, you won’t know which one moved the result.
Prefer behaviour over opinion
Connecting a repo beats "sounds useful"; a checkout click beats "I’d pay for that". Every type has a behavioural version — reach for it.
And the meta-pitfall: running experiments to feel diligent. If a result couldn’t change what you do next, you’re not testing — you’re collecting reassurance. Every experiment should have a decision waiting on the other side of it.
Where the results live
An experiment is only worth running if the answer sticks. In an Opportunity Solution Tree, the assumptions behind each solution are first-class nodes — and in Outcomify they carry a status (untested, validated, invalidated) and the evidence behind it: the fake-door number, the interview quote, the spike’s benchmark. Six months on, "we tested that" is a link, not a memory — and the next team doesn't re-run your experiment by accident.
Frequently Asked Questions
Common questions about product experiments
Keep exploring
How to Test Product Assumptions
The framework these experiments serve: surface, map by importance × evidence, test riskiest-first.
Riskiest Assumption Examples
16 real assumptions, each paired with the experiment that would settle it.
Customer Interviews to Opportunities
A deeper look at the interview — the experiment that also feeds your opportunity space.
Turn experiments into evidence that lasts
Outcomify keeps every assumption — and the result of every test — on the tree, with a status and the evidence attached, so your team always knows what’s been proven and what’s still faith.