Decide What Would Count as Working Before You Build It

Somebody drags a card into the Done column and posts in the team channel that it shipped. A few people react with a party emoji, and nobody in the thread asks what the work was for.

Medical journals have a rule for that question. Since 2005, the journals that follow the International Committee of Medical Journal Editors have considered a trial for publication only if, before its first patient joined, it registered what it would measure. In 2015 a team at Oxford checked 67 trials from five of those journals against their registrations. In 58 of them, the paper reported something different from what the trial had registered.

Look for the equivalent of that registration on the last thing your team shipped.

A setup flow goes to Done

Meridian is the fictional composite company I've used throughout The Operator's Guide to Product Leadership, the video series this essay accompanies. It has 40 people and 60 customers, and it sells shift scheduling software to logistics operators. Its roadmap promises one outcome for the second quarter: by the time it ends, a new customer with no IT department is running on their own inside a day. Meridian's strategy starts from the observation that a twenty-vehicle operator has no IT department, so the product has to work on the first day without an implementation project behind it. The team builds a self-serve setup flow that a new customer can finish without calling anyone, and it ships on time, as the quarter opens. The card goes to Done.

Josh Seiden, who wrote Outcomes Over Output, defines an outcome as "a change in human behavior that drives business results." The setup flow is Meridian's output. Measured, the roadmap's outcome is a new customer publishing a first schedule within a day of signing up without calling support, and nothing in the Done column can show whether that happened. In part two of the series, an audit of Meridian found that nobody was making its fourth decision, what outcome to expect and how to know. The Done column answered it by default.

Asking afterward

Suppose nobody at Meridian writes anything down. A month after the setup flow ships, the founder asks the head of product whether it worked, and she goes looking. More people have opened the setup page than before, and a customer said nice things about it on a call. She tells him it's working, and every number she gives him is true. The card stays in Done, and next quarter's roadmap gets planned without anything learned from this one.

A company that keeps shipping without learning what any of it did is what John Cutler calls a feature factory, in a 2016 list titled "12 Signs You're Working in a Feature Factory." His first sign is no measurement: "You have no idea if your work worked." His sixth is that product managers don't compare the benefit they expected with the benefit they got. "Developers have 'passing tests', but product managers do not." A developer's test is written before the code ships and says what passing means, and a product decision can ship with nothing of the kind.

Cutler's fix is a column after Done where the team checks what the work did. That gives the question a place on the board, but it still gets asked afterward, and a question asked afterward gets whatever answer the numbers allow.

What the trials showed

Before the journals' rule, a trial that measured several things could report whichever one came out well, and nobody reading the paper could tell. The registration made the switch visible, because a paper can be checked against it. The Oxford project, called COMPare, did the checking, and across its 67 trials it found 365 outcomes in the published papers that the registrations had never mentioned. The papers didn't say so.

Meridian's head of product was in the position of a trial with no registration. She chose which numbers to report after the results were known, so they were bound to look good.

The last line of the template

Barry O'Reilly published a template in 2013 that does the job of a registration for a product bet:

We believe this capability will result in this outcome. We will have confidence to proceed when we see a measurable signal.

Its last sentence, filled in before anyone builds anything, names the evidence that would count as success. Filled in with "customers will like it," it can be read as a yes whatever happens. The template doesn't ask for a number to beat, a date to check it on or a person to report it, and the page below adds those.

One page, written first

In part seven, Meridian wrote down its bar, the number that would count as enough, before the first customer call. The same move works before the first line of code. So run Meridian's second quarter again with one change: before anybody builds the setup flow, the head of product writes one page and attaches it to the card. It's a scoreboard, written before anyone knows the score.

The scoreboard, written before the build 1. The outcome: what a customer will do differently once the work exists. 2. The count: the behavior you'll count, and where the count comes from. 3. Today's number: the count as it stands before the work ships. 4. The bar: the number that would mean it worked. 5. The date: when you'll check the count against the bar. 6. The name: who reports the result, hit or miss.

Take an export button. Today, customers email support for a spreadsheet. The outcome is customers pulling the spreadsheet themselves, and the count is how many still email support for one.

Meridian's outcome is a new customer publishing a first schedule within a day of signing up without calling support, so a customer who called doesn't count, however the call went. A customer can tell the head of product the setup was easy after calling support to get through it. She counts the call, and so would I.

Without today's number, you're judging the result against a guess. Meridian signs about two new customers a month, so a quarter brings six. Last quarter, one of the six published a first schedule within a day without calling anyone. The bar the head of product writes is four of the next six, which would mean most new customers are running on their own inside a day.

In a 2019 talk, Cutler's advice was to make a promise small enough to keep, like revisiting each product decision a month or two after you make it. The head of product's date is the end of the quarter, and the name on the result is hers. Without a date, nothing schedules the check. With her name on it, reporting the result is her job, whatever the result turns out to be.

I put that page together from the people I've quoted. The trials show that writing the outcome down first catches a switched measure, and whether the page makes a product team decide better is untested.

Six customers a quarter

A company with millions of users could test the setup flow directly with a split test: half the new customers get the new setup, half get the old one, and you compare what they do. Ronny Kohavi ran experimentation at Microsoft and now teaches it in paid courses, and he says a product without enough users may not be able to run a split test at all. Six customers can't settle anything statistically. They can tell the head of product whether she hit the number she wrote before she looked, and with only six she can read every support call they make.

Most bets miss

Writing the bar first also shows you how often your bets miss. In a 2017 Harvard Business Review article, Kohavi reported that about a third of the ideas tested at Microsoft improved the metric they were designed to improve, and that at Google and Bing only 10 to 20 percent of experiments did. Those are companies running thousands of tests a year. In a 2024 talk, his advice to everyone else was to expect a rate nearer 10 or 15 percent than 30 or 40.

If he's right about everyone else, a scoreboard that tells the truth will be mostly misses, and a team whose every launch worked is probably grading after the fact. Google's own grading makes room for both kinds of result. For ambitious goals, its guide says that someone who always scores a perfect 1.0 isn't aiming high enough. For goals a team commits to, its internal playbook, which John Doerr later published, expects a 1.0, and a miss has to be explained.

Reporting a miss

A scoreboard that is mostly misses works only if a miss is safe to report. In a study published in 1996, Amy Edmondson looked at medication errors in eight hospital units. The units were part of a larger Harvard study of drug errors: investigators read charts and visited each unit, and staff could report errors confidentially. She expected the best-led units to have the fewest errors on record. They had the most. Her own reading is that the better teams "probably don't make more mistakes, but they are more able to discuss mistakes," and with eight units her data couldn't separate making more mistakes from reporting more of them.

A product team whose scoreboard never shows a miss has the same problem: its numbers can't tell you whether the team misses less or reports less. John Allspaw made the matching argument about outages at Etsy. If an engineer whose change brings the site down can describe exactly what happened without being punished, everyone else learns how the site came down, and if she's punished, the details stop coming. Applying his no-blame rule to a product bet that missed is my own extension, and I make it because everyone in the room watches what happens to the first person who reports a miss.

Three of six

The quarter ends, and the head of product pulls the card back out of Done. Three of the six new customers published a first schedule within a day without calling support. The bar was four, so the setup flow missed. Last quarter it was one, so three is triple, and had she picked the measure after the result, the tripling would have been the headline.

At the quarterly review she puts the scoreboard up for the founder and reads the bar, four of six, and then the result, three. The founder's first question is whether the tripling counts for anything. It counts, she tells him, and it's still a miss. Then she tells him what the support calls showed: every customer who called got stuck at the same step, setting up overtime rules. The founder asks what she wants to try next, and templates for the common overtime rules go on a new card with a scoreboard of their own.

The engineers who built the setup flow hear their work called a miss in a meeting where nothing happens to anyone for it. The roadmap still owes the same outcome.

A miss can also mean the measure was wrong. If a day turned out to be too short for an operator with two warehouses, the next card's scoreboard would say two days and why, and the setup flow would still count as a miss, because the bar on its own scoreboard was four.

Your next card

Before your team starts its next card, write down the number that would mean the work did its job.

Dave Saunders is the founder of Base Reality Group and a Fractional CPO for product companies. He was a founder and operator at Galen Robotics, where the surgical-robotics platform earned FDA De Novo authorization in 2023, and he managed a 35-patent portfolio licensed from Johns Hopkins. He wrote Founders Who Finish and publishes The Build. More about Dave →