Velo Blog

Tag: engineering

  • Why our retros stopped finding the real problem

    Why our retros stopped finding the real problem

    The Friday retro ritual

    Every team we have worked on runs the same shape of retro. Sixty minutes on a Friday, three columns in a Miro board, sticky notes for what went well, what went poorly, and what to try next. Someone dot-votes. Someone else copies the top three items into a Linear ticket that nobody opens again. We used to run it that way too.

    In Q2 of last year, we shipped a release that took down billing for four hours. The retro landed the following Friday. The dot-vote surfaced “unclear on-call handoff” as the top item. We wrote a Linear ticket to rewrite the on-call runbook. Six weeks later, we shipped a different release that broke webhook delivery for two hours. The retro found the same category of problem, worded slightly differently, and produced another ticket that also went nowhere.

    The on-call runbook was fine. The problem sat upstream of it, and nobody in the room was willing to say so on a Friday afternoon with the person who owned the decision sitting three seats away.

    Why the standard retro fails

    Retros as most teams run them optimise for social comfort, not for truth. Three failure modes we kept hitting:

    • Recency bias. The team remembers Thursday’s deploy noise, not Monday’s design decision that set the deploy up to fail.
    • Consensus bias. Dot-voting rewards items several people already agree on, which selects for symptoms over root causes. Root causes are usually held by one or two people who saw them early and stayed quiet.
    • Performance bias. A live meeting is a stage. People tell the version of the story that protects the relationship, not the version that would help the next team.

    The billing incident showed us all three. The engineer who had raised a concern about the migration plan two weeks earlier did not repeat that concern in the room. Nobody wanted to spend the last hour of the week relitigating a decision that felt settled.

    The retro found something true and small. It missed the something true and large, because the format could not hold it.

    What we do instead

    We replaced the Friday retro with three artefacts, spread across the week. None of them takes more time than the meeting they replaced. We have run this shape for eleven months across two product squads and one platform squad.

    1. A written pre-mortem, filed on Wednesday

    Whoever owned the incident, feature, or sprint outcome writes a one-page pre-mortem in Notion. It is not a report of what happened. It is a written attempt to answer one question: if this failure repeats in six months, what will the story be? The author writes it alone, without review, and posts it in the squad Slack channel by end of Wednesday. It is time-boxed to forty-five minutes. Long documents mean somebody is hiding.

    2. A two-question survey, sent Thursday morning

    Every person on the squad, plus two adjacent stakeholders (usually a designer and a customer support lead), gets a Google Form with two questions:

    1. What did you see, hear, or think during this work that you did not say out loud?
    2. If you had a private ten-minute conversation with the person most responsible for the outcome, what would you ask?

    Answers are anonymised by the facilitator and pasted into the Notion doc under the pre-mortem. Response rate sits above ninety percent because the questions are specific and the form takes under five minutes.

    3. One blameless conversation, Monday at 10am

    The squad meets for thirty minutes on Monday. The pre-mortem and the survey answers are already in the room. The facilitator, who is not the tech lead, reads three or four survey answers aloud and asks the author of the pre-mortem to respond to them. No sticky notes. No dot-voting. No action items produced in the meeting itself. Proposals get added to the Notion doc during the following twenty-four hours, once people have had time to think.

    What changed

    The billing incident was one of eight retros we ran through the old format. The webhook incident was the ninth. Between month four and month eleven of the new format, we ran six retros across incidents of similar severity. Two produced Linear tickets that closed within a sprint. Three produced changes to how we scope Datadog dashboards before we ship, not after. One produced a decision to stop building a feature that two engineers privately thought would not land, and had not raised in a Friday meeting.

    The change is not that we find more problems. It is that the problems we find are the ones that matter. Three shifts explain most of it:

    • Writing before speaking gives people room to admit things they would not admit in a room.
    • Splitting the process across three days lets recency bias fade.
    • Removing the ritual of “action items produced in the meeting” removes the pressure to produce something visible, which is what pushes teams toward the easy, wrong answer.

    What we still get wrong

    The Monday conversation is fragile. If the facilitator lets it become a debate about the pre-mortem’s conclusions, it collapses back into the old format. We have had two of those in the last year. Both times, the survey answers that mattered most did not get read aloud, and the meeting ended with everyone agreeing on a symptom.

    We have also not solved the problem of what to do when the person most responsible for the outcome is the tech lead running the process. We rotate facilitation to a peer squad’s engineer in those cases, but it is not a clean answer.

    The retro, as most teams run it, is a meeting that produces the feeling of learning without the substance of it. If your Linear board carries three open “improve on-call handoff” tickets from three different retros, that is the signal.

  • The quarter we shipped no features

    The quarter we shipped no features

    Last October, three days after our Q4 planning offsite, we made a decision that felt reckless at the time. We were going to spend the entire quarter without shipping a single new feature. No new modules, no new integrations, no new dashboards. Only bugs, docs, and the internal tools our engineers had been asking for since spring.

    Our head of sales, Mira, found out on a Monday morning during our weekly go-to-market sync. She went quiet for about eight seconds, then asked whether we were serious. We were.

    What sales was worried about

    Mira had four deals in the pipeline that hinged on a specific promise: a Snowflake connector we had been talking about since June. Two of those deals were mid-market, one was a renewal expansion, and one was a competitive replacement worth around 180k in annual contract value. She pulled up the deal notes in Notion and walked us through each one.

    The fear was reasonable. If we froze features for 90 days, three things could happen:

    • Prospects would walk to competitors who kept shipping.
    • Existing customers waiting on requested features would churn at renewal.
    • The sales team would lose narrative ammunition on discovery calls.

    We agreed to review the freeze monthly. If any of those signals showed up in the data, we would call it off. Mira asked us to write down what “showed up in the data” meant, so we did: net revenue retention below 108%, gross churn above 1.4% monthly, or two consecutive weeks of stalled pipeline movement on flagged deals.

    What we shipped instead

    The engineering team split into three squads. One squad, which we called Fixit, worked exclusively through Linear tickets tagged with the “customer-reported” label. Another squad, Docs, sat with our support lead every Tuesday to identify the top ten most-hit help center pages and rewrite them. The third squad, Tooling, built the internal admin console engineers had been begging for.

    By week six we had closed 247 bugs, some of which had been open for over a year. The Datadog dashboard we cared about, the one tracking p95 API latency, dropped from 840ms to 310ms after two engineers rewrote a query planner in the reporting service. Our support team went from 34 open Zendesk tickets on any given Friday to 9.

    The internal admin console was the surprise. Before Q4, resolving a customer-reported billing issue took an engineer about 40 minutes: pull data from three tables, reconcile in a Google Sheet, patch, verify. After the tooling squad shipped the console, our support engineers were doing the same work in under 4 minutes. They did not need to page anyone.

    By the end of week ten, our on-call rotation had gone from one incident per shift to one incident every six shifts. Two engineers told me they were sleeping better. One of them had been talking about leaving.

    What happened to churn

    Here is the part nobody predicted. Gross churn went down. Not by a huge amount, but measurably: from 1.2% monthly at the start of Q4 to 0.7% by December. Net revenue retention held at 114%.

    Mira’s Snowflake deals: three of the four closed anyway. The connector question came up on discovery calls, and the answer we gave, which was that we were spending the quarter on reliability instead of new surface area, played better than we expected. One of the buyers, a VP of data at a healthcare company, told us he had never heard a vendor say that out loud. He signed in November.

    Two effects we did not model

    First, our NPS moved from 42 to 51. We got unsolicited notes in Slack from customer success managers whose accounts had stopped filing tickets. Second, our engineering hiring pipeline got healthier. Three candidates in December mentioned during their onsite loop that they had read our internal writeup about the freeze and wanted to work at a place that took reliability seriously.

    What we would do differently

    We got lucky on a few things and would not repeat every choice.

    1. We underestimated how disorienting the freeze would feel to product managers. Two of them felt sidelined for six weeks before we figured out how to give them meaningful work reviewing customer feedback and shaping the Q1 roadmap.
    2. We should have communicated the freeze to customers on day one, not week three. When we finally sent the note explaining what we were doing, the response was overwhelmingly positive. We could have banked that goodwill earlier.
    3. We did not set clear exit criteria beyond the churn and NRR thresholds. When Q1 planning arrived, some of us wanted to extend the freeze another month, and we did not have a decision framework for that conversation.

    We are not going to do this every quarter. Growth still matters, and a company that only fixes bugs is a company that gets displaced. But we now know the shape of what a deliberate pause looks like, what it costs, and what it returns. Next time we consider one, the conversation will be shorter, and the fear in the room will be smaller.

  • Velocity is a lie detector not a speedometer

    Velocity is a lie detector not a speedometer

    Every quarter we watch another engineering team roll a velocity chart into a review deck, gesture at the bars trending up and to the right, and declare progress. We used to do it too. Then we ran a small experiment on our own Velo delivery team: we hid the velocity number from three squads for a full quarter and gave them cycle time instead. Two of the three squads shipped more features. All three reported that planning felt less theatrical.

    This is not an argument against measurement. It is an argument against a specific metric that has quietly stopped telling us what we think it tells us.

    Velocity measures the ruler, not the road

    Story points are a ruler the team invented. When we grade a team on how many units of their own ruler they produce per sprint, we should not be surprised when the ruler starts stretching. We have watched this happen on our own boards. A ticket that would have been a 3 in April becomes a 5 by July. Nobody lies. Everyone remembers “that thing that turned out harder than expected,” and the estimate drifts up. The chart climbs. The output does not.

    Goodhart’s law shows up in the standup:

    When a measure becomes a target, it ceases to be a good measure. Velocity is the most polite example of this rule we have found in software.

    The other failure mode is subtler. Velocity averages hide the shape of the work. A team can hit 42 points every sprint for six sprints and be quietly falling apart, because 40 of those points come from a single engineer who is one Slack DM away from resigning. The bar chart cannot see that. Cycle time can.

    Cycle time is boring, which is the point

    Cycle time is the elapsed clock between “in progress” and “done.” It is boring because it measures reality rather than an estimate. We cannot inflate it by talking about it in a Wednesday grooming session. It refuses to care about our narrative.

    Here is the comparison we now put in front of every engineering lead we hire:

    • Unit. Velocity uses story points, a team invented currency. Cycle time uses hours or days, a currency everyone shares.
    • Gameable by. Velocity is inflated by re estimation, ticket splitting, and status theatre. Cycle time is inflated only by shipping faster, which is what we wanted.
    • Sensitive to. Velocity is sensitive to who is in the room during planning poker. Cycle time is sensitive to review queues, environment flakiness, and handoffs, the things that slow us down.
    • What it hides. Velocity hides bottlenecks behind an average. Cycle time exposes them by widening the tail of the distribution.
    • Actionable signal. A dropping velocity prompts a debate about commitment. A widening cycle time prompts a debate about the pull request that has been open since Tuesday.

    What we changed on our own board

    We run Linear for tickets and pipe every state transition into a Datadog dashboard. Once a week, on Thursday afternoon, we look at three numbers together:

    1. Median cycle time for tickets closed that week.
    2. The 90th percentile of the same distribution.
    3. The count of pull requests older than 48 hours.

    We deliberately do not look at velocity anymore. When a stakeholder in Notion asks “how much did we ship this quarter,” we answer with a count of shipped tickets and a link to the changelog. If they push, we show the cycle time trend. Nobody has pushed twice.

    The 90th percentile is the metric that changed our behaviour the most. Medians are polite. They hide the ticket that sat in code review for eleven days because the reviewer was on parental leave and nobody rerouted it. The 90th percentile has forced us to build a bot in Slack that pings the review channel every morning at 9:15 with any pull request older than a day. Our median moved a little. Our tail moved a lot.

    The objections we still hear

    Two objections show up in every conversation about this, and both deserve a real answer.

    The first is that cycle time punishes big work. A refactor that takes two weeks will show up as a fat cycle time number and drag the median. Our response: split the refactor. Not into fake tickets that ship in isolation, but into shippable, reversible steps. If a piece of work cannot be split, that is itself a finding, and the fat number is telling us something true.

    The second objection is that cycle time can also be gamed. Engineers can open tickets late, close them early, or keep everything in draft. Fair. We have seen all three. The difference is that these games are visible in the version control log and in the Linear activity feed. Velocity inflation is invisible, because the ruler itself is invisible. A gamed cycle time becomes a conversation. A gamed velocity becomes a slide.

    The metric we would keep if we could only keep one

    If a new engineering leader joined tomorrow and asked which single number to track, we would not hesitate. Track the 90th percentile of cycle time, week over week, and set a target for the tail rather than the average. That number is close enough to reality that people can argue about it usefully, and far enough from planning theatre that it does not warp under pressure. Velocity charts belong in a museum next to lines of code per day. We stopped drawing them and started drawing something harder to fake.

  • What done means for a task on our team

    What done means for a task on our team

    Every team we worked on before Velo had a definition of done pinned to a wiki page nobody read. Ours did too, until a Wednesday standup in March when Priya asked whether the invoice retry work was finished, and four engineers gave four different answers. That morning cost us a customer refund and a two hour incident review. We decided the Notion page was not the problem. The definition was.

    Three tries that did not stick

    Our first attempt was a paragraph in Notion titled “shipping standards” that said tasks should be “merged, tested, and reviewed.” It read fine on the page. In practice, “tested” meant whatever the author felt like: a unit test, a manual walkthrough, or nothing if the diff was under twenty lines. We shipped a race condition in the billing worker three weeks later because the author had run the change against a fresh database and assumed that counted.

    The second attempt was a Linear checklist template with nine items. Everyone checked every box, because the boxes were reported by the author and the reviewer had no way to verify half of them without opening five other tabs. The checklist became a ritual, then a joke, then a template we quietly stopped applying to new tickets.

    The third attempt was strict: a task was done when a designated QA engineer signed off in a Slack thread. This lasted eleven days. Our QA lead, Ruth, went on holiday, and the queue backed up to forty two tickets. When she came back, half the context was gone and she had to re verify work from memory. We had traded ambiguity for a bottleneck.

    The four criteria we settled on

    After the third failure, we spent a Friday afternoon working through what we needed the definition to do. It had to be verifiable by someone other than the author, it had to survive one person being out, and it had to answer the question Priya asked in March without a debate. We landed on four criteria, in this order:

    • Works: the change does what the ticket says, verified against the acceptance criteria written before the branch was cut. If those criteria were vague, that gets fixed before the ticket moves to review, not after.
    • Tested: automated coverage exists for the new behavior, and the tests fail without the change. The reviewer runs the suite locally or points at a green CI badge tied to the merge commit.
    • Deployed: the change is live in production, not staging, not behind a flag that has never been flipped on for a real user. If the work sits behind a flag, done waits until the flag is on for the intended audience.
    • Observed: a human has confirmed the change behaves as expected in production, using logs, a Datadog dashboard, or a real user event. Not a synthetic ping. A trace of the feature being used, or a metric moving in the direction we predicted.

    The order matters. If “works” is unclear, testing the wrong thing is worse than not testing. If we skip “deployed” and call something done at merge, we hide half our incidents in the gap between main and production.

    The compromise on observed

    Observed was the criterion that almost killed the whole definition. Half the team pointed out, correctly, that internal only changes have no production traffic to watch. A new admin report, a migration script, an internal CLI: none of these throw off metrics on the customer dashboards we use for observability. Waiting for a real user event on an internal tool would mean waiting forever, or fabricating one.

    We debated dropping the criterion for internal work. We tried, for a sprint. Two internal tools broke silently and we found out from a support agent who could not load the refunds page. The criterion needed to survive.

    The compromise: for internal only changes, observed means the author or a teammate has used the feature in production for its intended purpose, with a Loom or a screenshot posted to the ticket. Not tested it. Used it. If the ticket is a migration, the observation is the query result after the migration ran. If it is a CLI, it is the terminal output from a real invocation on the real database.

    The distinction we care about is between “I believe this works” and “this has done its job for a real person, once.” The Loom feels heavy the first time. It stops feeling heavy the second time somebody catches a broken admin page before a customer does.

    How the four criteria show up in our week

    Every ticket in Linear now has four checkboxes matching the criteria. The author checks the first three. The reviewer, or on internal changes any teammate, checks observed and pastes the evidence. Our Monday planning meeting starts by pulling the list of tickets marked done in the last week and skimming the observation links. It takes eight minutes. In the six months since we adopted this, we have had two rollback situations that a proper observation caught before the on call engineer noticed. We have also had one case where the observation link was a screenshot of the wrong environment, which is a different problem, and one we are still working on.

    We do not think this definition is universal. It is what our team of eleven engineers, on a codebase with sixteen deploys a week, needs to keep the wiki page honest. If the shape of the team changes, we expect the definition to change with it.