Velo Blog

Tag: process

  • Why our retros stopped finding the real problem

    Why our retros stopped finding the real problem

    The Friday retro ritual

    Every team we have worked on runs the same shape of retro. Sixty minutes on a Friday, three columns in a Miro board, sticky notes for what went well, what went poorly, and what to try next. Someone dot-votes. Someone else copies the top three items into a Linear ticket that nobody opens again. We used to run it that way too.

    In Q2 of last year, we shipped a release that took down billing for four hours. The retro landed the following Friday. The dot-vote surfaced “unclear on-call handoff” as the top item. We wrote a Linear ticket to rewrite the on-call runbook. Six weeks later, we shipped a different release that broke webhook delivery for two hours. The retro found the same category of problem, worded slightly differently, and produced another ticket that also went nowhere.

    The on-call runbook was fine. The problem sat upstream of it, and nobody in the room was willing to say so on a Friday afternoon with the person who owned the decision sitting three seats away.

    Why the standard retro fails

    Retros as most teams run them optimise for social comfort, not for truth. Three failure modes we kept hitting:

    • Recency bias. The team remembers Thursday’s deploy noise, not Monday’s design decision that set the deploy up to fail.
    • Consensus bias. Dot-voting rewards items several people already agree on, which selects for symptoms over root causes. Root causes are usually held by one or two people who saw them early and stayed quiet.
    • Performance bias. A live meeting is a stage. People tell the version of the story that protects the relationship, not the version that would help the next team.

    The billing incident showed us all three. The engineer who had raised a concern about the migration plan two weeks earlier did not repeat that concern in the room. Nobody wanted to spend the last hour of the week relitigating a decision that felt settled.

    The retro found something true and small. It missed the something true and large, because the format could not hold it.

    What we do instead

    We replaced the Friday retro with three artefacts, spread across the week. None of them takes more time than the meeting they replaced. We have run this shape for eleven months across two product squads and one platform squad.

    1. A written pre-mortem, filed on Wednesday

    Whoever owned the incident, feature, or sprint outcome writes a one-page pre-mortem in Notion. It is not a report of what happened. It is a written attempt to answer one question: if this failure repeats in six months, what will the story be? The author writes it alone, without review, and posts it in the squad Slack channel by end of Wednesday. It is time-boxed to forty-five minutes. Long documents mean somebody is hiding.

    2. A two-question survey, sent Thursday morning

    Every person on the squad, plus two adjacent stakeholders (usually a designer and a customer support lead), gets a Google Form with two questions:

    1. What did you see, hear, or think during this work that you did not say out loud?
    2. If you had a private ten-minute conversation with the person most responsible for the outcome, what would you ask?

    Answers are anonymised by the facilitator and pasted into the Notion doc under the pre-mortem. Response rate sits above ninety percent because the questions are specific and the form takes under five minutes.

    3. One blameless conversation, Monday at 10am

    The squad meets for thirty minutes on Monday. The pre-mortem and the survey answers are already in the room. The facilitator, who is not the tech lead, reads three or four survey answers aloud and asks the author of the pre-mortem to respond to them. No sticky notes. No dot-voting. No action items produced in the meeting itself. Proposals get added to the Notion doc during the following twenty-four hours, once people have had time to think.

    What changed

    The billing incident was one of eight retros we ran through the old format. The webhook incident was the ninth. Between month four and month eleven of the new format, we ran six retros across incidents of similar severity. Two produced Linear tickets that closed within a sprint. Three produced changes to how we scope Datadog dashboards before we ship, not after. One produced a decision to stop building a feature that two engineers privately thought would not land, and had not raised in a Friday meeting.

    The change is not that we find more problems. It is that the problems we find are the ones that matter. Three shifts explain most of it:

    • Writing before speaking gives people room to admit things they would not admit in a room.
    • Splitting the process across three days lets recency bias fade.
    • Removing the ritual of “action items produced in the meeting” removes the pressure to produce something visible, which is what pushes teams toward the easy, wrong answer.

    What we still get wrong

    The Monday conversation is fragile. If the facilitator lets it become a debate about the pre-mortem’s conclusions, it collapses back into the old format. We have had two of those in the last year. Both times, the survey answers that mattered most did not get read aloud, and the meeting ended with everyone agreeing on a symptom.

    We have also not solved the problem of what to do when the person most responsible for the outcome is the tech lead running the process. We rotate facilitation to a peer squad’s engineer in those cases, but it is not a clean answer.

    The retro, as most teams run it, is a meeting that produces the feeling of learning without the substance of it. If your Linear board carries three open “improve on-call handoff” tickets from three different retros, that is the signal.

  • The quarter we shipped no features

    The quarter we shipped no features

    Last October, three days after our Q4 planning offsite, we made a decision that felt reckless at the time. We were going to spend the entire quarter without shipping a single new feature. No new modules, no new integrations, no new dashboards. Only bugs, docs, and the internal tools our engineers had been asking for since spring.

    Our head of sales, Mira, found out on a Monday morning during our weekly go-to-market sync. She went quiet for about eight seconds, then asked whether we were serious. We were.

    What sales was worried about

    Mira had four deals in the pipeline that hinged on a specific promise: a Snowflake connector we had been talking about since June. Two of those deals were mid-market, one was a renewal expansion, and one was a competitive replacement worth around 180k in annual contract value. She pulled up the deal notes in Notion and walked us through each one.

    The fear was reasonable. If we froze features for 90 days, three things could happen:

    • Prospects would walk to competitors who kept shipping.
    • Existing customers waiting on requested features would churn at renewal.
    • The sales team would lose narrative ammunition on discovery calls.

    We agreed to review the freeze monthly. If any of those signals showed up in the data, we would call it off. Mira asked us to write down what “showed up in the data” meant, so we did: net revenue retention below 108%, gross churn above 1.4% monthly, or two consecutive weeks of stalled pipeline movement on flagged deals.

    What we shipped instead

    The engineering team split into three squads. One squad, which we called Fixit, worked exclusively through Linear tickets tagged with the “customer-reported” label. Another squad, Docs, sat with our support lead every Tuesday to identify the top ten most-hit help center pages and rewrite them. The third squad, Tooling, built the internal admin console engineers had been begging for.

    By week six we had closed 247 bugs, some of which had been open for over a year. The Datadog dashboard we cared about, the one tracking p95 API latency, dropped from 840ms to 310ms after two engineers rewrote a query planner in the reporting service. Our support team went from 34 open Zendesk tickets on any given Friday to 9.

    The internal admin console was the surprise. Before Q4, resolving a customer-reported billing issue took an engineer about 40 minutes: pull data from three tables, reconcile in a Google Sheet, patch, verify. After the tooling squad shipped the console, our support engineers were doing the same work in under 4 minutes. They did not need to page anyone.

    By the end of week ten, our on-call rotation had gone from one incident per shift to one incident every six shifts. Two engineers told me they were sleeping better. One of them had been talking about leaving.

    What happened to churn

    Here is the part nobody predicted. Gross churn went down. Not by a huge amount, but measurably: from 1.2% monthly at the start of Q4 to 0.7% by December. Net revenue retention held at 114%.

    Mira’s Snowflake deals: three of the four closed anyway. The connector question came up on discovery calls, and the answer we gave, which was that we were spending the quarter on reliability instead of new surface area, played better than we expected. One of the buyers, a VP of data at a healthcare company, told us he had never heard a vendor say that out loud. He signed in November.

    Two effects we did not model

    First, our NPS moved from 42 to 51. We got unsolicited notes in Slack from customer success managers whose accounts had stopped filing tickets. Second, our engineering hiring pipeline got healthier. Three candidates in December mentioned during their onsite loop that they had read our internal writeup about the freeze and wanted to work at a place that took reliability seriously.

    What we would do differently

    We got lucky on a few things and would not repeat every choice.

    1. We underestimated how disorienting the freeze would feel to product managers. Two of them felt sidelined for six weeks before we figured out how to give them meaningful work reviewing customer feedback and shaping the Q1 roadmap.
    2. We should have communicated the freeze to customers on day one, not week three. When we finally sent the note explaining what we were doing, the response was overwhelmingly positive. We could have banked that goodwill earlier.
    3. We did not set clear exit criteria beyond the churn and NRR thresholds. When Q1 planning arrived, some of us wanted to extend the freeze another month, and we did not have a decision framework for that conversation.

    We are not going to do this every quarter. Growth still matters, and a company that only fixes bugs is a company that gets displaced. But we now know the shape of what a deliberate pause looks like, what it costs, and what it returns. Next time we consider one, the conversation will be shorter, and the fear in the room will be smaller.

  • How we unbreak a Wednesday sprint without cancelling it

    How we unbreak a Wednesday sprint without cancelling it

    Every team we know has had that sprint. Monday standup looked fine. Tuesday morning a payment webhook started dropping in staging, one engineer went out sick, and the design review pushed the checkout redesign back by two days. By Wednesday afternoon the burndown on our Linear board was flat, six tickets deep, and the sprint goal read like fiction.

    We used to cancel sprints in this situation. We stopped doing that around a year ago. Cancelling costs us the retrospective, the sense of finishing something, and the muscle memory of shipping on a cadence. What replaced it is a Wednesday recovery ritual that we run in about forty minutes.

    The Wednesday triage, not another standup

    We block thirty minutes on Wednesday at 2pm called “Sprint check”. It only fires when the burndown deviates more than twenty percent from the ideal line, which our Datadog dashboard flags in a Slack channel called #eng-signals. If the sprint is on track, the meeting is cancelled by 1:45pm and nobody joins.

    When it does fire, three people attend: the engineering lead for the squad, the product manager, and whoever picked up the on-call pager that week. No designers, no wider group. The point is a fast, honest read on the remaining ten working hours across four engineers.

    The question we ask is not “can we still finish everything?” It is “what one thing, if shipped by Friday, would make this sprint worth having run?”

    That one question forces a decision that the daily standup rarely produces. Standups report status. Wednesday triage rewrites the plan.

    Cut scope in the second half

    Once the anchor ticket is named, we walk the remaining Linear tickets and sort them into three buckets. We do this on a shared Notion page titled “Sprint 47 midweek reset” with three headings and drag ticket links under each.

    1. Ship this week. The anchor ticket and anything on its critical path. Usually two or three tickets.
    2. Defer to next sprint. Work that is not blocking anyone. Move it back to the backlog with a comment explaining why.
    3. Drop, do not defer. Tickets that felt urgent on Monday and no longer do. These get closed with a short note. If they matter again, someone will reopen them.

    The third bucket is the one that saves us. Roughly a quarter of what we plan on Monday gets dropped rather than deferred, and none of it has come back to bite us in the four sprints we have tracked this pattern.

    Slice the anchor, do not shrink it

    The highest value ticket is where teams tend to lie to themselves. On Wednesday we do not promise a smaller version of the same scope. We split the ticket into two Linear issues, and the parent becomes an epic.

    Take our checkout redesign example. The original ticket read: “Ship the redesigned checkout flow with saved cards, address autofill, and Apple Pay.” By Wednesday it was clear we had ten hours of engineering work left and about twenty hours of scope. The split looked like this:

    • PAY-412: Ship the redesigned checkout behind a feature flag, five percent rollout, saved cards only. Owner: Priya. Estimate: eight hours.
    • PAY-413: Address autofill and Apple Pay under the same flag. Owner: unassigned. Moved to next sprint.

    The slice we ship on Friday is a real, running thing in production, even if it sits behind a flag at five percent. Next sprint we widen it. What we avoid is the trap of promising the whole checkout by Friday, then delivering nothing and calling it a spike.

    The rolling change log

    Every scope change goes into a single Notion page we call the Sprint Ledger. One page per sprint, appended to as things move. Each entry has a timestamp, the ticket ID, what changed, and one line of why.

    Wed 14:32  PAY-401  Dropped. Duplicated by PAY-397 already in progress.
Wed 14:35  PAY-412  Split from PAY-388. Anchor for the week.
Wed 14:41  ONB-215  Deferred. Blocked on design; no unblock this week.

    The ledger takes about six minutes to fill in during triage. It is read twice: once by the wider squad on Wednesday afternoon, and once by the retro facilitator on the following Monday. Nobody hunts through Slack scrollback trying to remember what changed and why.

    What we get back

    Four things, measured over ten sprints since we started running this ritual:

    • We now finish about eighty percent of our stated sprint goal by Friday, up from around fifty percent when we would grind on the original plan or cancel outright.
    • Retros focus on cause, not blame. The ledger tells us what happened; we can talk about why.
    • Product managers push back less on Wednesday cuts, because the anchor is preserved and the ledger makes the trade visible.
    • On-call load in the second half of the sprint dropped, because we stopped shipping half finished work under time pressure.

    None of this requires new tooling. Linear, Notion, Slack, one recurring calendar block, and a rule about when it fires. The hardest part is not the process; it is the willingness on Wednesday afternoon to say out loud that Monday’s plan is no longer the plan, and to write down what replaced it before the day ends.

  • What done means for a task on our team

    What done means for a task on our team

    Every team we worked on before Velo had a definition of done pinned to a wiki page nobody read. Ours did too, until a Wednesday standup in March when Priya asked whether the invoice retry work was finished, and four engineers gave four different answers. That morning cost us a customer refund and a two hour incident review. We decided the Notion page was not the problem. The definition was.

    Three tries that did not stick

    Our first attempt was a paragraph in Notion titled “shipping standards” that said tasks should be “merged, tested, and reviewed.” It read fine on the page. In practice, “tested” meant whatever the author felt like: a unit test, a manual walkthrough, or nothing if the diff was under twenty lines. We shipped a race condition in the billing worker three weeks later because the author had run the change against a fresh database and assumed that counted.

    The second attempt was a Linear checklist template with nine items. Everyone checked every box, because the boxes were reported by the author and the reviewer had no way to verify half of them without opening five other tabs. The checklist became a ritual, then a joke, then a template we quietly stopped applying to new tickets.

    The third attempt was strict: a task was done when a designated QA engineer signed off in a Slack thread. This lasted eleven days. Our QA lead, Ruth, went on holiday, and the queue backed up to forty two tickets. When she came back, half the context was gone and she had to re verify work from memory. We had traded ambiguity for a bottleneck.

    The four criteria we settled on

    After the third failure, we spent a Friday afternoon working through what we needed the definition to do. It had to be verifiable by someone other than the author, it had to survive one person being out, and it had to answer the question Priya asked in March without a debate. We landed on four criteria, in this order:

    • Works: the change does what the ticket says, verified against the acceptance criteria written before the branch was cut. If those criteria were vague, that gets fixed before the ticket moves to review, not after.
    • Tested: automated coverage exists for the new behavior, and the tests fail without the change. The reviewer runs the suite locally or points at a green CI badge tied to the merge commit.
    • Deployed: the change is live in production, not staging, not behind a flag that has never been flipped on for a real user. If the work sits behind a flag, done waits until the flag is on for the intended audience.
    • Observed: a human has confirmed the change behaves as expected in production, using logs, a Datadog dashboard, or a real user event. Not a synthetic ping. A trace of the feature being used, or a metric moving in the direction we predicted.

    The order matters. If “works” is unclear, testing the wrong thing is worse than not testing. If we skip “deployed” and call something done at merge, we hide half our incidents in the gap between main and production.

    The compromise on observed

    Observed was the criterion that almost killed the whole definition. Half the team pointed out, correctly, that internal only changes have no production traffic to watch. A new admin report, a migration script, an internal CLI: none of these throw off metrics on the customer dashboards we use for observability. Waiting for a real user event on an internal tool would mean waiting forever, or fabricating one.

    We debated dropping the criterion for internal work. We tried, for a sprint. Two internal tools broke silently and we found out from a support agent who could not load the refunds page. The criterion needed to survive.

    The compromise: for internal only changes, observed means the author or a teammate has used the feature in production for its intended purpose, with a Loom or a screenshot posted to the ticket. Not tested it. Used it. If the ticket is a migration, the observation is the query result after the migration ran. If it is a CLI, it is the terminal output from a real invocation on the real database.

    The distinction we care about is between “I believe this works” and “this has done its job for a real person, once.” The Loom feels heavy the first time. It stops feeling heavy the second time somebody catches a broken admin page before a customer does.

    How the four criteria show up in our week

    Every ticket in Linear now has four checkboxes matching the criteria. The author checks the first three. The reviewer, or on internal changes any teammate, checks observed and pastes the evidence. Our Monday planning meeting starts by pulling the list of tickets marked done in the last week and skimming the observation links. It takes eight minutes. In the six months since we adopted this, we have had two rollback situations that a proper observation caught before the on call engineer noticed. We have also had one case where the observation link was a screenshot of the wrong environment, which is a different problem, and one we are still working on.

    We do not think this definition is universal. It is what our team of eleven engineers, on a codebase with sixteen deploys a week, needs to keep the wiki page honest. If the shape of the team changes, we expect the definition to change with it.