Tag: delivery

  • Why our retros stopped finding the real problem

    Why our retros stopped finding the real problem

    The Friday retro ritual

    Every team we have worked on runs the same shape of retro. Sixty minutes on a Friday, three columns in a Miro board, sticky notes for what went well, what went poorly, and what to try next. Someone dot-votes. Someone else copies the top three items into a Linear ticket that nobody opens again. We used to run it that way too.

    In Q2 of last year, we shipped a release that took down billing for four hours. The retro landed the following Friday. The dot-vote surfaced “unclear on-call handoff” as the top item. We wrote a Linear ticket to rewrite the on-call runbook. Six weeks later, we shipped a different release that broke webhook delivery for two hours. The retro found the same category of problem, worded slightly differently, and produced another ticket that also went nowhere.

    The on-call runbook was fine. The problem sat upstream of it, and nobody in the room was willing to say so on a Friday afternoon with the person who owned the decision sitting three seats away.

    Why the standard retro fails

    Retros as most teams run them optimise for social comfort, not for truth. Three failure modes we kept hitting:

    • Recency bias. The team remembers Thursday’s deploy noise, not Monday’s design decision that set the deploy up to fail.
    • Consensus bias. Dot-voting rewards items several people already agree on, which selects for symptoms over root causes. Root causes are usually held by one or two people who saw them early and stayed quiet.
    • Performance bias. A live meeting is a stage. People tell the version of the story that protects the relationship, not the version that would help the next team.

    The billing incident showed us all three. The engineer who had raised a concern about the migration plan two weeks earlier did not repeat that concern in the room. Nobody wanted to spend the last hour of the week relitigating a decision that felt settled.

    The retro found something true and small. It missed the something true and large, because the format could not hold it.

    What we do instead

    We replaced the Friday retro with three artefacts, spread across the week. None of them takes more time than the meeting they replaced. We have run this shape for eleven months across two product squads and one platform squad.

    1. A written pre-mortem, filed on Wednesday

    Whoever owned the incident, feature, or sprint outcome writes a one-page pre-mortem in Notion. It is not a report of what happened. It is a written attempt to answer one question: if this failure repeats in six months, what will the story be? The author writes it alone, without review, and posts it in the squad Slack channel by end of Wednesday. It is time-boxed to forty-five minutes. Long documents mean somebody is hiding.

    2. A two-question survey, sent Thursday morning

    Every person on the squad, plus two adjacent stakeholders (usually a designer and a customer support lead), gets a Google Form with two questions:

    1. What did you see, hear, or think during this work that you did not say out loud?
    2. If you had a private ten-minute conversation with the person most responsible for the outcome, what would you ask?

    Answers are anonymised by the facilitator and pasted into the Notion doc under the pre-mortem. Response rate sits above ninety percent because the questions are specific and the form takes under five minutes.

    3. One blameless conversation, Monday at 10am

    The squad meets for thirty minutes on Monday. The pre-mortem and the survey answers are already in the room. The facilitator, who is not the tech lead, reads three or four survey answers aloud and asks the author of the pre-mortem to respond to them. No sticky notes. No dot-voting. No action items produced in the meeting itself. Proposals get added to the Notion doc during the following twenty-four hours, once people have had time to think.

    What changed

    The billing incident was one of eight retros we ran through the old format. The webhook incident was the ninth. Between month four and month eleven of the new format, we ran six retros across incidents of similar severity. Two produced Linear tickets that closed within a sprint. Three produced changes to how we scope Datadog dashboards before we ship, not after. One produced a decision to stop building a feature that two engineers privately thought would not land, and had not raised in a Friday meeting.

    The change is not that we find more problems. It is that the problems we find are the ones that matter. Three shifts explain most of it:

    • Writing before speaking gives people room to admit things they would not admit in a room.
    • Splitting the process across three days lets recency bias fade.
    • Removing the ritual of “action items produced in the meeting” removes the pressure to produce something visible, which is what pushes teams toward the easy, wrong answer.

    What we still get wrong

    The Monday conversation is fragile. If the facilitator lets it become a debate about the pre-mortem’s conclusions, it collapses back into the old format. We have had two of those in the last year. Both times, the survey answers that mattered most did not get read aloud, and the meeting ended with everyone agreeing on a symptom.

    We have also not solved the problem of what to do when the person most responsible for the outcome is the tech lead running the process. We rotate facilitation to a peer squad’s engineer in those cases, but it is not a clean answer.

    The retro, as most teams run it, is a meeting that produces the feeling of learning without the substance of it. If your Linear board carries three open “improve on-call handoff” tickets from three different retros, that is the signal.

  • The five-day slice rule

    The five-day slice rule

    Every Monday at 10:15 our delivery lead opens Linear and runs a filter called Cycle Ready. If a ticket in the active cycle fails the filter, it gets flagged red and the ticket owner has until standup Tuesday to fix it or pull it. The filter checks one thing: can a single engineer start work Monday morning and demo something running by Friday afternoon.

    We call this the five-day slice rule. It has been the single biggest change to how our team ships since we moved off two-week sprints eighteen months ago.

    Why five days, not ten

    We tried ten-day cycles for most of 2024. Tickets came in sized small, medium, or large, and engineers estimated them in points. By day seven of a typical cycle, roughly a third of medium tickets were still labelled In Progress with no visible artifact. Reviewers had nothing to look at. QA had nothing to queue. The last three days always became a scramble.

    The pattern was consistent enough that we started tracking it in Datadog against our Linear webhook data. Tickets that produced no reviewable artifact by day three of a ten-day cycle had a 72 percent chance of slipping the cycle. Tickets that did produce something by day three slipped 8 percent of the time. The signal was loud.

    We cut the cycle in half and enforced a hard rule on entry: one person, Monday start, Friday demo. If it does not fit, split it before the cycle starts, not during.

    How we enforce the rule

    The rule lives in three places:

    • A Linear template with a required field called Friday demo artifact. Engineers cannot move a ticket into Ready for Cycle without filling it in. Sample entries: a PR merged behind a flag and hitting a staging endpoint; a Grafana panel showing p95 for the new route; a Loom of the empty state rendering with fixture data.
    • A pre-cycle review meeting on Friday afternoon called Slice Check. Twenty five minutes, four people: engineering manager, tech lead, product manager, delivery lead. We read the Friday demo artifact field for every candidate ticket. If anyone at the table cannot picture the demo, the ticket does not enter.
    • A Slack bot posting into the delivery channel every Wednesday at 4pm with the list of active-cycle tickets that have no PR opened and no draft artifact linked. The message pings the ticket owner directly.

    The bot is the piece that took the longest to trust. We tuned it for six weeks before people stopped arguing with it. The current heuristic: no draft PR, no Loom link, no Notion doc updated in the last 48 hours, and the ticket is past cycle midpoint. Three signals, one ping.

    What happens when a team pushes back

    The rule gets fought. Usually by whichever team is holding the largest piece of unsplit work. We have heard every version of the objection:

    You cannot split a database migration into five-day slices. The migration either runs or it does not.

    The auth rewrite is one atomic change. Splitting it means shipping something broken.

    We are being asked to do more planning work than shipping work.

    We take these seriously and we still hold the line. Every migration we have run in the last year has split. Every auth change has split. The planning cost is real and it front-loads. Our data shows the front-loaded planning cost is roughly 90 minutes per split ticket, and it saves an average of 6 hours of end-of-cycle scramble per unsplit ticket that slips.

    When a team insists a piece of work cannot split, we sit down with them for a 30 minute session with a whiteboard and the delivery lead. We have run this session 41 times. It has produced a valid split 39 times. The two exceptions were a vendor cutover with an external deadline and a hotfix that shipped inside a day.

    A concrete example: splitting the profile export ticket

    Last quarter we had a ticket that read: add data export for user profiles, including preferences, integrations, activity history, and audit logs, downloadable as a signed ZIP. Original estimate: two weeks. Owner: one engineer on the Growth pod.

    Under the old rules this would have entered a cycle whole. Under the slice rule we ran it through Slice Check on the Friday before, and split it into three tickets:

    1. Slice one, week of Jan 13. Endpoint scaffold plus preferences payload. Friday demo: hit POST /exports/profile in staging, receive a signed URL, download a ZIP containing a single preferences.json file. Behind a flag. One engineer, five days.
    2. Slice two, week of Jan 20. Add integrations and activity history to the payload. Friday demo: same endpoint, same flag, ZIP now contains three files. Handles the 90 percent case of activity records fitting in a single query batch.
    3. Slice three, week of Feb 3. Audit logs, pagination for large history sets, signed URL expiry policy, and flag flip. Friday demo: end to end run for a real customer account with 40k audit rows, timing recorded in the demo doc.

    We put a two-week gap between slice two and slice three on purpose. That gap ran a quiet beta with three friendly customers on slice two, and the feedback moved the audit log format before we built it.

    Total calendar time: roughly the same as the original two-week estimate would have been had it not slipped. The difference is that we had something demoable at three checkpoints instead of one hopeful checkpoint at the end.

    What we track

    We keep three numbers on a Notion page called Delivery Health:

    • Percent of active-cycle tickets that hit their Friday demo artifact. Current: 88 percent, target 85.
    • Median time from a ticket entering Ready for Cycle to its first PR opened. Current: 1.2 days.
    • Number of tickets that entered a cycle unsplit and slipped. Current: 2 this quarter, down from 14 the same quarter last year.

    The rule is not magic. It is a constraint that forces the planning conversation to happen on Friday instead of Wednesday of week two, when a slip is already priced in. If the demo cannot be pictured on Friday, the work is not ready. That is the whole rule.