Velo Blog

Category: Engineering

  • The debugging log we always regret losing

    The debugging log we always regret losing

    Every engineer on our team keeps a file called bug.md or scratch.txt or something equally forgettable. It lives in a folder that never gets checked into git. Nobody told us to make it. We all did anyway, because the third time a hard bug ate a full Thursday, we figured out that memory is the worst debugging tool we own.

    The file is append-only by convention, never by tooling. New entries go at the bottom with a timestamp and whatever we were poking at when the thought landed. When the bug closes, the log stays. When a laptop dies or we forget to sync it to iCloud, we mourn. One of us lost eight months of notes last October and still brings it up in retros.

    Here are three excerpts from the past quarter, lightly sanitized, that show why we would rather lose a Figma file than lose one of these.

    Excerpt one: the phantom timeout

    Tuesday, 2:14pm. Datadog spiked p95 on the checkout service to 8.2s. No deploys in 48 hours. Two engineers pulled in from a payments meeting.

    14:16 - checked prod dashboard, only /orders POST spiking, /orders GET fine
14:19 - pulled last 500 traces, 91% share a redis GET on session token
14:23 - ran KEYS scan against staging, 1.2M sessions cached from Friday load test
14:27 - Marcus reminded me eviction policy on this cluster is noeviction
14:31 - confirmed via redis-cli config get maxmemory-policy
14:33 - shipping maxmemory-policy allkeys-lru now, PR #4821
14:58 - p95 back to 340ms

    Six lines, one closed Linear ticket, one lesson written into our runbook. If we had captured a screenshot of the Datadog spike and nothing else, we would have a picture of the pain and none of the path out. Three months from now, when p95 climbs again on a Friday afternoon, this log will be the first thing we grep.

    Excerpt two: the flaky test that was not flaky

    This one ran across five days and two engineers. The Linear ticket collected fourteen comments. The log had forty entries. Only one of them mattered.

    Wed 10:04 - test_billing_webhook_signature failing on CI, passes local
Wed 10:22 - retried 3x on CI, one green one red one red
Wed 11:40 - added extra logging, deployed to CI branch
Wed 14:15 - noticed failures happen only on runners pulled after 13:00 UTC
Thu 09:30 - runner image was rebuilt yesterday, openssl bumped to 3.2
Thu 09:45 - openssl 3.2 changed HMAC digest default padding behavior
Thu 10:12 - our webhook signer never set padding explicitly, tests inherited default
Fri 16:20 - PR #4903 pins padding, backports to test util

    The Wednesday 14:15 note is the whole thing. It is a stray observation, not a fix. If we had been operating inside a Slack thread and a screenshot pipeline, that observation would have been buried under seventeen emoji reactions and lost forever. In the log, it sits alone on a line, waiting to be reread on Thursday morning when it finally clicks.

    Excerpt three: the null we could not explain

    Mon 08:50 - user report: dashboard renders "customer name: null" for 4 accounts
Mon 09:10 - all 4 accounts created between Aug 3 and Aug 5
Mon 09:12 - Aug 3 was our migration cutover date
Mon 09:14 - checked backfill script, filters WHERE created_at < '2025-08-03'
Mon 09:15 - accounts created ON Aug 3 fell into the gap
Mon 09:20 - one-off patch queued, wrote regression test, added to postmortem

    Twenty minutes, start to finish. A screenshot would have told us the UI was broken. The log told us we shipped a boundary bug in a migration script, and reminded us to check for the same class of mistake in the next migration on the roadmap.

    Why context beats screenshots

    Screenshots capture a state. Logs capture a decision tree. When we return to a screenshot, we see what we saw. When we return to a log, we see what we tried, what we ruled out, and what surprised us. The last of those is the reason we keep writing these files.

    A short list of what belongs in the log:

    • The exact command we ran, copy-pasted with its output
    • The question we were asking when we ran it
    • What we expected and what we got instead
    • Any stray observation that felt off but was not the bug
    • The commit hash, PR number, or Linear ID at the end

    A shorter list of what does not belong:

    • Long paragraphs
    • Screenshots
    • Anything we have to format before saving

    The rule that keeps the log useful is that it must be faster to add a line than to think about not adding it. The moment it feels like writing, it stops getting written. That is why nobody on our team keeps this file in Notion. Notion asks us to name the page, pick a template, and choose an emoji. By then the thought is gone.

    We have started nudging new engineers to keep one within their first week. Not as a policy. As a warning. On the day their laptop dies, they will remember which files they miss most. This is always one of them.

  • The two week feature flag rule

    The two week feature flag rule

    Most engineering blogs will tell you feature flags are free. Ship dark, roll out slow, keep the fallback path warm. We used to believe this too. Then we counted our flags.

    Last month we had 47 flags in LaunchDarkly. Nine were older than a quarter. Four were older than us remembering why they existed. Two of them fought each other in production and caused a payout retry loop that Datadog flagged at 2:14 on a Tuesday afternoon.

    Why the “keep it around, it’s cheap” argument is wrong

    The pitch for long lived flags is that they cost nothing. A boolean check, a config entry, a line in a dashboard. The real cost shows up somewhere else:

    • Every conditional doubles the state space a new engineer has to hold in their head when touching that file.
    • Test matrices grow multiplicatively. Two stale flags plus one new one gives you eight paths, and nobody writes eight tests.
    • Old flags rot silently. The useNewCheckout branch you shipped in March has drifted from the legacy branch it lives next to, and neither of you noticed.
    • On call gets worse. When something breaks, the first question is always “what flags are on for this customer?” and the answer takes twenty minutes to assemble.

    The flag was supposed to be a valve. Left open, it becomes a fork in the codebase you have to maintain twice.

    The rule we adopted

    Every flag has an owner and a fourteen day timer. At day fourteen, the losing branch gets deleted. Not deprecated, not marked for cleanup in Linear, deleted. If we shipped the new checkout behind a flag and it stuck, the old checkout code goes. If the rollout failed, the new code goes.

    The uncomfortable part: sometimes we delete a branch that turns out to be needed later, and we rewrite it. We accept that cost. A week of rework is cheaper than a year of dual maintenance.

    The mechanics are boring. Every Monday, a GitHub Action posts the list of aged flags to our #eng-hygiene Slack channel with the owner tagged. If the flag is still needed, the owner extends it once, by another week, and writes why in the thread. Second extensions get escalated to the Friday engineering sync.

    A flag that has been on for a month is not a flag. It is a feature you forgot to finish shipping.

    Since we started, our flag count has dropped from 47 to 12. Half the incidents we traced back to “unexpected flag interaction” have stopped happening. The rework tax has been real, and worth it.

  • How twelve of us run trunk-based development

    How twelve of us run trunk-based development

    We moved to trunk-based development eighteen months ago, when the team was seven engineers and the release train was groaning under its own weight. We are twelve now, split across three squads, and every one of us commits to main multiple times a day. What follows is the routine we settled into, and the two things that caught us off guard.

    None of this is theoretical. It is what we do between our Monday planning meeting and our Friday demo, using GitHub, Linear, Datadog, and a lot of small pull requests.

    The daily rhythm

    Our workflow rests on three habits that we practice without thinking about them now. When a new engineer joins, these are the three things we teach in the first week.

    • Feature flags before feature code. Every user-visible change ships behind a flag in LaunchDarkly. The flag lands in a separate PR, sometimes an hour before the feature work starts. That way the wiring is reviewed on its own, and the rollout is a config change rather than a deploy.
    • Branches that live less than a day. Our house rule is that a branch should not sleep. If the sun sets on your branch, you either open a draft PR to get eyes on it, or you merge whatever green subset you have behind a flag. We measure this: last quarter, the median branch age was 6 hours and 42 minutes.
    • Five-minute code review triage. At 10:15 and 15:15, whoever is on review rotation opens the GitHub PR queue and gives every open PR one of three responses: approved, one specific question, or a note that the review will land by end of day. No PR is allowed to sit without a signal for more than four hours during working time.

    The triage rule is the one that took the most discipline to adopt. Two twelve-minute windows a day sounds trivial, but it forced us to break up large PRs. If you cannot skim a diff in ninety seconds, the reviewer flags it and asks for a split. We settled on a soft cap of 400 lines of diff, and a hard cap of 800.

    What we do when things break

    Trunk-based development only works if main is trustworthy. Ours is protected by a required check that runs unit tests, a smoke suite against a staging environment, and a Datadog synthetic against three critical paths. When the check goes red, whoever pushed the offending commit has two options: revert within fifteen minutes, or roll forward within thirty. We track this in a Notion page called the Green Log. Since January, we have had eleven red-main incidents, and nine of them were resolved by revert.

    We optimise for the next commit, not the current one. A revert is not a failure. A stuck main is.

    What surprised us

    We expected the obvious wins: faster feedback, smaller blast radius, less time spent on release branches. Those all showed up. What we did not predict were two second-order effects that changed how the team behaves.

    Merge conflicts became rare. We assumed twelve people pushing to one branch would create a merge conflict every few hours. In practice, the opposite happened. Because everyone rebases against a fresh main multiple times a day, conflicts surface early, when the diffs are small and the context is still in someone’s head. Our GitHub metrics show a conflict rate of roughly one PR in forty, down from one in eight when we were running two-week release branches. The conflicts we do get are almost always resolved in under ten minutes by the author.

    Design conversations moved earlier. This is the change we care about most. When branches lived for a week, most technical debate happened at review time, when the code was already written and the author was invested. Now, because a branch cannot survive a day, engineers ask for a design opinion before they open the editor. Our #eng-design Slack channel used to see two or three threads a week. It now sees eight or nine, and most of them are fifteen-minute exchanges about approach, not aesthetics. A few times a month, one of those threads turns into a Figma sketch or a Linear ticket for a spike.

    We think the mechanism is simple. Short branches make the cost of throwing away code visible. If you have to justify a day of work, you tolerate risk. If you only have to justify two hours, you ask the question first.

    What we would tell a team starting today

    If we were setting this up again from scratch, we would do these things in this order:

    1. Put LaunchDarkly, or a homegrown equivalent, in place before you change any branching rules. Without flags, small merges are frightening.
    2. Write down the review triage times and defend them. Ours are on the team calendar as recurring events.
    3. Pick a diff-size cap and enforce it socially. Numbers matter less than the shared expectation that big PRs get split.
    4. Track red-main minutes as a team metric, not an individual one. We share the Green Log in our Friday retro.

    Trunk-based development is not a productivity trick. It is a set of constraints that make the team’s default behaviour better. Eighteen months in, we are shipping about twice as often as we used to, and the conversations we have about how to build things are earlier and more useful. Those two outcomes are worth more to us than any of the workflow mechanics that produce them.

  • How we unbreak a Wednesday sprint without cancelling it

    How we unbreak a Wednesday sprint without cancelling it

    Every team we know has had that sprint. Monday standup looked fine. Tuesday morning a payment webhook started dropping in staging, one engineer went out sick, and the design review pushed the checkout redesign back by two days. By Wednesday afternoon the burndown on our Linear board was flat, six tickets deep, and the sprint goal read like fiction.

    We used to cancel sprints in this situation. We stopped doing that around a year ago. Cancelling costs us the retrospective, the sense of finishing something, and the muscle memory of shipping on a cadence. What replaced it is a Wednesday recovery ritual that we run in about forty minutes.

    The Wednesday triage, not another standup

    We block thirty minutes on Wednesday at 2pm called “Sprint check”. It only fires when the burndown deviates more than twenty percent from the ideal line, which our Datadog dashboard flags in a Slack channel called #eng-signals. If the sprint is on track, the meeting is cancelled by 1:45pm and nobody joins.

    When it does fire, three people attend: the engineering lead for the squad, the product manager, and whoever picked up the on-call pager that week. No designers, no wider group. The point is a fast, honest read on the remaining ten working hours across four engineers.

    The question we ask is not “can we still finish everything?” It is “what one thing, if shipped by Friday, would make this sprint worth having run?”

    That one question forces a decision that the daily standup rarely produces. Standups report status. Wednesday triage rewrites the plan.

    Cut scope in the second half

    Once the anchor ticket is named, we walk the remaining Linear tickets and sort them into three buckets. We do this on a shared Notion page titled “Sprint 47 midweek reset” with three headings and drag ticket links under each.

    1. Ship this week. The anchor ticket and anything on its critical path. Usually two or three tickets.
    2. Defer to next sprint. Work that is not blocking anyone. Move it back to the backlog with a comment explaining why.
    3. Drop, do not defer. Tickets that felt urgent on Monday and no longer do. These get closed with a short note. If they matter again, someone will reopen them.

    The third bucket is the one that saves us. Roughly a quarter of what we plan on Monday gets dropped rather than deferred, and none of it has come back to bite us in the four sprints we have tracked this pattern.

    Slice the anchor, do not shrink it

    The highest value ticket is where teams tend to lie to themselves. On Wednesday we do not promise a smaller version of the same scope. We split the ticket into two Linear issues, and the parent becomes an epic.

    Take our checkout redesign example. The original ticket read: “Ship the redesigned checkout flow with saved cards, address autofill, and Apple Pay.” By Wednesday it was clear we had ten hours of engineering work left and about twenty hours of scope. The split looked like this:

    • PAY-412: Ship the redesigned checkout behind a feature flag, five percent rollout, saved cards only. Owner: Priya. Estimate: eight hours.
    • PAY-413: Address autofill and Apple Pay under the same flag. Owner: unassigned. Moved to next sprint.

    The slice we ship on Friday is a real, running thing in production, even if it sits behind a flag at five percent. Next sprint we widen it. What we avoid is the trap of promising the whole checkout by Friday, then delivering nothing and calling it a spike.

    The rolling change log

    Every scope change goes into a single Notion page we call the Sprint Ledger. One page per sprint, appended to as things move. Each entry has a timestamp, the ticket ID, what changed, and one line of why.

    Wed 14:32  PAY-401  Dropped. Duplicated by PAY-397 already in progress.
Wed 14:35  PAY-412  Split from PAY-388. Anchor for the week.
Wed 14:41  ONB-215  Deferred. Blocked on design; no unblock this week.

    The ledger takes about six minutes to fill in during triage. It is read twice: once by the wider squad on Wednesday afternoon, and once by the retro facilitator on the following Monday. Nobody hunts through Slack scrollback trying to remember what changed and why.

    What we get back

    Four things, measured over ten sprints since we started running this ritual:

    • We now finish about eighty percent of our stated sprint goal by Friday, up from around fifty percent when we would grind on the original plan or cancel outright.
    • Retros focus on cause, not blame. The ledger tells us what happened; we can talk about why.
    • Product managers push back less on Wednesday cuts, because the anchor is preserved and the ledger makes the trade visible.
    • On-call load in the second half of the sprint dropped, because we stopped shipping half finished work under time pressure.

    None of this requires new tooling. Linear, Notion, Slack, one recurring calendar block, and a rule about when it fires. The hardest part is not the process; it is the willingness on Wednesday afternoon to say out loud that Monday’s plan is no longer the plan, and to write down what replaced it before the day ends.