Velo Blog

Tag: feature-flags

  • The two week feature flag rule

    The two week feature flag rule

    Most engineering blogs will tell you feature flags are free. Ship dark, roll out slow, keep the fallback path warm. We used to believe this too. Then we counted our flags.

    Last month we had 47 flags in LaunchDarkly. Nine were older than a quarter. Four were older than us remembering why they existed. Two of them fought each other in production and caused a payout retry loop that Datadog flagged at 2:14 on a Tuesday afternoon.

    Why the “keep it around, it’s cheap” argument is wrong

    The pitch for long lived flags is that they cost nothing. A boolean check, a config entry, a line in a dashboard. The real cost shows up somewhere else:

    • Every conditional doubles the state space a new engineer has to hold in their head when touching that file.
    • Test matrices grow multiplicatively. Two stale flags plus one new one gives you eight paths, and nobody writes eight tests.
    • Old flags rot silently. The useNewCheckout branch you shipped in March has drifted from the legacy branch it lives next to, and neither of you noticed.
    • On call gets worse. When something breaks, the first question is always “what flags are on for this customer?” and the answer takes twenty minutes to assemble.

    The flag was supposed to be a valve. Left open, it becomes a fork in the codebase you have to maintain twice.

    The rule we adopted

    Every flag has an owner and a fourteen day timer. At day fourteen, the losing branch gets deleted. Not deprecated, not marked for cleanup in Linear, deleted. If we shipped the new checkout behind a flag and it stuck, the old checkout code goes. If the rollout failed, the new code goes.

    The uncomfortable part: sometimes we delete a branch that turns out to be needed later, and we rewrite it. We accept that cost. A week of rework is cheaper than a year of dual maintenance.

    The mechanics are boring. Every Monday, a GitHub Action posts the list of aged flags to our #eng-hygiene Slack channel with the owner tagged. If the flag is still needed, the owner extends it once, by another week, and writes why in the thread. Second extensions get escalated to the Friday engineering sync.

    A flag that has been on for a month is not a flag. It is a feature you forgot to finish shipping.

    Since we started, our flag count has dropped from 47 to 12. Half the incidents we traced back to “unexpected flag interaction” have stopped happening. The rework tax has been real, and worth it.

  • How twelve of us run trunk-based development

    How twelve of us run trunk-based development

    We moved to trunk-based development eighteen months ago, when the team was seven engineers and the release train was groaning under its own weight. We are twelve now, split across three squads, and every one of us commits to main multiple times a day. What follows is the routine we settled into, and the two things that caught us off guard.

    None of this is theoretical. It is what we do between our Monday planning meeting and our Friday demo, using GitHub, Linear, Datadog, and a lot of small pull requests.

    The daily rhythm

    Our workflow rests on three habits that we practice without thinking about them now. When a new engineer joins, these are the three things we teach in the first week.

    • Feature flags before feature code. Every user-visible change ships behind a flag in LaunchDarkly. The flag lands in a separate PR, sometimes an hour before the feature work starts. That way the wiring is reviewed on its own, and the rollout is a config change rather than a deploy.
    • Branches that live less than a day. Our house rule is that a branch should not sleep. If the sun sets on your branch, you either open a draft PR to get eyes on it, or you merge whatever green subset you have behind a flag. We measure this: last quarter, the median branch age was 6 hours and 42 minutes.
    • Five-minute code review triage. At 10:15 and 15:15, whoever is on review rotation opens the GitHub PR queue and gives every open PR one of three responses: approved, one specific question, or a note that the review will land by end of day. No PR is allowed to sit without a signal for more than four hours during working time.

    The triage rule is the one that took the most discipline to adopt. Two twelve-minute windows a day sounds trivial, but it forced us to break up large PRs. If you cannot skim a diff in ninety seconds, the reviewer flags it and asks for a split. We settled on a soft cap of 400 lines of diff, and a hard cap of 800.

    What we do when things break

    Trunk-based development only works if main is trustworthy. Ours is protected by a required check that runs unit tests, a smoke suite against a staging environment, and a Datadog synthetic against three critical paths. When the check goes red, whoever pushed the offending commit has two options: revert within fifteen minutes, or roll forward within thirty. We track this in a Notion page called the Green Log. Since January, we have had eleven red-main incidents, and nine of them were resolved by revert.

    We optimise for the next commit, not the current one. A revert is not a failure. A stuck main is.

    What surprised us

    We expected the obvious wins: faster feedback, smaller blast radius, less time spent on release branches. Those all showed up. What we did not predict were two second-order effects that changed how the team behaves.

    Merge conflicts became rare. We assumed twelve people pushing to one branch would create a merge conflict every few hours. In practice, the opposite happened. Because everyone rebases against a fresh main multiple times a day, conflicts surface early, when the diffs are small and the context is still in someone’s head. Our GitHub metrics show a conflict rate of roughly one PR in forty, down from one in eight when we were running two-week release branches. The conflicts we do get are almost always resolved in under ten minutes by the author.

    Design conversations moved earlier. This is the change we care about most. When branches lived for a week, most technical debate happened at review time, when the code was already written and the author was invested. Now, because a branch cannot survive a day, engineers ask for a design opinion before they open the editor. Our #eng-design Slack channel used to see two or three threads a week. It now sees eight or nine, and most of them are fifteen-minute exchanges about approach, not aesthetics. A few times a month, one of those threads turns into a Figma sketch or a Linear ticket for a spike.

    We think the mechanism is simple. Short branches make the cost of throwing away code visible. If you have to justify a day of work, you tolerate risk. If you only have to justify two hours, you ask the question first.

    What we would tell a team starting today

    If we were setting this up again from scratch, we would do these things in this order:

    1. Put LaunchDarkly, or a homegrown equivalent, in place before you change any branching rules. Without flags, small merges are frightening.
    2. Write down the review triage times and defend them. Ours are on the team calendar as recurring events.
    3. Pick a diff-size cap and enforce it socially. Numbers matter less than the shared expectation that big PRs get split.
    4. Track red-main minutes as a team metric, not an individual one. We share the Green Log in our Friday retro.

    Trunk-based development is not a productivity trick. It is a set of constraints that make the team’s default behaviour better. Eighteen months in, we are shipping about twice as often as we used to, and the conversations we have about how to build things are earlier and more useful. Those two outcomes are worth more to us than any of the workflow mechanics that produce them.