Tag: tech-debt

  • The two week feature flag rule

    The two week feature flag rule

    Most engineering blogs will tell you feature flags are free. Ship dark, roll out slow, keep the fallback path warm. We used to believe this too. Then we counted our flags.

    Last month we had 47 flags in LaunchDarkly. Nine were older than a quarter. Four were older than us remembering why they existed. Two of them fought each other in production and caused a payout retry loop that Datadog flagged at 2:14 on a Tuesday afternoon.

    Why the “keep it around, it’s cheap” argument is wrong

    The pitch for long lived flags is that they cost nothing. A boolean check, a config entry, a line in a dashboard. The real cost shows up somewhere else:

    • Every conditional doubles the state space a new engineer has to hold in their head when touching that file.
    • Test matrices grow multiplicatively. Two stale flags plus one new one gives you eight paths, and nobody writes eight tests.
    • Old flags rot silently. The useNewCheckout branch you shipped in March has drifted from the legacy branch it lives next to, and neither of you noticed.
    • On call gets worse. When something breaks, the first question is always “what flags are on for this customer?” and the answer takes twenty minutes to assemble.

    The flag was supposed to be a valve. Left open, it becomes a fork in the codebase you have to maintain twice.

    The rule we adopted

    Every flag has an owner and a fourteen day timer. At day fourteen, the losing branch gets deleted. Not deprecated, not marked for cleanup in Linear, deleted. If we shipped the new checkout behind a flag and it stuck, the old checkout code goes. If the rollout failed, the new code goes.

    The uncomfortable part: sometimes we delete a branch that turns out to be needed later, and we rewrite it. We accept that cost. A week of rework is cheaper than a year of dual maintenance.

    The mechanics are boring. Every Monday, a GitHub Action posts the list of aged flags to our #eng-hygiene Slack channel with the owner tagged. If the flag is still needed, the owner extends it once, by another week, and writes why in the thread. Second extensions get escalated to the Friday engineering sync.

    A flag that has been on for a month is not a flag. It is a feature you forgot to finish shipping.

    Since we started, our flag count has dropped from 47 to 12. Half the incidents we traced back to “unexpected flag interaction” have stopped happening. The rework tax has been real, and worth it.