Tag: postmortem

  • The debugging log we always regret losing

    The debugging log we always regret losing

    Every engineer on our team keeps a file called bug.md or scratch.txt or something equally forgettable. It lives in a folder that never gets checked into git. Nobody told us to make it. We all did anyway, because the third time a hard bug ate a full Thursday, we figured out that memory is the worst debugging tool we own.

    The file is append-only by convention, never by tooling. New entries go at the bottom with a timestamp and whatever we were poking at when the thought landed. When the bug closes, the log stays. When a laptop dies or we forget to sync it to iCloud, we mourn. One of us lost eight months of notes last October and still brings it up in retros.

    Here are three excerpts from the past quarter, lightly sanitized, that show why we would rather lose a Figma file than lose one of these.

    Excerpt one: the phantom timeout

    Tuesday, 2:14pm. Datadog spiked p95 on the checkout service to 8.2s. No deploys in 48 hours. Two engineers pulled in from a payments meeting.

    14:16 - checked prod dashboard, only /orders POST spiking, /orders GET fine
    14:19 - pulled last 500 traces, 91% share a redis GET on session token
    14:23 - ran KEYS scan against staging, 1.2M sessions cached from Friday load test
    14:27 - Marcus reminded me eviction policy on this cluster is noeviction
    14:31 - confirmed via redis-cli config get maxmemory-policy
    14:33 - shipping maxmemory-policy allkeys-lru now, PR #4821
    14:58 - p95 back to 340ms
    

    Six lines, one closed Linear ticket, one lesson written into our runbook. If we had captured a screenshot of the Datadog spike and nothing else, we would have a picture of the pain and none of the path out. Three months from now, when p95 climbs again on a Friday afternoon, this log will be the first thing we grep.

    Excerpt two: the flaky test that was not flaky

    This one ran across five days and two engineers. The Linear ticket collected fourteen comments. The log had forty entries. Only one of them mattered.

    Wed 10:04 - test_billing_webhook_signature failing on CI, passes local
    Wed 10:22 - retried 3x on CI, one green one red one red
    Wed 11:40 - added extra logging, deployed to CI branch
    Wed 14:15 - noticed failures happen only on runners pulled after 13:00 UTC
    Thu 09:30 - runner image was rebuilt yesterday, openssl bumped to 3.2
    Thu 09:45 - openssl 3.2 changed HMAC digest default padding behavior
    Thu 10:12 - our webhook signer never set padding explicitly, tests inherited default
    Fri 16:20 - PR #4903 pins padding, backports to test util
    

    The Wednesday 14:15 note is the whole thing. It is a stray observation, not a fix. If we had been operating inside a Slack thread and a screenshot pipeline, that observation would have been buried under seventeen emoji reactions and lost forever. In the log, it sits alone on a line, waiting to be reread on Thursday morning when it finally clicks.

    Excerpt three: the null we could not explain

    Mon 08:50 - user report: dashboard renders "customer name: null" for 4 accounts
    Mon 09:10 - all 4 accounts created between Aug 3 and Aug 5
    Mon 09:12 - Aug 3 was our migration cutover date
    Mon 09:14 - checked backfill script, filters WHERE created_at < '2025-08-03'
    Mon 09:15 - accounts created ON Aug 3 fell into the gap
    Mon 09:20 - one-off patch queued, wrote regression test, added to postmortem
    

    Twenty minutes, start to finish. A screenshot would have told us the UI was broken. The log told us we shipped a boundary bug in a migration script, and reminded us to check for the same class of mistake in the next migration on the roadmap.

    Why context beats screenshots

    Screenshots capture a state. Logs capture a decision tree. When we return to a screenshot, we see what we saw. When we return to a log, we see what we tried, what we ruled out, and what surprised us. The last of those is the reason we keep writing these files.

    A short list of what belongs in the log:

    • The exact command we ran, copy-pasted with its output
    • The question we were asking when we ran it
    • What we expected and what we got instead
    • Any stray observation that felt off but was not the bug
    • The commit hash, PR number, or Linear ID at the end

    A shorter list of what does not belong:

    • Long paragraphs
    • Screenshots
    • Anything we have to format before saving

    The rule that keeps the log useful is that it must be faster to add a line than to think about not adding it. The moment it feels like writing, it stops getting written. That is why nobody on our team keeps this file in Notion. Notion asks us to name the page, pick a template, and choose an emoji. By then the thought is gone.

    We have started nudging new engineers to keep one within their first week. Not as a policy. As a warning. On the day their laptop dies, they will remember which files they miss most. This is always one of them.