It is Wednesday. The sprint is half done, two people are mid-feature, and a message arrives: members cannot complete enrolment. Real users, right now, in production.
What you do in the next ten minutes determines whether this costs the team two hours or two days. And the most common mistake is the one that feels most responsible: pulling everyone onto it.
First, establish severity — properly
Everything that arrives urgently is described as urgent. Before you move anyone, find out what you are actually dealing with. Three questions, two minutes.
| Question | Why it changes your response |
|---|---|
| Is data being lost or corrupted? | If yes, this outranks everything. Stop the bleeding before diagnosing. |
| How many users, and can they work around it? | All users blocked is different from one client with an alternative path |
| Is it getting worse? | A growing queue or climbing error rate is a different clock from a stable failure |
Those give you a tier, and the tier tells you how much of the sprint to sacrifice.
Critical — data at risk, or everyone blocked with no workaround. The sprint stops. You will not make the commitment and that is correct.
High — significant group blocked, workaround exists or can be created. One or two people move. The sprint bends.
Looks urgent, is not — one user, cosmetic, or affecting something used monthly. It goes in the backlog with a note. Saying this out loud is part of the job.
Pulling the whole team onto a production bug feels decisive and is usually counterproductive. Four engineers cannot debug one issue in parallel — three of them will read the same logs, ask the responder questions, and slow down the person actually making progress. Meanwhile the sprint stops for everyone rather than for one person.
Assign one responder and one communicator
Two roles, and for anything short of a full outage they can be different people so that neither job blocks the other.
The responder debugs. Nobody else touches the problem. They get quiet and they get whatever access they need.
The communicator — often you — handles support, the client, and the rest of the team. Their job is to protect the responder from being asked for status every ten minutes, because context switching is what turns a forty-minute fix into an afternoon.
Everyone else keeps working on the sprint. That is not callousness, it is the whole point: an incident should cost you one person's day, not five.
Choose the responder deliberately
The instinct is to take it yourself, because you are fastest and it is high pressure. Sometimes that is right. Often it is not.
If you take every incident, you become the only person who can handle one, and you are unavailable for everything else every time production hiccups. Where the severity allows it, hand it to someone else and stay close — sit with them, ask what they have ruled out, point at the next place to look rather than at the answer.
Incidents are the fastest way engineers learn a system. Taking them all yourself is hoarding the most valuable training you have.
Stop the bleeding before finding the cause
Engineers want to understand before acting. In an incident that order is frequently wrong.
If there is a fast way to reduce impact — roll back the release, disable the feature flag, switch the vendor call to its fallback, put a banner on the page — do it first. It buys you the ability to diagnose calmly instead of with support tickets arriving.
Mitigation and diagnosis are different activities. Do the first one under pressure and the second one without it.
The counter-case: if rolling back would destroy the evidence and the impact is tolerable, capture what you need first. Say that decision out loud so everyone knows it was a choice.
Protect the sprint on purpose
Something urgent arrives most sprints. If you plan at full capacity every time, you fail the commitment every time and the team learns that sprint planning is theatre.
Two things fix this:
Leave real headroom. Whatever your historical interruption rate is — and you can measure it — do not commit that portion. If you consistently lose fifteen percent to production work, plan for eighty-five.
Rotate an on-call or support role. One person each sprint whose planned work is deliberately light and who picks up whatever arrives. It protects four people completely instead of disrupting five partially, and it spreads system knowledge around.
Say something about the sprint, immediately
When an incident pulls someone off planned work, tell the stakeholder that day. Not at the review, when it is a surprise and it is too late to react.
“We had a production issue this morning affecting enrolment. It is resolved. It cost us most of a day, so the reporting screen is at risk for Friday — I will confirm tomorrow. Everything else is on track.”
Three sentences. It costs nothing and it keeps the client's trust intact, because they find out from you rather than from a missing demo.
When the fix is ready, do not skip the process
Pressure invites shortcuts, and hotfixes cause a disproportionate share of second incidents. A rushed fix merged without review is how a problem affecting two hundred users becomes a problem affecting everyone.
The minimum, even at 8pm:
- Someone other than the author reads the diff. Five minutes, not a full review.
- The change is as small as it can possibly be — fix the symptom now, do the proper repair as a planned piece of work.
- You can state specifically how you will confirm it worked, before you deploy it.
- You know how to roll it back.
Verify with data, not with a feeling. The incident is over when the error rate returns to baseline, not when the deploy finishes.
Then separate the patch from the repair
Hotfixes are usually tactical — a guard clause, a retry, a null check. They stop the pain and they leave the underlying cause in place.
Create the follow-up ticket while the context is fresh, that same day, with what you actually learned. If you do not, the tactical fix becomes permanent and you will meet the same bug in a slightly different shape in four months.
The review question that matters
Afterwards, do not spend long on what caused it. You know that. Ask:
Why did a user find this before we did?
The answer is almost always that the signal existed and nobody was watching it. The error rate had been non-zero since Monday. The queue had been growing for two days. Nothing alerted because nothing was configured to.
The genuinely valuable output of an incident is not the fix — it is the alert that means next time your dashboard tells you before your client does.
The short version
Establish severity before moving anyone. Not everything urgent is critical.
One responder, one communicator. Everyone else stays on the sprint.
Mitigate first, diagnose second.
Hand it to someone else when you can, and stay close.
Tell the stakeholder about the sprint impact the same day.
Keep review even under pressure. Small fix, known rollback, verified with data.
Ticket the real repair immediately, and add the alert you were missing.
Handled this way, a production incident costs one person a day and leaves the system better monitored than it was. Handled badly, it costs the whole sprint and leaves nothing behind except relief.
