There is a specific kind of silence that only exists at 3:00 AM, right after your phone has finished screaming at you and you realize you are the only person on the on-call rotation. It’s not a peaceful silence; it’s the heavy, ringing quiet of a production environment that has just gone dark. I remember sitting in my kitchen during a particularly nasty outage at my old insurance firm, staring at a lukewarm cup of coffee and wondering exactly how to handle an outage alone without losing my mind or making a mistake that would haunt my commit history for years. Most people will tell you to follow a complex, multi-step incident response framework, but when you’re staring at a broken dashboard in the dark, textbook theory is useless.
I’m not here to sell you on some expensive observability suite or a theoretical way to manage stress. I want to talk about the actual, unglamorous mechanics of staying sane and effective when there is no one to bounce ideas off of. I’m going to share the practical, hard-won lessons I’ve gathered from fifteen years of being wrong in production, focusing on how to triage without panic, how to document your way out of a hole, and how to ensure that when you finally go back to sleep, you aren’t leaving a ticking time bomb behind.
Table of Contents
Emergency Triage Procedures When the Dashboard Turns Red
When the dashboard turns red and the alerts start screaming, your first instinct is usually to start clicking things. Don’t. The most effective emergency triage procedures start with a single, uncomfortable step: stop and look at the telemetry before you touch a single line of code. I’ve spent too many nights trying to “fix” a service only to realize I was actually just masking a database deadlock by restarting a container. You need to establish a baseline of what is actually broken versus what is just noisy. Is it a spike in latency, a surge in 5xx errors, or is the entire ingress layer just failing to route?
Once you’ve identified the shape of the failure, you need to move into solo engineer troubleshooting steps that prioritize stability over elegance. This is not the time to refactor a function or try to find the “perfect” fix. Your goal is to arrest the bleeding. If a recent deployment triggered this, roll it back immediately. If it’s a resource exhaustion issue, scale up, even if it’s expensive. I always tell myself: stabilize first, investigate later. You can perform a proper autopsy once the system is breathing again; right now, you’re just trying to keep it on life support.
Solo Engineer Troubleshooting Steps That Actually Work Under Pressure
When you’re the only one staring at a spiking error rate, your brain starts playing tricks on you. You’ll feel an urge to start changing configuration values or rolling back deployments immediately, but that’s usually a trap. The first step in my personal set of solo engineer troubleshooting steps is to stop moving and start observing. I’ve learned the hard way that “fixing” something before you actually understand the telemetry is just a fast way to turn a minor hiccup into a total blackout. I grab a physical notepad—not a digital one that might be lagging—and I write down exactly what I see: the timestamp, the specific error codes, and the last known good state.
Once the initial panic subsides, I focus on isolating the blast radius. If I can determine whether the issue is a localized database deadlock or a global networking failure, I’ve already won half the battle. This is where reducing mean time to resolution solo becomes a matter of discipline rather than speed. I don’t hunt for the “why” yet; I hunt for the “where.” Is it one specific availability zone? A single microservice? By narrowing the scope, I avoid the rabbit hole of chasing ghosts in perfectly healthy subsystems. I treat it like a triage in a field hospital: stabilize the patient first, ask about their medical history later.
Five things I’ve learned from being the only person on call
- Stop trying to find the root cause immediately. When you’re alone, your first job isn’t to be a detective; it’s to be a firefighter. Get the service back to a stable state—even if that means rolling back a “perfect” deployment or temporarily disabling a non-critical feature—before you start digging through logs to find out why it happened.
- Write everything down in a scratchpad as you go. I’ve spent far too many hours trying to reconstruct my own thought process from an hour ago. Open a simple text file or a physical notebook and log every command you run and every change you make. It prevents you from running in circles and gives you a breadcrumb trail if you need to hand over the reins to someone else later.
- Beware the “one more small change” trap. There is a specific kind of madness that sets in at 3 AM where you think, “If I just tweak this one timeout value, it might fix it.” Usually, it doesn’t, and it just adds another variable to the mess. If a change doesn’t have a clear, documented purpose, don’t touch it until the system is stable.
- Communicate even if no one is listening. Even if you are the only one on the incident bridge, post updates to the status channel or the team Slack every 15-30 minutes. It keeps people from pinging you with “is it fixed yet?” and it creates a timestamped record of the incident that makes the eventual post-mortem much less painful.
- Know when to call for backup, even if it feels like a failure. There is no prize for suffering in silence. If you’ve been staring at the same dashboard for forty minutes without a single new lead, you aren’t “working the problem”—you’re stuck. Wake someone up. A fresh pair of eyes is worth more than another hour of your own exhausted intuition.
The things I keep in my back pocket
Stop trying to fix the root cause while the house is still on fire; focus entirely on stopping the bleeding first, even if that means a “dirty” fix like a hard restart or a rollback.
Trust your logs more than your intuition, because when you’re tired and stressed, your brain will try to convince you that a deployment didn’t cause the spike when it absolutely did.
Document your “why” in real-time, not just your “what,” so that when you finally get some sleep, you aren’t staring at your own commands wondering what the hell you were thinking.
The Morning After
At the end of the day, managing an outage by yourself isn’t about being a hero or having a magical command of every microservice in the stack. It’s about the discipline of staying calm when the metrics look like a cliff edge. You triage the immediate bleeding, you follow your own breadcrumbs through the logs, and you resist the urge to make massive, sweeping changes just to see something move. If you’ve done the work—if you’ve documented your steps and communicated clearly even when it felt like shouting into a void—then you’ve already won the most important battle. The goal isn’t just to bring the service back up; it’s to ensure you aren’t re-solving the exact same mystery six months from now because you were too tired to write it down.
Eventually, the sun will come up, the alerts will quiet down, and you’ll realize that the system is stable again. Don’t just close your laptop and go to sleep. Take ten minutes to jot down the one thing that actually fixed it, even if it was just a lucky guess or a specific config tweak. We all break things in production; it’s practically a rite of passage in this industry. The difference between a senior engineer and a junior one isn’t that the senior one doesn’t cause outages, but that they treat every failure as a permanent upgrade to their own mental documentation. Go get some sleep, but leave a note for your future self.
Frequently Asked Questions
What do I do if I realize the mistake was actually my own recent deployment?
The moment you realize it was your own code, the instinct is to panic or, worse, try to “fix” it with another hotfix. Don’t. If your deployment caused the spike, revert it immediately. Roll back to the last known stable state. It feels embarrassing to undo your work, but your priority is restoring service, not proving you can debug in real-time. We can figure out the “why” tomorrow; right now, we just need the dashboard to turn green.
How do I decide when to stop digging for the root cause and just start the rollback?
The moment you realize you’re playing detective instead of fixing the leak, stop. If you’ve spent twenty minutes tracing a stack trace and haven’t found the “why,” you’re just burning time while the business loses money. I have a rule: if the fix isn’t obvious within two diagnostic steps, I roll back. You can perform the autopsy on the dead code tomorrow morning. Right now, your job isn’t to be smart; it’s to restore service.
At what point do I finally call my manager or the on-call lead if I'm still stuck?
The “thirty-minute rule” is my personal baseline. If I’ve spent half an hour digging through logs and changing nothing but my own heart rate, it’s time to escalate. Don’t wait until you’re drowning; call the lead when you’ve exhausted your immediate hypotheses but haven’t found the root cause. It’s better to wake someone up while you still have a clear summary of the problem than to call them three hours later when you’re exhausted and guessing.
How do I keep a decent paper trail of my actions while I'm actually in the middle of the chaos?
Don’t try to write a formal report while the house is burning. You’ll fail, and you’ll lose your focus. Instead, keep a “chaos log”—a simple, raw scratchpad. I usually just keep a dedicated Slack channel or a temporary notepad open and dump timestamped fragments: “14:02 – scaled up pods, no change,” or “14:10 – DB connection pool exhausted.” It feels messy, but it saves you three hours of mental gymnastics during the post-mortem.
