I spent three hours last Tuesday digging through a mountain of “INFO” level noise just to find the one trace that explained why a specific payment worker was silently retrying a dead socket. It’s the classic mistake: we treat logging like a garbage disposal where we throw everything in, hoping the useful bits won’t get lost. Most people think the answer to observability is just buying a bigger Datadog budget or implementing some “industry standard” structured logging library, but they’re missing the point entirely. Knowing how to decide what to log isn’t about coverage; it’s about signal density. If your logs don’t tell a story that helps you reconstruct a failure without a debugger, you aren’t actually monitoring anything—you’re just paying to store digital landfill.
I’m not going to give you a checklist of every possible event type or a lecture on JSON schema best practices. Instead, I want to share the mental framework I’ve built after a decade of seeing systems fail in ways no one predicted. I’ll show you how to filter out the fluff and focus on the specific data points that actually matter when the pager goes off. This is about building a trail of breadcrumbs that makes the next person—which might be you, six months from now—actually able to fix the problem.
Table of Contents
Why Most Monitoring System Telemetry Fails You
Most teams treat monitoring system telemetry like a digital landfill. They think that if they just throw enough data into a bucket, the answer to whatever is currently melting the production environment will eventually float to the top. It doesn’t. Instead, you end up with a massive, expensive pile of noise that makes it impossible to find the signal when you actually need it. I’ve spent far too many hours staring at dashboards filled with thousands of identical “User logged in” messages, only to realize they tell me absolutely nothing about why the connection pool was exhausted ten minutes later.
The real problem is usually a complete lack of structured logging principles. When your logs are just a collection of unstructured strings, you aren’t actually observing your system; you’re just recording its screams. Without a consistent schema, you can’t query your data to find patterns, which means your error tracing strategies are basically just glorified Ctrl+F searches through a haystack. We often fall into the trap of thinking more data equals more visibility, but in reality, we are just increasing the cognitive load required to solve a problem. If you can’t pivot from an error to the specific transaction ID that caused it in under three clicks, your telemetry isn’t helping you—it’s just costing you money.
Finding the Signal Amidst the Noise
The problem with most logging setups is that they treat all data as equal. When you’re staring at a dashboard during a live incident, you don’t need a play-by-play of every successful HTTP 200 request; you need the context that explains the one 500 error that just tripped the alarm. This is where most people fail at reducing log verbosity. They log everything because they’re afraid of missing something, but they end up burying the actual cause of death under a mountain of triviality.
To fix this, I’ve had to get strict with my log level management. If a log line doesn’t help me answer a specific question—like “which user triggered this retry loop?” or “what was the state of the cache when this timed out?”—it shouldn’t be there. I try to follow basic structured logging principles not because they look good in a README, but because searching through unstructured blobs of text at 2 AM is a special kind of hell. You want your logs to be machine-readable so you can actually query them when the system starts acting up. If you can’t filter your way to the truth in under thirty seconds, you aren’t logging; you’re just storing digital garbage.
Five Rules for Not Drowning in Your Own Logs
- Log the “Why,” not just the “What.” It’s useless to see `Order processing failed`. I need to see `Order processing failed because the payment gateway returned a 402 after the retry limit was hit`. If the log doesn’t point me toward a root cause, it’s just expensive noise.
- Focus on state transitions. Most production bugs aren’t about a single line of code failing; they’re about a system getting into a state it shouldn’t be in. Log when a process moves from `PENDING` to `PROCESSING` and, more importantly, exactly why it failed to move to `COMPLETED`.
- Context is king, but don’t be a hoarder. Attach the `correlation_id` and the `user_id` to every meaningful event, but stop dumping the entire request payload into the log stream. You don’t need the user’s profile picture URL to debug a database deadlock.
- Treat log levels like they actually matter. If everything is an `ERROR`, then nothing is. I’ve seen teams where the error dashboard is a permanent sea of red because someone decided a missing optional field was a “critical error.” Use `WARN` for things that are weird but recoverable, and save `ERROR` for when someone is actually getting woken up.
- Test your logs as part of your feature work. I’ve started treating log statements like code—if I’m writing a new service, I’m writing the logs alongside it. If I can’t simulate a failure in my local environment and see a clear, actionable log message explaining it, the feature isn’t finished.
The Post-Mortem Litmus Test
Before you add a new log line, ask yourself: “If this triggers an alert at 2 AM, does it actually tell me what broke, or does it just tell me that something is broken?”
Stop logging state transitions that you can already infer from the surrounding context; if you know a user is in a ‘checkout’ flow, you don’t need a log entry for every single millisecond of that flow unless the state actually changes.
Prioritize the ‘why’ over the ‘what’—logging `order_processed_failed` is useless unless you also capture the specific error code and the input parameters that caused the logic to trip.
The Long View on Logging
At the end of the day, deciding what to log isn’t an academic exercise in data retention; it’s a tactical decision about your future sanity. You don’t need a mountain of telemetry that just makes your storage bill explode and your dashboards unreadable. You need the specific, high-cardinality breadcrumbs—the user IDs, the correlation IDs, and the specific state transitions—that turn a frantic search through raw text into a surgical investigation. If a log line doesn’t answer a question you’ve actually asked during a post-mortem, it’s just expensive noise that’s getting in your way.
I’ve spent enough late nights staring at scrolling terminal buffers to know that more data rarely equals faster resolution. The goal is to build a system that respects your time and your cognitive load. Treat your logs like a well-maintained codebase: keep them purposeful, keep them lean, and keep them useful for the person who has to step in when things go sideways. Stop trying to capture everything and start capturing the right things. Your future self, especially the one waking up at 2 AM to a P0 alert, will thank you for it.
Frequently Asked Questions
How do I stop my logging from becoming a massive, expensive bill in our cloud provider's dashboard without missing the important stuff?
The short answer is: stop treating logs like a dumping ground for state. If you’re logging every successful request body just to “have it,” you’re just paying a tax for your own indecision.
At what point does a "trace" become too much data to actually be useful during an incident?
A trace becomes useless the moment you spend more time filtering through its spans than actually diagnosing the root cause. If I’m staring at a trace with five thousand spans trying to find the one database call that timed out, the instrumentation has failed me. You’ve hit the limit when the sheer volume of telemetry hides the latency spike rather than highlighting it. If you can’t see the “why” within thirty seconds of opening the trace, you’re just looking at expensive noise.
How do I convince my manager that spending time on structured logging is more important than shipping that new feature?
Don’t talk about “observability” or “technical debt”—those are just buzzwords that sound like expensive chores. Talk about the cost of a missed incident.
If I change a service's internal logic, how much of my existing logging schema do I actually need to rewrite?
If you’re rewriting your entire schema every time you touch internal logic, your logging strategy is too coupled to your implementation. That’s a recipe for burnout.
