I still remember the sound of my phone buzzing on the nightstand at 3:14 AM during my time at the logistics startup. It wasn’t a high-level architectural debate that woke me up; it was a frantic Slack alert about a retry loop that had just turned a single delayed webhook into ten thousand duplicate shipping orders. Most tutorials on how to write idempotent code treat it like a theoretical math problem or a fancy pattern you add to your microservices to look smart. But in the real world, idempotency isn’t about elegance—it’s about preventing a total system meltdown when the network inevitably decides to flake out right when you’re trying to sleep.
I’m not going to waste your time with academic definitions or complex distributed locking abstractions that will only break in a different, more expensive way. Instead, I want to walk through the actual patterns I’ve used to keep databases sane and customers happy. I’ll show you the trade-offs of using idempotency keys versus state checks, and more importantly, I’ll tell you where these implementations usually fail in production. My goal is to give you a set of practical tools so that the next time a retry hits your endpoint, your system just shrugs and moves on.
Table of Contents
Handling Duplicate Requests in Apis Before They Escalate
The most common way this goes wrong is by assuming the network is reliable. It isn’t. A client sends a POST request to charge a customer, the server processes it successfully, but then the connection drops before the client gets the `200 OK`. The client, being a well-behaved piece of software, retries. Without a strategy for handling duplicate requests in APIs, you’ve just charged that customer twice. I’ve seen this happen in payment gateways where the retry logic was too aggressive and the backend was too trusting.
The standard way to stop this is through a solid idempotency key implementation. Instead of just blindly processing every incoming request, you require the client to send a unique identifier—usually a UUID—in the header. On the server side, you check if that key has been seen before within a certain window. If it has, you don’t run the business logic again; you simply return the cached response from the first successful attempt.
The trick is making sure this check is actually safe. If you check the key, then run the logic, then save the result, you’ve created a race condition. To do this right, you need to wrap the key check and the actual operation within a single unit of work to maintain database transaction atomicity. If you can’t guarantee that the “check” and the “act” happen as one inseparable event, you’re just waiting for a race condition to ruin your weekend.
Ensuring Request Safety in Microservices During Network Partitions
In a microservices architecture, the network is essentially a collection of lies. A service might successfully process a request, but the acknowledgment gets swallowed by a network partition or a sudden pod restart. From the perspective of the caller, that request failed. If the caller follows standard retry logic best practices and tries again, you’re suddenly dealing with a duplicate execution. Without a way to recognize that this is the same intent as the first attempt, you risk double-billing a customer or creating duplicate shipping orders.
I’ve seen this go sideways when teams assume that a simple “check if exists” query is enough. In a distributed system, that check and the subsequent write aren’t naturally a single unit of work. To actually achieve ensuring request safety in microservices, you have to tie the idempotency key directly to your persistence layer. This usually means using a unique constraint on an idempotency key within a single database transaction atomicity block. If the second request hits the database while the first is still committing, the database itself becomes your source of truth, rejecting the duplicate before any side effects can leak out. It’s not the most elegant way to write code, but it’s the only way that keeps me sleeping through the night.
Five things I’ve learned from watching retries go wrong
- Stop relying on the client to be well-behaved. I used to think a well-designed API would only send a request once, but networks are messy and developers get frustrated. You have to assume the client will hit that ‘Submit’ button five times in a row because the UI didn’t show a loading spinner fast enough.
- Use a unique idempotency key, but don’t just make it a random UUID. I’ve seen cases where the key was generated on the client side, but if the client crashed and restarted, it generated a new key for the same logical action. The key needs to be tied to the actual intent of the operation, not just the lifecycle of a single HTTP request.
- Database constraints are your last line of defense, not an afterthought. I once spent an entire afternoon debugging a race condition where two identical requests passed the application-level checks but both hit the database at the same millisecond. A unique constraint on your idempotency key column is the only thing that actually sleeps at night.
- Your response must be consistent, even if the work is already done. If a request comes in for an operation that succeeded ten minutes ago, don’t return a 409 Conflict or an error. Return the same 200 OK (or whatever your success code is) and the same payload you sent the first time. If the client thinks it failed, it’s just going to keep hammering you.
- Be careful with side effects that happen outside your primary transaction. If you update a row in Postgres but then try to send a Slack notification or a webhook, and that second part fails, you’re in a weird state. I’ve learned to move those side effects into an outbox pattern so the entire operation—the data change and the notification—is treated as a single, idempotent unit.
Lessons from the post-mortem
Idempotency isn’t a “nice-to-have” feature for your API; it’s the only thing standing between a transient network hiccup and a customer being charged three times for the same order.
Stop trying to build perfect distributed systems and start building systems that can fail gracefully; if a retry is going to happen, make sure the second attempt is as boring as the first.
Always design with the assumption that the client is going to retry, the network is going to drop the packet, and your database is going to be under heavy load—if your logic can’t survive that combination, it’s not production-ready.
The Bottom Line
At the end of the day, idempotency isn’t some academic concept you study to pass a system design interview; it is a practical defense mechanism. We’ve looked at how to catch duplicate API requests before they hit your business logic and how to keep your microservices from spiraling when the network inevitably decides to stop cooperating. Whether you are using unique idempotency keys in your database or implementing state checks to ensure a transition only happens once, the goal is the same: making sure a single retry doesn’t turn into a catastrophic duplicate transaction. It is about building systems that are resilient to the chaos of the real world, rather than assuming the world will always behave perfectly.
I know it feels like extra work. It’s tempting to skip the state checks and the key validation when you’re racing to hit a sprint deadline. But I’ve spent enough late nights staring at a terminal to know that the time you “save” now is usually paid back with interest during a production incident. Writing idempotent code is an investment in your own future sleep. Don’t just write code that works when things are perfect; write code that knows how to fail gracefully when things go wrong. Your future self—and whoever inherits your codebase after you—will thank you for it.
Frequently Asked Questions
How do I handle idempotency when the operation involves multiple database writes that aren't wrapped in a single transaction?
This is where things usually fall apart. If you can’t wrap everything in a single ACID transaction, you have to stop thinking about “all or nothing” and start thinking about “eventual consistency.” I usually reach for an outbox pattern or a state machine. You record your intent in a single table first, then move through the steps. If the process dies halfway through, your background worker looks at the state and knows exactly where to pick up.
At what point does storing idempotency keys become a storage bottleneck or a performance drag on the database?
It becomes a problem when your key-store grows faster than your cleanup strategy can handle. If you’re throwing every single request into a relational table without a TTL or a compaction strategy, you’ll eventually see latency spike during index lookups. I’ve seen it happen: the DB spends more time managing the index for these keys than actually serving the business logic. Use a fast, TTL-backed KV store like Redis for the keys, not your primary source of truth.
How do I deal with "partial success" scenarios where the side effect happened but the response never reached the client?
This is where the “ghost side effect” problem lives. The database updated, but the network died before the client got the `200 OK`. If the client retries without a unique idempotency key, you’re double-charging or double-shipping. I’ve learned you can’t just rely on the response; you have to make the client send a client-generated UUID with every intent. If the server sees that same UUID again, it shouldn’t re-run the logic—it should just return the cached success.
Is there a way to implement this without forcing every single developer on the team to manually check for keys in every service?
You don’t want to play that game. If you make every dev manually check for keys, someone will forget, and that’s when the double-billing happens.
