It was 2:14 AM, the kind of hour where the only thing keeping you awake is the blue light of your monitor and the growing realization that your database is currently being hammered by a loop you didn’t write. I wasn’t looking at a complex architectural failure or some high-level distributed systems theory; I was staring at a single, mundane task that had somehow decided to execute itself twice, triggering a cascade of duplicate payments that would eventually haunt my sleep for a week. If you are currently staring at your logs trying to figure out how to debug a background job that runs twice, you don’t need a lecture on eventual consistency or a deep dive into the internals of your message broker. You just need to know where the leaks are actually happening.
I’m not going to sell you on a new library or a fancy observability suite that costs more than my monthly mortgage. Instead, I’m going to walk through the specific, messy ways I’ve seen this happen—from visibility timeouts to the classic mistake of assuming your code is idempotent just because you wrote it once. My goal is to give you a mental checklist of the actual failure points so that the next time this happens, you can fix it in ten minutes and get back to sleep.
Table of Contents
Chasing Ghosts in at Least Once Delivery Semantics
Most people start looking for bugs in their application logic, assuming they missed an `if` statement or a loop condition. That’s usually a waste of time. In most modern architectures, the culprit isn’t your code—it’s the fundamental reality of at-least-once delivery semantics. Most message brokers and task queues are designed to guarantee that a message gets delivered, but they make no promises about it happening exactly once. If a worker picks up a job but fails to send an acknowledgment back to the queue before the message queue visibility timeout expires, the broker assumes the worker died. It then puts that same job back on the queue for someone else to grab.
This is where the real headache starts. You end up in a loop where a job is processed, takes a long time to finish, and then gets picked up by a second worker while the first one is still grinding away. It looks like a ghost in the machine, but it’s actually just a standard distributed systems race condition. I’ve spent more nights than I’d like to admit staring at logs where two different containers were executing the exact same business logic simultaneously. If you aren’t designing your workers with the assumption that they will eventually see the same task twice, you aren’t building for production; you’re building for a happy path that doesn’t exist.
When Message Queue Visibility Timeouts Betray You
Most of the time, when a job runs twice, the culprit isn’t a bug in your logic, but a fundamental misunderstanding of your infrastructure. If you are using something like AWS SQS, you have to deal with the message queue visibility timeout. When a worker picks up a task, the queue doesn’t delete it immediately; it just hides it from other workers for a set period. If your processing logic takes longer than that timeout—maybe because of a sudden spike in database latency or a particularly heavy payload—the queue assumes the worker died. It makes the message visible again, and suddenly, a second worker grabs it while the first one is still grinding away.
This is where you run head-first into distributed systems race conditions. You end up with two different processes trying to mutate the same state simultaneously. I learned this the hard way at that insurance firm; we had a timeout set too aggressively for a report generation task, and we spent an entire afternoon figuring out why the same PDF was being emailed to clients multiple times. You can’t just “fix” the timeout by making it longer, though—that just moves the goalposts. The real solution is to stop relying on the queue to be perfect and start implementing an idempotent consumer pattern so that the second execution realizes the first one is already halfway home.
Five things I actually look for when the logs start lying to me
- Stop looking at the code and start looking at the idempotency key. If your job doesn’t check if it has already finished a specific unit of work before it starts, you aren’t debugging; you’re just waiting for the next duplicate.
- Check your visibility timeout against your actual execution time. If your job takes 45 seconds to run but your queue’s visibility timeout is set to 30, the queue thinks the worker died and hands the job to someone else. It’s not a ghost; it’s just bad math.
- Audit your retry policies for “silent” failures. A lot of times, a job isn’t running twice because of a bug, but because it’s failing halfway through, hitting a retry limit, and then getting re-queued by a separate dead-letter logic that isn’t playing nice with your primary worker.
- Trace the producer, not just the consumer. I’ve spent too many nights realizing the background job was fine, but the service sending the message was accidentally firing two separate events because of a poorly handled HTTP retry loop in the upstream service.
- Log the unique message ID from the queue provider, not just your internal application ID. If you can’t tell if two logs represent the same physical message being retried or two different messages with the same payload, you’re flying blind.
The Hard Truths
Stop assuming “exactly-once” delivery is a thing you can just toggle on; if your system relies on a job running precisely one time to maintain state, you’ve already lost.
If your visibility timeout is shorter than the time it takes to actually process the job, you aren’t debugging a bug, you’re debugging a race condition you built yourself.
Idempotency isn’t a “nice-to-have” architectural pattern for later—it is the only real defense against the inevitable reality of duplicate messages.
Stop Chasing the Ghost
At the end of the day, debugging duplicate jobs isn’t about finding a magical configuration setting; it’s about accepting that distributed systems are inherently messy. You can’t simply “fix” at-least-once delivery or prevent every possible network hiccup from triggering a retry. Instead, you have to design for the inevitability of the double-run. This means moving away from the hope that things will work perfectly and moving toward idempotency as a requirement, not an afterthought. Whether you are checking a unique constraint in your database or implementing a distributed lock, the goal is the same: making sure that when the second job arrives, it realizes the first one already did the work and quietly exits without making a mess.
I’ve spent far too many nights staring at logs trying to find the “why” behind a single duplicate execution, only to realize that the “why” doesn’t matter as much as the resilience of the system. We build these complex machines to do our work for us, but they will fail, they will retry, and they will occasionally trip over themselves. Don’t let the pursuit of a perfectly deterministic world burn you out. Build things that are robust enough to fail gracefully, write down what you learned when the pager goes off, and then go get some sleep. That is how you survive a career in production.
Frequently Asked Questions
If I can't change the "at-least-once" nature of my message queue, what's the most lightweight way to implement idempotency without adding massive latency?
If you can’t change the queue, change how you treat the incoming message. Don’t go for a massive distributed lock if you don’t have to. The most lightweight way is a simple “check-and-set” using your existing database. Grab a unique identifier from the message—a transaction ID or a UUID—and try to insert it into a dedicated `processed_jobs` table with a unique constraint. If the insert fails, the job is a duplicate. Stop there. Don’t run the business logic.
How do I distinguish between a genuine network retry and a worker process that just died halfway through a task?
To tell them apart, you have to look at the state of the world after the failure. If the job is a network retry, your idempotent logic should have already caught the first attempt and returned a success or a “already done” status. If the worker died halfway through, you’ll likely find a partial record in your database or a dangling side effect. Check your telemetry for the gap between “job started” and “job finished”—if that gap is a sudden silence, the process died.
Is there a way to trace a specific job ID through the entire lifecycle so I can actually see where the duplication event happened in the logs?
You can, but you have to stop relying on the job runner’s internal IDs. They’re often ephemeral or change when a job retries. Instead, I always inject a custom `correlation_id` at the very moment the intent to work is created—usually before it even hits the queue. I pass that ID through the message payload. If you log that specific ID at every hop—producer, queue, consumer, and database—you’ll see exactly where the ghost appears.
At what point does trying to make a job "exactly-once" become a case of diminishing returns compared to just making it safer to fail?
The moment you start chasing “exactly-once” delivery, you’ve already lost the battle. You’ll end up fighting distributed systems theory, adding massive latency, and introducing complex coordination logic that will inevitably fail in its own unique way. I’ve seen teams burn weeks trying to achieve mathematical perfection. Stop. It’s almost always better to accept that things will run twice and focus your energy on making your jobs idempotent. Make them safe to fail, not impossible to repeat.
