A Small Cleanup Function Caused a Production Incident

When users started receiving duplicate notifications, the notification service was the obvious suspect.

It wasn’t.

The real issue was hiding inside a cleanup function that ran before every background job.

What started as a “duplicate notification” bug turned into a lesson about retries, background workers, and why the order of seemingly harmless cleanup code matters.

The Setup

Our application processes water quality tests asynchronously.

Once a test is completed, a PostgreSQL backed job queue takes over. A background worker generates an AI summary, creates a PDF report, and finally sends the completion notification to the user.

The processing pipeline looked roughly like this..

To avoid processing the same test multiple times, the worker also removes stale jobs for that test before continuing.

At least, that was the intention.

The Symptom

The first report came from a user who received the completion message twice.

Not the admission notification.

The final report notification.

At first, the obvious suspects were the notification providers.

Maybe WhatsApp retried the request.

Maybe our push notification service sent it twice.

Maybe the worker itself had executed twice.

The logs quickly ruled those out.

The notification service wasn’t sending duplicate requests.

Instead, two different background jobs were independently reaching the point where they dispatched the completion notification.

The real question became:

Why were two jobs processing the same test?

Following the Queue

Every completed test eventually becomes a raw pipeline job.

Before doing any expensive work, the worker was supposed to cancel stale jobs belonging to the same test.

The implementation looked roughly like this:

await waterQualityService.listenWaterQuality(payload);
await cancelStaleRawJobs(...);

Notice the problem?

The cleanup happened after the expensive work had already begun.

By the time the worker attempted to cancel stale jobs, every competing job had already entered listenWaterQuality(). If multiple raw jobs existed for the same test, they all began processing independently before any cleanup occurred.

Inside that function, the worker checks whether the AI summary and PDF already exist.

If they do, it assumes it’s recovering from a previous interrupted execution and safely continues.

Each of those stale jobs therefore reached the same conclusion:

The report already exists, so dispatch the completion notification.

What was actually happening

Multiple stale jobs.

Multiple completion notifications.

The cleanup happened too late to stop them.

A Second Problem Appeared

While investigating this, I found another issue.

The cleanup function itself occasionally crashed.

It queried stale jobs directly from PostgreSQL and passed the database’s numeric job identifier into the queue library’s cancellation API.

The library expected its own UUID-based identifier instead.

One incorrect assumption was enough to throw an exception before any useful work began.

The worker treated that exception as a job failure.

The queue retried the job.

The retry executed the same cleanup code.

The cleanup failed again.

Instead of cancelling stale jobs, the worker repeatedly failed before making progress.

That created a retry storm and left even more stale jobs waiting in the queue.

The duplicate notifications were no longer surprising.

The system was unintentionally giving multiple jobs the opportunity to reach the completion stage.

Rethinking the Cleanup

At this point I had two problems to solve.

The first was making stale job cancellation reliable.

The second was making sure stale jobs never reached the processing pipeline in the first place.

I considered fixing the queue library API usage and continuing to cancel jobs one at a time.

Instead, I chose a different approach.

The worker only needed to cancel jobs that hadn’t started executing, so I replaced the loop with a single SQL statement that atomically marked eligible jobs as cancelled.

More importantly, I changed when the cleanup happened.

Instead of this:

processJob();
cancelStaleJobs();

the worker now does this:

cancelStaleJobs();
processJob();
cancelStaleJobs();

How the updated flow prevents duplicate processing

The first cleanup removes stale jobs before they can execute.

The second cleanup catches any new stale jobs that might have been created while the worker was processing.

In practice, the first cleanup ensures stale jobs are cancelled before they can continue through the processing pipeline.

Everyone else is cancelled before they ever reach the notification stage.

Why I Chose SQL Instead of the Queue API

I don’t generally recommend bypassing a library’s public API. In most cases, fixing the identifier mismatch and continuing to use the supported cancellation API would have been the safer long-term choice.

In this particular case, though, the operation I needed was effectively a bulk state transition on queued jobs. A single SQL UPDATE allowed all eligible jobs to be marked as cancelled in one statement instead of cancelling them individually through the API.

The worker only needed to cancel jobs that were still waiting or retrying it never interrupted active work.

This made the cleanup atomic, idempotent, and removed the exception path that was causing retries.

The trade-off is tighter coupling to the queue library’s storage schema.

Normally I’d avoid that.

In this case, the simpler and more reliable cleanup was worth the maintenance cost.

What I Learned

A few lessons stood out from this incident.

1. The first symptom is rarely the real bug.

Users reported duplicate notifications.

The investigation uncovered two independent problems inside the job worker.

2. Ordering matters.

The cleanup logic itself wasn’t wrong.

Running it after the pipeline was.

Moving the exact same function earlier in the execution flow completely changed the system’s behavior.

3. Retries amplify deterministic failures.

Retrying a job is useful when the failure is temporary.

When the failure happens every single time, retries only create more work and leave the system further behind.

4. Understand the assumptions behind your libraries.

The queue library wasn’t broken.

My assumption about which identifier its API expected was.

That small misunderstanding was enough to trigger a much larger production issue.

The code changes themselves weren’t particularly large.

Most of the work went into understanding how two small issues interacted with each other before they became visible to users.

That’s the part of debugging production systems I enjoy the most.