Idempotency Keys in Distributed Job Systems

One of the most overlooked reliability problems in distributed systems is duplicate requests.

A user clicks a button twice. A mobile client retries after a timeout. A load balancer retransmits a request. An API gateway retries after a transient network failure.

Without protection, these duplicate requests can produce duplicate side effects:

  • Two welcome emails
  • Two invoices
  • Two payment attempts
  • Two image processing jobs
  • Two webhook deliveries

Codily addresses this problem using idempotency keys on its POST /jobs endpoint. Rather than creating multiple jobs for the same logical operation, the system recognizes repeated submissions and safely returns the original job.

This article explains why idempotency matters and how it improves reliability in asynchronous architectures.

What Is Idempotency?

An operation is idempotent if performing it multiple times has the same effect as performing it once.

For example:

Create Job (first request)
        │
        ▼
Job Created

Create Job (same request again)
        │
        ▼
Return Existing Job

No duplicate processing occurs.

This differs from ordinary POST semantics, where every request typically creates a new resource.

Why Async Systems Need It

Background job systems intentionally decouple request submission from execution.

A client sends a request:

Client
   │
   ▼
POST /jobs
   │
   ▼
Queue
   │
   ▼
Worker

Now imagine the client loses its network connection before receiving the response.

From the client's perspective, it's impossible to know whether:

  • The server never received the request.
  • The server created the job successfully.
  • The response was lost after processing.

The safest client behavior is usually to retry.

Without idempotency, that retry creates another identical job.

The Idempotency-Key Header

Codily solves this by allowing callers to attach an Idempotency-Key header to job submissions.

Example:

POST /jobs
Idempotency-Key: order-42-email

The key identifies the logical operation rather than the HTTP request itself.

If the same client retries with the same key, Codily recognizes that the operation has already been accepted.

Deduplication Logic

Internally, the workflow is straightforward.

Incoming Request
        │
        ▼
Read Idempotency-Key
        │
        ▼
Lookup Existing Job
        │
 ┌──────┴────────┐
 │               │
Found         Not Found
 │               │
 ▼               ▼
Return      Create New
Existing       Job

The lookup typically uses a unique identifier scoped to the tenant, ensuring that two different organizations can safely reuse the same key without conflict.

A Simplified Database Model

A representative schema might include:

CREATE TABLE jobs (
    id UUID PRIMARY KEY,
    tenant_id TEXT NOT NULL,
    idempotency_key TEXT,
    status TEXT NOT NULL
);

To guarantee uniqueness within a tenant:

CREATE UNIQUE INDEX idx_jobs_tenant_idempotency
ON jobs (
    tenant_id,
    idempotency_key
);

This ensures that repeated submissions for the same logical operation resolve to a single job record.

What Happens on Duplicate Submission?

Suppose the first request succeeds:

POST /jobs
Key: invoice-501
        │
        ▼
Job Created
        │
        ▼
Job ID:
abc123

Later, the client retries with the same key:

POST /jobs
Key: invoice-501
        │
        ▼
Existing Job Found
        │
        ▼
Return Job ID:
abc123

No second job is inserted into the queue.

From the client's perspective, the retry succeeds while preserving exactly one logical operation.

Why This Matters in Practice

Preventing Duplicate Emails

Imagine a password reset workflow.

Without idempotency:

Retry
   │
   ▼
Two Email Jobs
   │
   ▼
User Receives
Two Emails

With idempotency:

Retry
   │
   ▼
Existing Job Returned
   │
   ▼
One Email Sent

The user experience remains consistent even if the client retries multiple times.

Preventing Duplicate Invoice Generation

Generating invoices often triggers downstream effects such as PDF creation, notifications, and accounting updates.

Creating two jobs accidentally could lead to duplicate records and reconciliation problems.

An idempotency key tied to the invoice identifier ensures that retries reuse the original job rather than generating another.

Preventing Duplicate Webhooks

Webhook producers commonly retry failed deliveries.

Without deduplication, downstream systems may process the same business event repeatedly.

Associating an idempotency key with each event lets consumers recognize retransmissions and avoid duplicate side effects.

Client Retries Become Safe

Many HTTP libraries automatically retry transient failures.

For example:

Request
   │
Timeout
   │
Retry
   │
Retry
   │
Retry

When each retry carries the same idempotency key, the server can safely return the original job rather than enqueueing multiple copies.

This transforms unreliable networks from a correctness problem into a simple availability concern.

Relationship to At-Least-Once Delivery

Distributed queues frequently provide at-least-once delivery, meaning a message may be processed more than once.

Idempotency complements this model.

Even if:

  • the client retries,
  • the queue redelivers,
  • or the worker restarts after a crash,

the logical operation remains consistent because duplicate requests resolve to the same underlying work item or are processed safely.

Choosing Good Idempotency Keys

Keys should represent the business operation rather than a random retry attempt.

Examples:

order-982-confirmation-email

payment-50021

invoice-2026-00045

user-123-password-reset

Poor choices include timestamps or randomly generated values that change with every retry, since those defeat deduplication entirely.

Handling Expiration

Some systems retain idempotency keys indefinitely, while others expire them after a configurable period.

Retention policies depend on business requirements:

  • Payment operations may require long-lived guarantees.
  • Temporary background jobs may only need protection for several days.
  • Event replay systems may archive historical mappings separately.

The key principle is that retries occurring within the expected retry window should resolve to the original operation.

Designing Idempotent Workers

Submission deduplication is only part of the story.

Workers should also be written to tolerate repeated execution where practical.

For example:

INSERT INTO emails (
    message_id,
    recipient
)
VALUES ($1, $2)
ON CONFLICT (message_id)
DO NOTHING;

Using deterministic identifiers or conflict-aware writes prevents duplicate side effects if a task is replayed.

Final Thoughts

Retries are an unavoidable part of distributed systems. Networks fail, clients disconnect, and intermediaries resend requests. Without safeguards, those retries can produce duplicate jobs and unintended side effects.

Codily's Idempotency-Key support on POST /jobs provides a simple but powerful solution: identify the logical operation, deduplicate repeated submissions, and return the original job instead of creating a new one. Whether the task is sending an email, generating an invoice, or processing a webhook, idempotency transforms retries from a source of inconsistency into a safe and expected part of the system's reliability strategy.