A deep dive into Codily's webhook architecture for notifying external systems when jobs reach terminal states. Learn about the webhook payload format, supported events, the 5-second timeout with no retry policy, and why a fire-and-forget approach is often the right tradeoff for asynchronous job processing systems.

Designing Webhooks for Job State Changes

Background job systems rarely operate in isolation. Once a task finishes, fails, or times out, another system often needs to know about it.

An e-commerce platform may wait for invoice generation before notifying a customer. A deployment pipeline might trigger the next stage after a build completes. A monitoring dashboard may need to update its status in real time.

Rather than forcing clients to continuously poll for updates, Codily provides webhook notifications whenever a job reaches a terminal state.

This article explains how the webhook system works, why it only emits terminal events, and why a fire-and-forget delivery model is sufficient for many asynchronous workloads.

Why Use Webhooks?

Without webhooks, clients repeatedly ask the API whether anything has changed.

Client
   │
   ├── GET /jobs/123
   ├── GET /jobs/123
   ├── GET /jobs/123
   ├── GET /jobs/123
   ▼
Server

This polling wastes bandwidth and compute.

With webhooks:

Client
   │
Submit Job
   │
   ▼

Codily
   │
(wait)
   │
Job Completes
   │
   ▼
POST Webhook

The consumer receives updates only when meaningful events occur.

Terminal States Only

Codily sends notifications when jobs enter one of several terminal states:

completed
failed
dead_lettered
timeout
cancelled

These represent the final outcome of execution.

Pending
    │
    ▼
Running
    │
 ┌──┼─────────────┬───────────────┐
 ▼  ▼             ▼               ▼

Completed   Failed   Dead Lettered

Timeout     Cancelled

Because these states conclude processing, downstream systems can safely react without waiting for additional transitions.

Configuring a Webhook

When submitting a job, clients may provide a callback endpoint.

Example:

{
  "type": "email",
  "payload": {
    "to": "user@example.com"
  },
  "webhook_url": "https://example.com/webhooks/jobs"
}

The worker stores this URL alongside the job metadata and invokes it once processing reaches a terminal state.

Payload Shape

A webhook request includes both metadata and the final job state.

A representative payload looks like:

{
  "event": "job.completed",
  "job": {
    "id": "123",
    "status": "success"
  },
  "duration_ms": 312,
  "message": "job completed successfully"
}

Including the complete job object allows receivers to process the notification without making an additional lookup request in many cases.

Supported Events

Codily emits distinct event types corresponding to terminal outcomes.

For example:

job.completed

job.failed

job.dead_lettered

job.timeout

job.cancelled

Consumers can selectively subscribe to events that matter for their workflows while ignoring others.

Sending the Notification

Once processing finishes, the worker performs an asynchronous HTTP POST to the configured endpoint.

Conceptually:

Worker
   │
   ▼
Job Finishes
   │
   ▼
HTTP POST
   │
   ▼
External System

Importantly, webhook delivery occurs outside the critical execution path so that notification latency does not delay worker availability.

The 5-Second Timeout

Webhook requests use a short timeout window.

POST Webhook

Maximum Wait:
5 Seconds

If the remote endpoint does not respond within five seconds, the request is abandoned.

This prevents slow or unavailable receivers from tying up worker resources indefinitely.

In distributed systems, bounded waiting is often preferable to unlimited retries.

No Automatic Retries

One notable design decision is that Codily does not retry failed webhook deliveries.

If delivery fails because of:

network issues,
receiver downtime,
DNS failures,
or timeouts,

the worker logs the failure and continues.

Conceptually:

POST Webhook
      │
      ▼
Success?
 │          │
Yes         No
 │           │
 ▼           ▼
Done      Log Failure

No retry queue or replay mechanism is involved.

Why "Fire and Forget" Works Here

At first glance, guaranteed delivery sounds preferable.

However, guarantees introduce significant complexity:

retry scheduling,
duplicate delivery handling,
replay infrastructure,
exponential backoff,
persistence,
and delivery tracking.

For many job-processing systems, those costs outweigh the benefits.

The webhook acts primarily as a convenience notification, not the source of truth.

The Job Record Remains Authoritative

Even if webhook delivery fails, the underlying job state remains safely persisted.

PostgreSQL
     │
     ▼
Final Job State
     │
     ▼
Webhook Sent

External systems can always query the API directly if they need confirmation or reconciliation.

Because durable state exists elsewhere, occasional missed notifications do not imply data loss.

Polling as a Fallback

Clients that require stronger guarantees can combine webhooks with periodic reconciliation.

Webhook
    │
    ▼
Received?

   │        │

 Yes       No
  │         │
  ▼         ▼
Done   Poll API Later

This hybrid model provides responsive updates while preserving correctness even when notifications are lost.

Why Guaranteed Delivery Is Expensive

Building a guaranteed webhook system typically requires:

persistent outbound queues,
acknowledgement tracking,
retry policies,
dead-letter handling,
deduplication,
exponential backoff,
replay tooling,
and idempotent receivers.

In effect, the notification subsystem begins to resemble another distributed messaging platform.

Unless the business domain truly requires these guarantees, simpler architectures often prove easier to operate.

Designing Idempotent Consumers

Although Codily avoids retries, webhook receivers should still assume duplicate events are possible due to network behavior or manual replay.

A common pattern is to process events by unique job identifier:

Receive Event
      │
      ▼
Seen Before?
 │          │
No         Yes
 │           │
 ▼           ▼
Process    Ignore

Idempotent consumers remain correct even if the same notification arrives multiple times.

Real-World Use Cases

Terminal-state webhooks are useful for many workflows:

Sending user notifications after report generation
Triggering downstream ETL pipelines
Updating dashboards
Informing deployment systems
Synchronizing CRM records
Recording audit events
Starting dependent business processes

Because notifications occur asynchronously, client applications remain decoupled from worker execution.

Simplicity as an Architectural Choice

One lesson from building asynchronous systems is that not every component needs exactly-once guarantees.

Sometimes the correct design is simply:

Persist authoritative state.
Attempt notification.
Log failures.
Continue processing.

By keeping webhook delivery lightweight, Codily minimizes operational overhead while still providing timely updates for most integrations.

Final Thoughts

Codily's webhook system demonstrates a pragmatic approach to event notifications. When jobs reach terminal states—whether completed, failed, dead_lettered, timeout, or cancelled—the platform sends an asynchronous HTTP POST containing the final job state and metadata. Deliveries are bounded by a five-second timeout and intentionally omit automatic retries, embracing a fire-and-forget philosophy.

This design works because the webhook is a notification mechanism rather than the canonical source of truth. The durable job record remains stored elsewhere, allowing clients to reconcile state through the API if necessary. By avoiding the complexity of guaranteed delivery while preserving a clear and consistent event model, Codily achieves a practical balance between reliability, simplicity, and operational efficiency.

Designing Webhooks for Job State Changes: Why Codily Uses Fire-and-Forget Notifications