Kafka DLQ + Circuit Breaker Design
Distributed systems fail in unpredictable ways. Consumers crash, databases become unavailable, downstream services experience elevated latency, and network partitions occur without warning.
Rather than attempting to prevent every failure, Upblit's ingestion pipeline is designed around a different philosophy:
Failures are inevitable. Recovery should be automatic.
This post explores how Kafka, dead letter queues (DLQs), circuit breakers, and exponential backoff work together to create a resilient ingestion architecture capable of surviving transient and persistent failures.
High-Level Architecture
A simplified ingestion flow looks like this:
Producer
│
▼
Kafka Topic
│
▼
Consumer Group
│
├─────────────► PostgreSQL
│
├─────────────► Vector Embedding Service
│
└─────────────► Object Storage
│
▼
Success → Commit Offset
Failure
│
Retry → Backoff → Retry
│
Retries Exhausted
│
▼
Dead Letter Queue (DLQ)
The primary objective is to avoid data loss while preventing cascading failures across dependent services.
Why Kafka?
Kafka provides durable, ordered event streams and consumer groups that can scale horizontally.
Instead of processing requests synchronously, producers publish events and return immediately. Consumers independently process those events at their own pace.
This architecture offers:
- High throughput
- Replay capability
- Horizontal scalability
- Durable persistence
- Loose coupling between services
However, reliability depends heavily on how failures are handled.
Dead Letter Queues
Not every failed message should be discarded.
Some failures are temporary:
- Network interruptions
- Service restarts
- Database failovers
Others are permanent:
- Corrupted payloads
- Invalid schemas
- Missing required fields
Messages that repeatedly fail processing are redirected to a Dead Letter Queue (DLQ) instead of blocking the main stream.
Main Topic
│
▼
Consumer
│
┌────┴────┐
│ Success │
│ │
▼ ▼
Commit Retry
Offset │
▼
Retry Limit
│
▼
Dead Letter Queue
This ensures ingestion continues while problematic events are isolated for inspection.
Circuit Breakers Protect Dependencies
Retrying an unavailable service indefinitely can make outages worse.
For example, suppose the vector embedding service experiences elevated latency.
Without protection:
100 consumers
│
▼
100 retries
│
▼
500 retries
│
▼
1000 retries
The service becomes overwhelmed exactly when it is least capable of handling load.
Instead, Upblit wraps external dependencies in a circuit breaker.
The breaker has three states:
Closed
Requests flow normally.
Consumer → Service
Open
After repeated failures, requests fail immediately without contacting the dependency.
Consumer ──X──► Service
This prevents unnecessary load and reduces latency.
Half-Open
After a cooldown period, a limited number of requests are allowed through.
If they succeed consistently, the breaker closes.
If failures continue, it reopens.
This simple mechanism prevents cascading failures while allowing automatic recovery.
Failure Mode 1: Consumer Crash
A consumer process may terminate unexpectedly because of:
- Process panics
- Node failures
- Container eviction
- Out-of-memory conditions
Kafka's offset model makes recovery straightforward.
A message is processed only after successful completion.
Poll message
│
▼
Process
│
▼
Success?
│ │
Yes No
│ │
▼ ▼
Commit No Commit
Offset
If the consumer crashes before committing, another consumer in the group eventually reprocesses the message.
This provides at-least-once delivery, making idempotent handlers especially important.
Failure Mode 2: Database Unavailability
Persistent storage is often the most critical dependency.
Imagine PostgreSQL temporarily becomes unavailable.
Instead of dropping events:
- Processing fails.
- The offset remains uncommitted.
- The consumer retries with exponential backoff.
- The circuit breaker eventually opens.
- Once the database recovers, normal processing resumes.
Because offsets are committed only after successful writes, events remain safely stored in Kafka.
Failure Mode 3: Vector Service Downtime
Generating embeddings frequently depends on GPU-backed infrastructure or external APIs.
Potential issues include:
- Timeout errors
- Capacity exhaustion
- Rate limiting
- Model deployment failures
The ingestion worker retries transient failures with increasing delays.
If failures continue beyond configured limits, the event is moved into the DLQ for later replay instead of indefinitely blocking the partition.
This keeps ingestion throughput healthy while preserving failed events for investigation.
Exponential Backoff Strategy
Retries should not happen immediately.
Instead, Upblit spaces attempts using exponential backoff:
delay = min(base × 2^attempt, maxDelay)
Example configuration:
| Attempt | Delay |
|---|---|
| 1 | 2 seconds |
| 2 | 4 seconds |
| 3 | 8 seconds |
| 4 | 16 seconds |
| 5 | 32 seconds |
| 6+ | 60 seconds (capped) |
Pseudo-code:
func backoff(attempt int) time.Duration {
delay := base * (1 << attempt)
if delay > maxDelay {
delay = maxDelay
}
return delay
}
Exponential delays reduce pressure on failing services while giving transient issues time to recover naturally.
DLQ Replay Workflow
A dead letter queue is not the end of the pipeline.
Operators can inspect failed messages to determine the root cause.
Typical workflow:
- Detect repeated failures.
- Store the event in the DLQ.
- Fix the underlying issue.
- Replay messages into the primary topic.
- Process normally.
Because failed events are preserved, no data is silently discarded.
Idempotency Matters
Retries imply duplicate execution is possible.
Consumers should therefore behave idempotently.
For example:
INSERT INTO documents (
id,
tenant_id,
content
)
VALUES ($1, $2, $3)
ON CONFLICT (id)
DO NOTHING;
Similarly, updates should use deterministic identifiers or version checks so replaying the same Kafka message does not create inconsistent state.
Observability
A resilient pipeline requires visibility into failures.
Useful metrics include:
- Consumer lag
- Retry count
- DLQ growth rate
- Circuit breaker state
- Processing latency
- Failed dependency calls
- Success rate after replay
Alerting on these indicators allows operators to detect issues before customers notice degraded service.
Putting It All Together
The complete recovery flow can be summarized as follows:
Kafka Event
│
▼
Consumer
│
▼
Business Logic
│
├──────────────► PostgreSQL
│
├──────────────► Vector Service
│
└──────────────► Other Dependencies
│
▼
Success?
│ │
Yes No
│ │
▼ ▼
Commit Offset Retry
│
Exponential Backoff
│
Circuit Breaker Check
│
Retries Exhausted?
│ │
No Yes
│ │
▼ ▼
Retry Again DLQ
│
▼
Manual Replay
Final Thoughts
Building a reliable ingestion system is less about eliminating failures and more about designing for graceful degradation. Kafka provides durable event storage, dead letter queues isolate unrecoverable messages, circuit breakers prevent cascading outages, and exponential backoff gives dependent services room to recover.
Combined, these patterns allow Upblit's ingestion pipeline to withstand consumer crashes, database outages, and vector service downtime while preserving data integrity and maintaining steady throughput. Instead of treating failures as exceptional events, the system assumes they will happen and makes recovery a first-class part of the architecture.