How Codily Implements Delayed Job Scheduling with Redis ZSETs

Background job systems rarely execute every task immediately. Some jobs must wait for a scheduled time, while others need delayed retries after transient failures. Designing this efficiently requires more than a simple FIFO queue.

Codily solves this problem by combining Redis Sorted Sets (ZSETs) with a lightweight scheduler goroutine that periodically promotes ready jobs into execution queues. This approach keeps scheduling fast, scalable, and simple while avoiding unnecessary polling overhead.

In this article, we'll explore the design in detail, including timestamp scoring, atomic promotion, race condition avoidance, and exponential retry backoff.

---

Why a Sorted Set?

Redis provides multiple data structures, but delayed scheduling maps naturally to a Sorted Set.

Each member is stored alongside a numeric score:

Member: job_uuid
Score : scheduled_execution_time

Codily stores delayed jobs in a ZSET where the score is the Unix epoch timestamp in nanoseconds.

For example:

codily:delayed

Score (Unix ns)           Member
──────────────────────────────────────────────
1750000000000000000   →   job-a
1750000002000000000   →   job-b
1750000005000000000   →   job-c

Because Redis automatically keeps entries ordered by score, the scheduler can efficiently identify jobs whose execution time has arrived.

---

Why Unix Epoch Nanoseconds?

Many schedulers operate with second-level precision, but Codily stores timestamps using nanoseconds.

Advantages include:

• Extremely fine scheduling resolution
• Deterministic ordering for closely timed jobs
• Compatibility with Go's time.UnixNano()
• Avoidance of rounding errors during calculations

Conceptually:

readyAt := time.Now().Add(delay).UnixNano()

The resulting integer becomes the ZSET score.

---

Overall Scheduling Flow

The architecture separates delayed storage from execution queues.

Client
    │
    ▼
Submit Job
    │
    ▼
Redis ZSET (Delayed Queue)
    │
    │  Every 2 Seconds
    ▼
Scheduler Goroutine
    │
    ▼
Promote Ready Jobs
    │
    ▼
Priority Redis Lists
    │
    ▼
Worker Pool
    │
    ▼
Execute Job

Workers never poll the delayed set directly. Instead, they consume only from ready queues, simplifying processing logic.

---

Scheduler Goroutine

A dedicated scheduler runs every two seconds.

Its responsibility is straightforward:

1. Read jobs whose scheduled time has passed.
2. Remove them from the delayed ZSET.
3. Push them into the appropriate execution queue.

Pseudo-code:

ticker := time.NewTicker(2 * time.Second)

for range ticker.C {
    now := time.Now().UnixNano()

    jobs := redis.ZRangeByScore(
        "codily:delayed",
        "-inf",
        strconv.FormatInt(now, 10),
    )

    promote(jobs)
}

The two-second interval provides a practical balance between responsiveness and minimal Redis overhead.

---

Selecting Ready Jobs

Redis efficiently retrieves eligible jobs using score ranges.

ZRANGEBYSCORE codily:delayed -inf current_time

This returns only jobs scheduled at or before the current instant.

Example:

Current Time = 500

Delayed Queue

Score   Job
---------------
300     A
450     B
650     C
900     D

Only A and B are promoted during this scheduler iteration.

---

Atomic Promotion with ZREM + RPUSH

Promotion consists of two logical operations:

1. Remove the job from the delayed set.
2. Push the job ID into the execution queue.

Conceptually:

ZREM  delayed job_id
RPUSH ready_queue job_id

In Codily, these operations are issued together using a Redis pipeline, minimizing network round trips and ensuring the scheduler processes promotions as one coordinated unit.

Pseudo-code:

pipe.ZRem(ctx, "codily:delayed", jobID)
pipe.RPush(ctx, "codily:jobs:medium", jobID)

_, err := pipe.Exec(ctx)

The pipeline batches commands efficiently while maintaining their intended ordering.

---

Avoiding Race Conditions

Without coordinated promotion logic, multiple scheduler instances could attempt to process the same delayed entry.

Consider this unsafe sequence:

Scheduler A → Reads job X

Scheduler B → Reads job X

Scheduler A → Pushes X

Scheduler B → Pushes X

The result would be duplicate execution.

By removing promoted entries from the delayed set as part of the promotion workflow before subsequent scheduler passes observe them, Codily minimizes the window in which duplicate promotion can occur. Combined with idempotent job processing and centralized scheduling responsibility, this design avoids practical race conditions while remaining lightweight.

---

Worker Queues Remain Simple

Execution queues are ordinary Redis Lists.

Workers block on them using commands similar to:

BLPOP

Once a delayed job is promoted, it becomes indistinguishable from any immediately submitted job.

This separation keeps worker logic focused entirely on execution rather than scheduling decisions.

---

Delayed Retries

The same delayed queue powers retry behavior.

Suppose a transient dependency fails.

Instead of immediately retrying:

Failure
    │
    ▼
Delayed Queue
    │
(wait)
    │
    ▼
Scheduler
    │
    ▼
Ready Queue
    │
    ▼
Retry

The retry delay is encoded directly into the ZSET score.

No additional scheduling subsystem is required.

---

Exponential Backoff

Retry timing follows exponential growth:

delay = min(base × 2^retry_count, max_delay)

For example:

Retry	Delay
1	2 seconds
2	4 seconds
3	8 seconds
4	16 seconds
5	32 seconds
6+	60 seconds (capped)

A representative implementation:

func retryDelay(retryCount int) time.Duration {
    delay := baseDelay * (1 << retryCount)

    if delay > maxDelay {
        return maxDelay
    }

    return delay
}

The scheduler computes:

nextExecution := time.Now().
    Add(retryDelay(attempt)).
    UnixNano()

and inserts the job back into the delayed ZSET with that score.

---

Why Not Sleep in Workers?

An alternative design would have workers sleep until retry time.

That approach has drawbacks:

• Worker threads remain occupied.
• Memory usage increases with pending retries.
• Large retry volumes reduce throughput.
• Horizontal scaling becomes harder.

Using Redis as the scheduling source allows workers to remain free until work is actually ready.

---

Complexity and Performance

Redis Sorted Sets provide efficient ordering operations.

Typical characteristics include:

• ZADD → O(log N)
• ZREM → O(log N)
• ZRANGEBYSCORE → O(log N + M), where M is the number of returned jobs

This makes the structure suitable even for large delayed queues, as retrieval cost grows logarithmically with queue size.

---

Failure Recovery

If the scheduler process restarts unexpectedly, delayed jobs remain safely stored in Redis.

On restart:

1. The scheduler resumes polling.
2. Expired scores are detected.
3. Ready jobs are promoted.
4. Workers continue processing.

Because scheduling state lives in Redis rather than process memory, recovery is straightforward and does not require rebuilding timers.

---

Final Thoughts

Codily's delayed scheduling system demonstrates how a small set of Redis primitives can implement reliable, scalable background execution. By storing execution timestamps as Unix epoch nanoseconds in a Sorted Set, scanning every two seconds for ready jobs, and promoting them through coordinated ZREM + RPUSH operations, the scheduler cleanly separates timing concerns from execution.

The same mechanism also powers exponential retry backoff, enabling transient failures to recover automatically without blocking workers or introducing unnecessary complexity. The result is a lightweight scheduling architecture that remains predictable, efficient, and easy to reason about even as workloads scale.