Learn how Codily implements a scalable worker pool in Go using configurable goroutines, Redis `BLPOP`, PostgreSQL job state management, and retry logic. This beginner-to-intermediate guide walks through the architecture and includes a simplified implementation of the core worker loop.

Building a Go Worker Pool from Scratch

Many backend applications need to perform work that shouldn't block an HTTP request. Sending emails, resizing images, processing uploads, or transforming data are all excellent candidates for background execution.

One common solution is a worker pool: a set of long-running processes that continuously consume jobs from a queue and execute them independently of user requests.

Codily implements this pattern using Go goroutines, Redis queues, and PostgreSQL as the source of truth. The result is a lightweight yet reliable architecture capable of handling concurrent workloads while tracking every job's lifecycle.

This article explains how the system works and provides a simplified implementation suitable for backend engineers learning Go concurrency.

Why Use a Worker Pool?

Imagine an API endpoint that sends an email.

Without background processing:

Client
   │
   ▼
HTTP Request
   │
   ▼
Send Email
   │
(wait...)
   │
   ▼
Return Response

The user waits for the email operation to complete.

With a worker pool:

Client
   │
   ▼
HTTP Request
   │
   ▼
Create Job
   │
   ▼
Return Immediately

Worker Pool
     │
     ▼
Execute Email

The API remains responsive while workers handle expensive tasks asynchronously.

Core Components

Codily's worker architecture consists of four major pieces:

API server that creates jobs
Redis for queueing job identifiers
PostgreSQL for durable job state
Worker pool that executes queued work

A simplified flow looks like this:

API
 │
 ▼
PostgreSQL
 │
 ▼
Redis Queue
 │
 ▼
Go Worker Pool
 │
 ▼
Job Processor
 │
 ▼
Update Status

Redis stores lightweight queue entries, while PostgreSQL keeps authoritative metadata and execution history.

Configurable Concurrency

Instead of spawning unlimited goroutines, Codily uses a configurable worker count.

WORKER_CONCURRENCY = 5

Worker 1
Worker 2
Worker 3
Worker 4
Worker 5

At startup, the application launches exactly that many goroutines.

A simplified example:

for i := 0; i < workerConcurrency; i++ {
	go worker(ctx)
}

This makes throughput predictable and prevents uncontrolled resource usage.

Reading Jobs with `BLPOP`

Workers obtain work from Redis using the blocking list operation BLPOP.

Unlike polling in a tight loop, BLPOP waits until work becomes available.

Conceptually:

Worker
   │
   ▼
BLPOP Queue
   │
   ▼
Wait...
   │
Job Arrives
   │
   ▼
Return Job ID

Because workers sleep while queues are empty, CPU utilization remains low.

Codily can also prioritize queues by listening to multiple lists in order, ensuring high-priority jobs are consumed before lower-priority work.

PostgreSQL as the Source of Truth

Redis stores only queue entries.

The complete job definition—including payload, retry count, timestamps, and status—is persisted in PostgreSQL.

Once a worker receives a job identifier, it loads the corresponding record:

Redis
  │
  ▼
Job ID
  │
  ▼
PostgreSQL Lookup
  │
  ▼
Full Job Record

This design keeps queue messages lightweight while ensuring durable state survives restarts.

Marking Jobs as Running

Before processing begins, the worker updates the database to reflect ownership of the task.

For example:

UPDATE jobs
SET
    status = 'running',
    started_at = NOW()
WHERE id = $1;

Recording this transition enables:

execution tracking,
timeout detection,
monitoring dashboards,
and recovery after worker failures.

The database always reflects the current lifecycle stage of every job.

The Core Worker Loop

At its heart, a worker repeatedly:

waits for a queued job,
loads it,
marks it as running,
executes business logic,
records the outcome.

A simplified implementation might look like:

func worker(ctx context.Context) {
	for {
		jobID, err := dequeue(ctx)
		if err != nil {
			continue
		}

		job, err := loadJob(ctx, jobID)
		if err != nil {
			continue
		}

		markRunning(ctx, job.ID)

		if err := process(job); err != nil {
			handleFailure(ctx, job)
			continue
		}

		markSuccess(ctx, job.ID)
	}
}

Real production systems add logging, metrics, cancellation, retries, and timeout handling, but the overall structure remains remarkably simple.

Handling Success

When processing completes successfully, the worker records the terminal state.

UPDATE jobs
SET
    status = 'success',
    completed_at = NOW()
WHERE id = $1;

At this point, no further retries are necessary.

The job remains in the database for auditing and inspection.

Handling Failures

Failures do not necessarily indicate permanent problems.

Temporary issues include:

network interruptions,
unavailable external APIs,
transient database errors,
or rate limiting.

Instead of giving up immediately, the worker evaluates whether the error is retryable.

Failure
   │
   ▼
Retryable?
 │        │
Yes       No
 │         │
 ▼         ▼
Retry    Mark Failed

This distinction significantly improves system resilience.

Retry with Exponential Backoff

Repeatedly retrying a failing dependency can make outages worse.

Instead, Codily spaces retries using exponential backoff.

The delay is computed as:

delay = min(base × 2^retryCount, maxDelay)

Example progression:

Attempt	Delay
1	2 seconds
2	4 seconds
3	8 seconds
4	16 seconds
5	32 seconds
6+	60 seconds (capped)

After calculating the delay, the job is rescheduled rather than executed immediately.

Recovering from Worker Crashes

Suppose a worker crashes midway through processing.

Because job state is stored in PostgreSQL and queue state lives in Redis, recovery is possible.

A scheduler can detect stale running jobs and return them to the queue for another attempt.

Worker Crash
      │
      ▼
Job Stuck?
      │
      ▼
Recovery Process
      │
      ▼
Requeue
      │
      ▼
Another Worker Executes

This ensures transient infrastructure failures do not permanently lose work.

Scaling the Pool

Increasing throughput is straightforward.

Simply increase WORKER_CONCURRENCY or deploy additional worker processes.

Queue

 │
 ▼

Worker A
Worker B
Worker C
Worker D
Worker E

Because each worker operates independently, the system scales horizontally with minimal coordination.

Why Goroutines Work So Well

Go's lightweight concurrency primitives make worker pools particularly elegant.

Unlike operating system threads, goroutines have small initial memory footprints and are multiplexed efficiently by the runtime scheduler.

This allows applications to maintain many concurrent workers without excessive overhead, while channels, contexts, and synchronization primitives provide a clean foundation for cancellation and coordination.

Final Thoughts

A worker pool is one of the most useful patterns in backend engineering, and Go provides an excellent environment for implementing it. Codily's design combines configurable goroutine concurrency, Redis BLPOP queues, PostgreSQL-backed job state, and retry-aware execution to create a reliable asynchronous processing system.

By separating queue management from durable state and keeping the core execution loop simple, developers gain a scalable architecture that is easy to understand, monitor, and extend. For engineers learning Go concurrency, building a worker pool like this is an excellent introduction to practical distributed systems design.

Building a Go Worker Pool from Scratch: How Codily Processes Background Jobs