Explore how Codily implements job dependencies using `dependency_ids` to create a lightweight Directed Acyclic Graph (DAG) scheduler. Learn how dependent jobs wait for prerequisites, how cycles are prevented, and how this approach compares to workflow engines like Airflow and Temporal while remaining simple and efficient

Job Dependencies — Building a Mini DAG Scheduler

Many background tasks are independent, but real-world systems often require ordered execution.

You cannot generate a report before importing data. You should not send a completion email before image processing finishes. A deployment pipeline may need to build, test, and validate artifacts before publishing.

Codily addresses these scenarios with job dependencies through the dependency_ids field, effectively creating a lightweight Directed Acyclic Graph (DAG) scheduler without the complexity of a dedicated workflow orchestration platform.

This article explores how the design works and where it fits in the broader ecosystem of workflow engines.

What Is a DAG?

A Directed Acyclic Graph is a collection of tasks connected by directed edges that never form cycles.

For example:

Import Data
      │
      ▼
Generate Embeddings
      │
      ▼
Index Documents
      │
      ▼
Notify User

Each node depends on the successful completion of its predecessor.

Unlike a simple queue, execution order is determined by dependency relationships rather than submission time.

Representing Dependencies

Codily stores prerequisite jobs using a dependency_ids field.

Example:

{
  "id": "job-index",
  "type": "index_documents",
  "dependency_ids": [
    "job-import",
    "job-embed"
  ]
}

This means job-index should execute only after both upstream jobs have completed successfully.

The dependency graph is therefore encoded directly in metadata rather than requiring a separate orchestration engine.

Submission Flow

Creating dependent jobs follows a simple pattern.

Client
    │
    ▼
Submit Job
    │
    ▼
Store dependency_ids
    │
    ▼
Persist in PostgreSQL
    │
    ▼
Enter Scheduling System

The scheduler or worker can later inspect those dependencies before execution.

Waiting Instead of Failing

When a worker dequeues a job, it first checks whether every dependency has reached a successful terminal state.

Pseudo-code:

func dependenciesSatisfied(job Job) bool {
    for _, dep := range job.DependencyIDs {
        if !isSuccessful(dep) {
            return false
        }
    }

    return true
}

If prerequisites are incomplete, the job simply waits rather than executing prematurely.

Conceptually:

Worker
   │
   ▼
Read Job
   │
   ▼
Dependencies Complete?
   │            │
  Yes           No
   │            │
   ▼            ▼
Execute     Wait / Requeue

This prevents downstream tasks from operating on incomplete or inconsistent state.

A Practical Example

Suppose an application ingests uploaded documents.

The workflow might look like:

Upload File
      │
      ▼
Extract Text
      │
      ▼
Generate Embeddings
      │
      ▼
Store Vector Index
      │
      ▼
Send Completion Email

Each stage depends on the previous one succeeding.

If embedding generation fails, indexing and notification remain blocked until recovery.

Supporting Parallelism

Dependencies are not limited to chains.

Multiple independent jobs can execute concurrently before converging into a shared downstream task.

           OCR
          /
Upload ──┤
          \
           Metadata Extraction

         │
         ▼

     Build Search Index

Here:

OCR runs independently.
Metadata extraction runs independently.
Index construction waits until both finish.

This naturally exposes available parallelism while preserving correctness.

How Dependent Jobs Wait

Codily does not need a heavyweight workflow engine to enforce ordering.

Instead, workers evaluate dependencies during scheduling or dequeue.

If upstream jobs remain incomplete:

execution is deferred,
the dependent job is left pending or requeued,
and workers continue processing other available tasks.

This avoids busy waiting and keeps compute resources focused on executable work.

Handling Failures

Dependencies also influence failure propagation.

Suppose:

Job A
   │
   ▼
Job B
   │
   ▼
Job C

If Job A fails permanently, Job B cannot safely execute.

Possible strategies include:

keeping downstream jobs pending,
marking them as blocked,
cancelling dependent execution,
or requiring manual intervention before retry.

The exact policy depends on application requirements, but dependency awareness ensures that invalid execution sequences are avoided.

Avoiding Cycles

The defining property of a DAG is that it contains no cycles.

An invalid graph would look like:

Job A
  ▲   │
  │   ▼
Job B

Neither task can begin because each depends on the other.

To prevent this, Codily validates dependency relationships before accepting them.

Typical checks include:

ensuring a job does not depend on itself,
rejecting references that create circular paths,
validating that prerequisite identifiers exist,
enforcing acyclic dependency chains.

By rejecting cycles during submission, the runtime avoids deadlocks and simplifies scheduling logic.

Worker Execution Logic

A simplified execution loop might resemble:

for {
    job := dequeue()

    if !dependenciesSatisfied(job) {
        requeue(job)
        continue
    }

    process(job)
}

The worker only performs business logic after verifying that prerequisites have completed successfully.

This keeps dependency enforcement centralized and predictable.

Why Not Build Everything in Airflow or Temporal?

Workflow orchestration platforms like Apache Airflow and Temporal provide significantly more functionality than a lightweight dependency model.

They support features such as:

rich workflow definitions,
long-running orchestration,
timers,
retries,
branching,
signals,
state persistence,
versioned workflows,
and visual execution graphs.

Those capabilities are invaluable for complex pipelines, but they also introduce operational overhead and conceptual complexity.

Codily intentionally focuses on a narrower use case: expressing straightforward execution order between jobs without requiring a separate orchestration layer.

Comparing the Approaches

Feature	Codily Dependencies	Full Workflow Engines
Simple prerequisite execution	✅	✅
Directed acyclic dependency graphs	✅	✅
Lightweight implementation	✅	❌
Minimal operational overhead	✅	❌
Complex branching logic	Limited	✅
Long-running orchestrations	Limited	✅
Human interaction steps	❌	✅
Visual workflow management	❌	Commonly available

For many applications, lightweight dependencies are sufficient and easier to maintain than adopting a comprehensive orchestration platform.

Scaling Dependency Checks

As dependency graphs grow, efficient lookups become increasingly important.

Common optimizations include:

indexing job identifiers,
caching terminal states,
batching dependency queries,
minimizing repeated database reads,
and storing only identifiers in queue messages while keeping authoritative state in PostgreSQL.

These techniques allow dependency evaluation to remain fast even when coordinating large numbers of jobs.

Real-World Use Cases

The dependency_ids model enables many practical workflows:

Build → Test → Deploy pipelines
File upload → Virus scan → Publish
Data import → Validation → Analytics
Image upload → Resize → CDN invalidation
Document ingestion → Embedding generation → Search indexing
Payment confirmation → Invoice generation → Customer notification

Each workflow benefits from explicit ordering while retaining the scalability of asynchronous processing.

Final Thoughts

By introducing dependency_ids, Codily transforms a traditional background job queue into a lightweight DAG scheduler capable of coordinating ordered execution across related tasks. Workers defer processing until prerequisites succeed, parallel stages can execute independently, and cycle detection ensures the dependency graph remains valid.

The result is a practical middle ground: enough orchestration to model real-world pipelines without the operational cost of adopting a full-fledged workflow engine like Airflow or Temporal. For many backend systems, this balance delivers the flexibility developers need while keeping the scheduling architecture simple, predictable, and easy to operate.

Job Dependencies in Codily: Building a Lightweight DAG Scheduler with `dependency_ids`