Skip to main content
The controller coordinates the lifecycle of every Ingest Engine pipeline. It accepts pipeline definitions, compiles SQL, schedules work, coordinates checkpoints, applies supported configuration changes, and recovers pipelines after failures.

​
Responsibilities

The controller owns the control-plane decisions for a pipeline:
  • Compilation - validates the pipeline definition and produces an executable dataflow plan
  • Scheduling - requests worker capacity and assigns pipeline tasks to available workers
  • Lifecycle management - starts, stops, restarts, and rescales pipelines
  • Checkpoint coordination - creates consistent recovery points and coordinates transactional sink commits
  • Health management - monitors running work and initiates recovery when a worker or task fails
  • Status and metrics - exposes pipeline progress, errors, and operational metrics
Workers execute the dataflow itself. The controller does not sit in the record-processing path, so pipeline data continues to move directly between workers and external systems.

​
Durable control-plane state

Ingest Engine separates control-plane metadata from pipeline checkpoint data:
  • A relational database stores pipeline definitions, lifecycle status, and checkpoint records.
  • The configured checkpoint storage backend stores the operator state required to recover a pipeline.
  • Live worker assignments and in-progress coordination state are rebuilt when the controller starts.
If the controller restarts, it reconciles persisted pipeline status with the available workers. Pipelines that were running are recovered from their latest successful checkpoint rather than from an incomplete checkpoint attempt.

​
Pipeline statuses

The API and console expose the following lifecycle statuses: Public pipeline lifecycle Intermediate statuses describe work in progress. Automation should wait for Running, Finished, Stopped, or Failed rather than assuming that a pipeline is ready immediately after an API request.

​
Scheduling

The scheduler translates a pipeline’s parallelism and resource requirements into worker capacity. It is responsible for:
  • starting enough workers for the requested workload
  • assigning each operator subtask to a worker with available capacity
  • replacing failed workers
  • releasing capacity when a pipeline stops or finishes
Ingest Engine supports several deployment models: Scheduling is capacity-aware, but it does not change the logical semantics of the pipeline. Increasing parallelism creates more subtasks; it does not create independent copies of a pipeline. HTTP-source pipelines that need multiple independently addressable replicas use Pipeline Replica Sets.

​
Checkpoints and recovery

At the configured interval, the controller coordinates a checkpoint across all active tasks. A checkpoint becomes eligible for recovery only after every participating task has successfully reported its state. For sinks that support transactional commits, the controller starts the commit step only after the checkpoint is complete. This keeps source progress, operator state, and committed sink output aligned to the same recovery point. See Checkpointing and exactly-once delivery for the processing guarantees and sink-specific behavior. When a worker or task fails, Ingest Engine:
  1. marks the running attempt as unhealthy;
  2. stops or replaces affected worker capacity;
  3. restores the pipeline from its latest successful checkpoint; and
  4. resumes processing from the source positions recorded by that checkpoint.
Work performed after the last successful checkpoint may be replayed. Whether that replay remains exactly once at the destination depends on the source and sink guarantees described in the connector documentation. Retryable failures are recovered automatically up to the configured restart limit. Fatal failures, or pipelines that exhaust that limit, enter Failed and remain there until a user restarts them or changes the configuration.

​
Restarting, stopping, and rescaling

  • Safe restart - takes a final checkpoint before restarting. Use it when preserving the latest processed position is more important than restart speed.
  • Force restart - skips the final checkpoint and recovers from the latest previously successful checkpoint. Recent work may be replayed.
  • Graceful stop - takes a final checkpoint and then enters Stopped.
  • Immediate stop - stops processing without waiting for a new checkpoint.
  • Rescale - checkpoints the current execution, prepares capacity for the new parallelism, and restores state across the new task layout.
Configuration changes that can be applied safely at runtime are reflected through the pipeline lifecycle. Changes that require new task placement move the pipeline through an intermediate status such as Rescaling or Restarting; status and errors remain available through the API while the change is in progress.
Last modified on October 1, 2026