Controller

The controller coordinates the lifecycle of every e6 Ingestion Engine pipeline. It accepts pipeline definitions, compiles SQL, schedules work, coordinates checkpoints, applies supported configuration changes, and recovers pipelines after failures.

Responsibilities

The controller owns the control-plane decisions for a pipeline:

• Compilation - validates the pipeline definition and produces an executable dataflow plan
• Scheduling - requests worker capacity and assigns pipeline tasks to available workers
• Lifecycle management - starts, stops, restarts, and rescales pipelines
• Checkpoint coordination - creates consistent recovery points and coordinates transactional sink commits
• Health management - monitors running work and initiates recovery when a worker or task fails
• Status and metrics - exposes pipeline progress, errors, and operational metrics

Workers execute the dataflow itself. The controller does not sit in the record-processing path, so pipeline data continues to move directly between workers and external systems.

Durable control-plane state

e6 Ingestion Engine separates control-plane metadata from pipeline checkpoint data:

• A relational database stores pipeline definitions, lifecycle status, and checkpoint records.
• The configured checkpoint storage backend stores the operator state required to recover a pipeline.
• Live worker assignments and in-progress coordination state are rebuilt when the controller starts.

If the controller restarts, it reconciles persisted pipeline status with the available workers. Pipelines that were running are recovered from their latest successful checkpoint rather than from an incomplete checkpoint attempt.

Pipeline statuses

The API and console expose the following lifecycle statuses:

Public pipeline lifecycle

Status	Meaning
Created	The pipeline definition has been accepted and is waiting to be compiled.
Compiling	e6 Ingestion Engine is validating SQL and preparing the execution plan.
Scheduling	Worker capacity is being prepared and tasks are being assigned.
Running	All required tasks are active and the pipeline is processing data.
Recovering	e6 Ingestion Engine is restoring the pipeline after a worker or task failure.
Rescaling	The pipeline is applying a parallelism change.
Restarting	A user-requested restart is in progress.
CheckpointStopping	A graceful stop is waiting for its final checkpoint to complete.
Stopping	The pipeline is stopping without waiting for additional processing.
Finishing	Bounded sources have completed and downstream work is draining.
Finished	A bounded pipeline completed successfully.
Stopped	The pipeline was stopped by a user and can be started again.
Failed	The pipeline could not recover automatically and requires user action.

Intermediate statuses describe work in progress. Automation should wait for Running, Finished, Stopped, or Failed rather than assuming that a pipeline is ready immediately after an API request.

Scheduling

The scheduler translates a pipeline's parallelism and resource requirements into worker capacity. It is responsible for:

• starting enough workers for the requested workload
• assigning each operator subtask to a worker with available capacity
• replacing failed workers
• releasing capacity when a pipeline stops or finishes

e6 Ingestion Engine supports several deployment models:

Deployment model	Worker management
Kubernetes	Workers run as pods and use the resource settings supplied by the cluster configuration.
Node-based	Workers run on registered machines that advertise available task capacity.
Local process	Workers run as local processes for development and testing.
Embedded	Controller and worker execution share one process for single-process usage.

Scheduling is capacity-aware, but it does not change the logical semantics of the pipeline. Increasing parallelism creates more subtasks; it does not create independent copies of a pipeline. HTTP-source pipelines that need multiple independently addressable replicas use Pipeline Replica Sets.

Checkpoints and recovery

At the configured interval, the controller coordinates a checkpoint across all active tasks. A checkpoint becomes eligible for recovery only after every participating task has successfully reported its state.

For sinks that support transactional commits, the controller starts the commit step only after the checkpoint is complete. This keeps source progress, operator state, and committed sink output aligned to the same recovery point. See Checkpointing and exactly-once delivery for the processing guarantees and sink-specific behavior.

When a worker or task fails, e6 Ingestion Engine:

1. marks the running attempt as unhealthy;
2. stops or replaces affected worker capacity;
3. restores the pipeline from its latest successful checkpoint; and
4. resumes processing from the source positions recorded by that checkpoint.

Work performed after the last successful checkpoint may be replayed. Whether that replay remains exactly once at the destination depends on the source and sink guarantees described in the connector documentation.

Retryable failures are recovered automatically up to the configured restart limit. Fatal failures, or pipelines that exhaust that limit, enter Failed and remain there until a user restarts them or changes the configuration.

Restarting, stopping, and rescaling

• Safe restart - takes a final checkpoint before restarting. Use it when preserving the latest processed position is more important than restart speed.
• Force restart - skips the final checkpoint and recovers from the latest previously successful checkpoint. Recent work may be replayed.
• Graceful stop - takes a final checkpoint and then enters Stopped.
• Immediate stop - stops processing without waiting for a new checkpoint.
• Rescale - checkpoints the current execution, prepares capacity for the new parallelism, and restores state across the new task layout.

Configuration changes that can be applied safely at runtime are reflected through the pipeline lifecycle. Changes that require new task placement move the pipeline through an intermediate status such as Rescaling or Restarting; status and errors remain available through the API while the change is in progress.