> For the complete documentation index, see [llms.txt](https://docs.e6data.com/ingestion-engine/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.e6data.com/ingestion-engine/architecture/controller.md). # Controller The controller coordinates the lifecycle of every e6 Ingestion Engine pipeline. It accepts pipeline definitions, compiles SQL, schedules work, coordinates checkpoints, applies supported configuration changes, and recovers pipelines after failures. ## Responsibilities The controller owns the control-plane decisions for a pipeline: * **Compilation** - validates the pipeline definition and produces an executable dataflow plan * **Scheduling** - requests worker capacity and assigns pipeline tasks to available workers * **Lifecycle management** - starts, stops, restarts, and rescales pipelines * **Checkpoint coordination** - creates consistent recovery points and coordinates transactional sink commits * **Health management** - monitors running work and initiates recovery when a worker or task fails * **Status and metrics** - exposes pipeline progress, errors, and operational metrics Workers execute the dataflow itself. The controller does not sit in the record-processing path, so pipeline data continues to move directly between workers and external systems. ## Durable control-plane state e6 Ingestion Engine separates control-plane metadata from pipeline checkpoint data: * A relational database stores pipeline definitions, lifecycle status, and checkpoint records. * The configured checkpoint storage backend stores the operator state required to recover a pipeline. * Live worker assignments and in-progress coordination state are rebuilt when the controller starts. If the controller restarts, it reconciles persisted pipeline status with the available workers. Pipelines that were running are recovered from their latest successful checkpoint rather than from an incomplete checkpoint attempt. ## Pipeline statuses The API and console expose the following lifecycle statuses: ![Public pipeline lifecycle](https://1725648790-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FuztQ31u5OyLCkPgUTosp%2Fuploads%2Fgit-blob-599afe13073886442eeb826cae2b1ea3a5cd64e8%2Fcontroller-diagrams-pipeline-state-machine.svg?alt=media) | Status | Meaning | | ---------------------- | ----------------------------------------------------------------------------- | | **Created** | The pipeline definition has been accepted and is waiting to be compiled. | | **Compiling** | e6 Ingestion Engine is validating SQL and preparing the execution plan. | | **Scheduling** | Worker capacity is being prepared and tasks are being assigned. | | **Running** | All required tasks are active and the pipeline is processing data. | | **Recovering** | e6 Ingestion Engine is restoring the pipeline after a worker or task failure. | | **Rescaling** | The pipeline is applying a parallelism change. | | **Restarting** | A user-requested restart is in progress. | | **CheckpointStopping** | A graceful stop is waiting for its final checkpoint to complete. | | **Stopping** | The pipeline is stopping without waiting for additional processing. | | **Finishing** | Bounded sources have completed and downstream work is draining. | | **Finished** | A bounded pipeline completed successfully. | | **Stopped** | The pipeline was stopped by a user and can be started again. | | **Failed** | The pipeline could not recover automatically and requires user action. | Intermediate statuses describe work in progress. Automation should wait for `Running`, `Finished`, `Stopped`, or `Failed` rather than assuming that a pipeline is ready immediately after an API request. ## Scheduling The scheduler translates a pipeline's parallelism and resource requirements into worker capacity. It is responsible for: * starting enough workers for the requested workload * assigning each operator subtask to a worker with available capacity * replacing failed workers * releasing capacity when a pipeline stops or finishes e6 Ingestion Engine supports several deployment models: | Deployment model | Worker management | | ----------------- | ---------------------------------------------------------------------------------------- | | **Kubernetes** | Workers run as pods and use the resource settings supplied by the cluster configuration. | | **Node-based** | Workers run on registered machines that advertise available task capacity. | | **Local process** | Workers run as local processes for development and testing. | | **Embedded** | Controller and worker execution share one process for single-process usage. | Scheduling is capacity-aware, but it does not change the logical semantics of the pipeline. Increasing parallelism creates more subtasks; it does not create independent copies of a pipeline. HTTP-source pipelines that need multiple independently addressable replicas use [Pipeline Replica Sets](/ingestion-engine/architecture/pipeline-replica-sets.md). ## Checkpoints and recovery At the configured interval, the controller coordinates a checkpoint across all active tasks. A checkpoint becomes eligible for recovery only after every participating task has successfully reported its state. For sinks that support transactional commits, the controller starts the commit step only after the checkpoint is complete. This keeps source progress, operator state, and committed sink output aligned to the same recovery point. See [Checkpointing and exactly-once delivery](/ingestion-engine/architecture/checkpointing.md) for the processing guarantees and sink-specific behavior. When a worker or task fails, e6 Ingestion Engine: 1. marks the running attempt as unhealthy; 2. stops or replaces affected worker capacity; 3. restores the pipeline from its latest successful checkpoint; and 4. resumes processing from the source positions recorded by that checkpoint. Work performed after the last successful checkpoint may be replayed. Whether that replay remains exactly once at the destination depends on the source and sink guarantees described in the connector documentation. Retryable failures are recovered automatically up to the configured restart limit. Fatal failures, or pipelines that exhaust that limit, enter `Failed` and remain there until a user restarts them or changes the configuration. ## Restarting, stopping, and rescaling * **Safe restart** - takes a final checkpoint before restarting. Use it when preserving the latest processed position is more important than restart speed. * **Force restart** - skips the final checkpoint and recovers from the latest previously successful checkpoint. Recent work may be replayed. * **Graceful stop** - takes a final checkpoint and then enters `Stopped`. * **Immediate stop** - stops processing without waiting for a new checkpoint. * **Rescale** - checkpoints the current execution, prepares capacity for the new parallelism, and restores state across the new task layout. Configuration changes that can be applied safely at runtime are reflected through the pipeline lifecycle. Changes that require new task placement move the pipeline through an intermediate status such as `Rescaling` or `Restarting`; status and errors remain available through the API while the change is in progress. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.e6data.com/ingestion-engine/architecture/controller.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.