> For the complete documentation index, see [llms.txt](https://docs.e6data.com/ingestion-engine/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.e6data.com/ingestion-engine/architecture/overview.md). # Overview e6 Ingestion Engine is a distributed stream-processing engine that connects data sources to data sinks through continuously running SQL pipelines. At a high level, it consists of a controller, a pool of workers, connector integrations, and durable checkpoint storage. ## System model ![e6 Ingestion Engine system model](https://1725648790-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FuztQ31u5OyLCkPgUTosp%2Fuploads%2Fgit-blob-f1b51e628355158bd7789628a4e5103b7eb16188%2Foverview-diagrams-architecture-overview.svg?alt=media) ### Controller The [controller](/ingestion-engine/architecture/controller.md) provides the control plane for e6 Ingestion Engine. It accepts pipeline definitions through the HTTP API or Kubernetes resources, validates their configuration and SQL, distributes work across available capacity, and coordinates pipeline lifecycle and recovery. The controller manages pipelines but does not process their records. ### Workers [Workers](/ingestion-engine/architecture/data-plane.md) provide the compute capacity that runs pipelines. They read from sources, apply SQL transformations, maintain any required processing state, and write results to sinks. Workers are replaceable. Durable recovery information is stored in checkpoints so a pipeline can recover when worker capacity becomes unavailable. ### Connectors Connectors define how data enters and leaves a pipeline. Source connectors read or receive records, while sink connectors deliver processed results to external systems. Connector capabilities determine supported parallelism, replay behavior, and delivery guarantees. See the [connector documentation](/ingestion-engine/connectors/connectors.md) for the contract of each integration. e6 Ingestion Engine's [shared data services](/ingestion-engine/architecture/shared-data-services.md) consolidate CDC capture, durable ingestion, and Iceberg commit coordination across pipelines. ### Checkpoint storage e6 Ingestion Engine periodically records pipeline state and source progress in durable checkpoint storage. These [checkpoints](/ingestion-engine/architecture/checkpointing.md) provide a recovery point when a pipeline is restarted after a failure. Checkpoint data serves a different purpose from destination data: it is used for recovery rather than as pipeline output. ## How a pipeline runs 1. A user submits a pipeline through the HTTP API or a Kubernetes resource. 2. The controller validates the pipeline and prepares its execution plan. 3. Work is distributed across workers according to the pipeline's parallelism and available capacity. 4. Workers continuously read records, process them with SQL, and write results to sinks. 5. Successful checkpoints record the state and progress needed for recovery. 6. If processing is interrupted, e6 Ingestion Engine recovers the pipeline from its latest usable checkpoint. ## Scaling and availability Pipeline parallelism controls how much work can be processed concurrently. Worker CPU and memory determine the capacity available for that work. Workers can be added or replaced without changing the pipeline definition. In Kubernetes deployments, worker capacity is provided through managed pods. Some connectors have additional scaling constraints, so parallelism should be chosen using the relevant connector documentation. ## Processing and delivery guarantees Checkpointing coordinates processing progress across a pipeline, but the end-to-end guarantee depends on the source and sink: * Replayable sources can resume from progress recorded in a checkpoint. * Sinks that support atomic or transactional commits can provide checkpoint-aligned, exactly-once visibility. * Other integrations may provide different guarantees or require additional configuration. Always use the source and sink documentation as the authoritative delivery contract for a pipeline. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.e6data.com/ingestion-engine/architecture/overview.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.