Overview

e6 Ingestion Engine is a distributed stream-processing engine that connects data sources to data sinks through continuously running SQL pipelines. At a high level, it consists of a controller, a pool of workers, connector integrations, and durable checkpoint storage.

System model

e6 Ingestion Engine system model

Controller

The controller provides the control plane for e6 Ingestion Engine. It accepts pipeline definitions through the HTTP API or Kubernetes resources, validates their configuration and SQL, distributes work across available capacity, and coordinates pipeline lifecycle and recovery.

The controller manages pipelines but does not process their records.

Workers

Workers provide the compute capacity that runs pipelines. They read from sources, apply SQL transformations, maintain any required processing state, and write results to sinks.

Workers are replaceable. Durable recovery information is stored in checkpoints so a pipeline can recover when worker capacity becomes unavailable.

Connectors

Connectors define how data enters and leaves a pipeline. Source connectors read or receive records, while sink connectors deliver processed results to external systems.

Connector capabilities determine supported parallelism, replay behavior, and delivery guarantees. See the connector documentation for the contract of each integration.

e6 Ingestion Engine's shared data services consolidate CDC capture, durable ingestion, and Iceberg commit coordination across pipelines.

Checkpoint storage

e6 Ingestion Engine periodically records pipeline state and source progress in durable checkpoint storage. These checkpoints provide a recovery point when a pipeline is restarted after a failure.

Checkpoint data serves a different purpose from destination data: it is used for recovery rather than as pipeline output.

How a pipeline runs

1. A user submits a pipeline through the HTTP API or a Kubernetes resource.
2. The controller validates the pipeline and prepares its execution plan.
3. Work is distributed across workers according to the pipeline's parallelism and available capacity.
4. Workers continuously read records, process them with SQL, and write results to sinks.
5. Successful checkpoints record the state and progress needed for recovery.
6. If processing is interrupted, e6 Ingestion Engine recovers the pipeline from its latest usable checkpoint.

Scaling and availability

Pipeline parallelism controls how much work can be processed concurrently. Worker CPU and memory determine the capacity available for that work.

Workers can be added or replaced without changing the pipeline definition. In Kubernetes deployments, worker capacity is provided through managed pods.

Some connectors have additional scaling constraints, so parallelism should be chosen using the relevant connector documentation.

Processing and delivery guarantees

Checkpointing coordinates processing progress across a pipeline, but the end-to-end guarantee depends on the source and sink:

• Replayable sources can resume from progress recorded in a checkpoint.
• Sinks that support atomic or transactional commits can provide checkpoint-aligned, exactly-once visibility.
• Other integrations may provide different guarantees or require additional configuration.

Always use the source and sink documentation as the authoritative delivery contract for a pipeline.