Ingest Engine is a distributed stream-processing engine that connects data sources to data sinks through continuously running SQL pipelines. At a high level, it consists of a controller, a pool of workers, connector integrations, and durable checkpoint storage.
The controller provides the control plane for Ingest Engine. It accepts pipeline definitions through the HTTP API or Kubernetes resources, validates their configuration and SQL, distributes work across available capacity, and coordinates pipeline lifecycle and recovery.The controller manages pipelines but does not process their records.
Workers provide the compute capacity that runs pipelines. They read from sources, apply SQL transformations, maintain any required processing state, and write results to sinks.Workers are replaceable. Durable recovery information is stored in checkpoints so a pipeline can recover when worker capacity becomes unavailable.
Connectors define how data enters and leaves a pipeline. Source connectors read or receive records, while sink connectors deliver processed results to external systems.Connector capabilities determine supported parallelism, replay behavior, and delivery guarantees. See the connector documentation for the contract of each integration.Ingest Engine’s shared data services consolidate CDC capture, durable ingestion, and Iceberg commit coordination across pipelines.
Ingest Engine periodically records pipeline state and source progress in durable checkpoint storage. These checkpoints provide a recovery point when a pipeline is restarted after a failure.Checkpoint data serves a different purpose from destination data: it is used for recovery rather than as pipeline output.
Pipeline parallelism controls how much work can be processed concurrently. Worker CPU and memory determine the capacity available for that work.Workers can be added or replaced without changing the pipeline definition. In Kubernetes deployments, worker capacity is provided through managed pods.Some connectors have additional scaling constraints, so parallelism should be chosen using the relevant connector documentation.
