Ingest Engine is a distributed streaming ingestion engine by e6data. It connects data sources to data sinks through real-time SQL pipelines, running natively on Kubernetes.
Ingest Engine continuously ingests data from sources like Kafka, databases (via CDC), HTTP endpoints, and cloud storage - transforms it with SQL - and writes it to sinks like Apache Iceberg, Delta Lake, and Kafka topics. All processing happens in real time with exactly-once delivery guarantees.
SQL-native stream processing - Write standard SQL (based on Apache DataFusion) extended with streaming constructs like tumbling windows, session windows, event-time joins, and CDC-style update streams.Exactly-once semantics - Asynchronous barrier snapshots with two-phase commit ensure every record is processed exactly once, even across failures and restarts.Event-time processing - All operations use event time (embedded in the data), not wall-clock time. This makes results deterministic and correct regardless of when data arrives.Stateful operations - Windowed aggregations, stream-stream joins, and lookup joins with automatic state management and fault-tolerant checkpointing.Kubernetes-native - Deployed via Helm with Custom Resource Definitions (CRDs). Define tables, pipelines, and connection profiles as Kubernetes resources.
Controller - The brain of the system. Compiles SQL into distributed execution plans, schedules work across nodes, and coordinates checkpoints.
Workers - Stateless processes that execute the dataflow. Workers can be killed and restarted at any time - all state lives in checkpoint storage (S3, GCS, or Azure Blob).
Pipelines are compiled into distributed DAGs (directed acyclic graphs) where each node runs as a parallel subtask across workers. See the Architecture deep-dive for full details.
