> ## Documentation Index > Fetch the complete documentation index at: https://docs.e6data.com/llms.txt > Use this file to discover all available pages before exploring further. # Introduction to Ingest Engine Ingest Engine is a distributed streaming ingestion engine by [e6data](https://e6data.com). It connects data sources to data sinks through real-time SQL pipelines, running natively on Kubernetes. ## What Ingest Engine Does Ingest Engine continuously ingests data from sources like Kafka, databases (via CDC), HTTP endpoints, and cloud storage - transforms it with SQL - and writes it to sinks like Apache Iceberg, Delta Lake, and Kafka topics. All processing happens in real time with exactly-once delivery guarantees. Introduction Diagram ## Key Capabilities **SQL-native stream processing** - Write standard SQL (based on Apache DataFusion) extended with streaming constructs like tumbling windows, session windows, event-time joins, and CDC-style update streams. **Exactly-once semantics** - Asynchronous barrier snapshots with two-phase commit ensure every record is processed exactly once, even across failures and restarts. **Event-time processing** - All operations use event time (embedded in the data), not wall-clock time. This makes results deterministic and correct regardless of when data arrives. **Stateful operations** - Windowed aggregations, stream-stream joins, and lookup joins with automatic state management and fault-tolerant checkpointing. **Kubernetes-native** - Deployed via Helm with Custom Resource Definitions (CRDs). Define tables, pipelines, and connection profiles as Kubernetes resources. ## Architecture Ingest Engine has two main components: * **Controller** - The brain of the system. Compiles SQL into distributed execution plans, schedules work across nodes, and coordinates checkpoints. * **Workers** - Stateless processes that execute the dataflow. Workers can be killed and restarted at any time - all state lives in checkpoint storage (S3, GCS, or Azure Blob). Pipelines are compiled into distributed DAGs (directed acyclic graphs) where each node runs as a parallel subtask across workers. See the [Architecture](/ingest/concepts/architecture) deep-dive for full details. ## SQL Support Ingest Engine's SQL dialect is based on Apache DataFusion and extended for streaming. See the [SQL overview](/ingest/sql) for usage guidance and the full reference. * **Windows** - Tumbling, sliding (hopping), and session windows for time-bucketed aggregations * **Joins** - Windowed joins, updating joins, and lookup joins against external systems * **Functions** - Math, string, JSON, datetime, array, struct, regex, hashing, aggregate, and window functions * **DDL** - `CREATE TABLE` with connector configuration, virtual fields, schema inference * **CDC** - Read and write Debezium-formatted change streams for updating semantics ## Continue Continue with [Choose Your Pipeline](/ingest/get-started/choose-your-pipeline) to select the source, destination, and guide for your use case. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.