> For the complete documentation index, see [llms.txt](https://docs.e6data.com/ingestion-engine/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.e6data.com/ingestion-engine/get-started/readme.md).

# Introduction to e6 Ingestion Engine

e6 Ingestion Engine is a distributed streaming ingestion engine by [e6data](https://e6data.com). It connects data sources to data sinks through real-time SQL pipelines, running natively on Kubernetes.

## What e6 Ingestion Engine Does

e6 Ingestion Engine continuously ingests data from sources like Kafka, databases (via CDC), HTTP endpoints, and cloud storage - transforms it with SQL - and writes it to sinks like Apache Iceberg, Delta Lake, and Kafka topics. All processing happens in real time with exactly-once delivery guarantees.

![Introduction Diagram](/files/oLf0ZXkb09WUje9WcIOD)

## Key Capabilities

**SQL-native stream processing** - Write standard SQL (based on Apache DataFusion) extended with streaming constructs like tumbling windows, session windows, event-time joins, and CDC-style update streams.

**Exactly-once semantics** - Asynchronous barrier snapshots with two-phase commit ensure every record is processed exactly once, even across failures and restarts.

**Event-time processing** - All operations use event time (embedded in the data), not wall-clock time. This makes results deterministic and correct regardless of when data arrives.

**Stateful operations** - Windowed aggregations, stream-stream joins, and lookup joins with automatic state management and fault-tolerant checkpointing.

**Kubernetes-native** - Deployed via Helm with Custom Resource Definitions (CRDs). Define tables, pipelines, and connection profiles as Kubernetes resources.

## Architecture

e6 Ingestion Engine has two main components:

* **Controller** - The brain of the system. Compiles SQL into distributed execution plans, schedules work across nodes, and coordinates checkpoints.
* **Workers** - Stateless processes that execute the dataflow. Workers can be killed and restarted at any time - all state lives in checkpoint storage (S3, GCS, or Azure Blob).

Pipelines are compiled into distributed DAGs (directed acyclic graphs) where each node runs as a parallel subtask across workers. See the [Architecture](/ingestion-engine/architecture/overview.md) deep-dive for full details.

## SQL Support

e6 Ingestion Engine's SQL dialect is based on Apache DataFusion and extended for streaming. See the [SQL overview](/ingestion-engine/sql/sql.md) for usage guidance and the full reference.

* **Windows** - Tumbling, sliding (hopping), and session windows for time-bucketed aggregations
* **Joins** - Windowed joins, updating joins, and lookup joins against external systems
* **Functions** - Math, string, JSON, datetime, array, struct, regex, hashing, aggregate, and window functions
* **DDL** - `CREATE TABLE` with connector configuration, virtual fields, schema inference
* **CDC** - Read and write Debezium-formatted change streams for updating semantics

## Continue

Continue with [Choose Your Pipeline](/ingestion-engine/get-started/choose-your-pipeline.md) to select the source, destination, and guide for your use case.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.e6data.com/ingestion-engine/get-started/readme.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.