Skip to main content

Ingest Engine is a distributed streaming ingestion engine that runs natively on Kubernetes.

It reads from Kafka, databases (through CDC), HTTP endpoints and cloud storage, transforms the data with SQL, and writes it to Apache Iceberg, Delta Lake and Kafka topics. Processing is real time, with exactly-once delivery. Add it when you want live data in your lakehouse. Query Engine does not need it.

Build your first pipeline

Choose your pipeline

01 · Read
Connect a source
Kafka, Confluent, Kinesis, PostgreSQL CDC, HTTP, filesystem
02 · Transform
Write streaming SQL
Windows, joins, stateful aggregations, event-time processing
03 · Write
Land query-ready tables
Apache Iceberg, Delta Lake, Kafka topics

​
Key capabilities

  • SQL-native stream processing. Standard SQL, based on Apache DataFusion, with streaming constructs such as tumbling and session windows, event-time joins and CDC update streams.
  • Exactly-once delivery. Asynchronous barrier snapshots with two-phase commit process every record exactly once, even across failures and restarts.
  • Event-time processing. Operations use the time embedded in the data, not wall-clock time, so results are the same whenever the data arrives.
  • Stateful operations. Windowed aggregations, stream-stream joins and lookup joins, with state checkpointed automatically.
  • Kubernetes-native. Deployed with Helm. Tables, pipelines and connection profiles are Kubernetes resources.

​
What do you want to do?

Get started
Ship a pipeline end to end
Kafka to IcebergPostgreSQL CDC to IcebergHTTP to Iceberg
Connectors
Connect sources and sinks
Message queues: Kafka, Confluent, KinesisChange data capture from databasesLakehouse sinks: Iceberg and Delta Lake
Streaming SQL
Transform data in flight
Streaming SQL quick startWindowsJoins and stateful queries
Operate and reference
Run it and look it up
ConfigurationObservability and metricsHTTP API, generated from the OpenAPI spec

​
How it works

Ingest Engine has two parts. The controller compiles SQL into distributed execution plans, schedules work across nodes and coordinates checkpoints. Workers are stateless processes that run the dataflow. They can be restarted at any time, because all state lives in checkpoint storage (S3, GCS or Azure Blob). How Ingest Engine moves data from sources to sinks

Architecture

Controller, workers and pipeline replica sets.

Checkpointing

How exactly-once delivery survives failures.

Watermarks

How event time drives windows and joins.
/query/Query the tables with Query Engine/agentflow/Run agent workflows on the fresh data with AgentFlowShared · /platform/Workspaces, identity and tokens in Platform
Docs owner: Ingest teamReport a problem with this page
Last modified on October 26, 2026