> ## Documentation Index
> Fetch the complete documentation index at: https://docs.e6data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Ingest Engine

> Turn live data into query-ready tables.

export const Owner = ({team}) => <div className="e6-owner">
    <span>Docs owner: {team}</span>
    <a href="mailto:help@e6data.com">Report a problem with this page</a>
  </div>;

export const Addon = ({label, title, href}) => <a href={href} className="e6-addon">
    <span className="e6-addon-label">{label}</span>
    <span className="e6-addon-title">{title}</span>
  </a>;

export const TaskCard = ({label, title, links = []}) => <div className="e6-task">
    <span className="e6-eyebrow">{label}</span>
    <div className="e6-task-title">{title}</div>
    {links.map(l => <a key={l.href + l.title} href={l.href} className="e6-mode-link">
        <span>{l.title}</span>
        <svg width="16" height="16" viewBox="0 0 16 16" fill="none" stroke="currentColor" strokeWidth="1.6" aria-hidden="true">
          <path d="M6 3.5L10.5 8 6 12.5" />
        </svg>
      </a>)}
  </div>;

export const Flow = ({steps = []}) => <div className="e6-flow">
    {steps.map(s => <div className="e6-flow-step" key={s.num}>
        <div className="e6-flow-num">{s.num}</div>
        <div className="e6-flow-title">{s.title}</div>
        <div className="e6-flow-desc">{s.desc}</div>
      </div>)}
  </div>;

<p className="e6-lede">Ingest Engine is a distributed streaming ingestion engine that runs natively on Kubernetes.</p>

It reads from Kafka, databases (through CDC), HTTP endpoints and cloud storage, transforms the data with SQL, and writes it to Apache Iceberg, Delta Lake and Kafka topics. Processing is real time, with exactly-once delivery. Add it when you want live data in your lakehouse. Query Engine does not need it.

<Columns cols={2}>
  <Card title="Build your first pipeline" icon="play" href="/ingest/get-started/first-pipeline" horizontal />

  <Card title="Choose your pipeline" icon="git-fork" href="/ingest/get-started/choose-your-pipeline" horizontal />
</Columns>

<Flow
  steps={[
{ num: "01 · Read", title: "Connect a source", desc: "Kafka, Confluent, Kinesis, PostgreSQL CDC, HTTP, filesystem" },
{ num: "02 · Transform", title: "Write streaming SQL", desc: "Windows, joins, stateful aggregations, event-time processing" },
{ num: "03 · Write", title: "Land query-ready tables", desc: "Apache Iceberg, Delta Lake, Kafka topics" },
]}
/>

## What do you want to do?

<div className="e6-tasks">
  <TaskCard
    label="Get started"
    title="Ship a pipeline end to end"
    links={[
{ title: "Kafka to Iceberg", href: "/ingest/examples/kafka-to-iceberg" },
{ title: "PostgreSQL CDC to Iceberg", href: "/ingest/examples/postgres-cdc-to-iceberg" },
{ title: "HTTP to Iceberg", href: "/ingest/examples/http-to-iceberg" },
]}
  />

  <TaskCard
    label="Connectors"
    title="Connect sources and sinks"
    links={[
{ title: "Message queues: Kafka, Confluent, Kinesis", href: "/ingest/connectors/message-queues" },
{ title: "Change data capture from databases", href: "/ingest/connectors/databases/cdc" },
{ title: "Lakehouse sinks: Iceberg and Delta Lake", href: "/ingest/connectors/lakehouse" },
]}
  />

  <TaskCard
    label="Streaming SQL"
    title="Transform data in flight"
    links={[
{ title: "Streaming SQL quick start", href: "/ingest/sql/quick-start" },
{ title: "Windows", href: "/ingest/sql/streaming/windows" },
{ title: "Joins and stateful queries", href: "/ingest/sql/streaming/joins" },
]}
  />

  <TaskCard
    label="Operate and reference"
    title="Run it and look it up"
    links={[
{ title: "Configuration", href: "/ingest/operations/configuration" },
{ title: "Observability and metrics", href: "/ingest/operations/observability" },
{ title: "HTTP API, generated from the OpenAPI spec", href: "/api-reference" },
]}
  />
</div>

## How it works

<Columns cols={3}>
  <Card title="Architecture" href="/ingest/concepts/architecture">Controller, workers and pipeline replica sets.</Card>
  <Card title="Checkpointing" href="/ingest/concepts/checkpointing">How exactly-once delivery survives failures.</Card>
  <Card title="Watermarks" href="/ingest/concepts/watermarks">How event time drives windows and joins.</Card>
</Columns>

## Related products

<div className="e6-addons">
  <Addon label="/query/" title="Query the tables with Query Engine" href="/query" />

  <Addon label="/agentflow/" title="Run agent workflows on the fresh data with AgentFlow" href="/agentflow" />

  <Addon label="Shared · /platform/" title="Workspaces, identity and tokens in Platform" href="/platform" />
</div>

<Owner team="Ingest team" />


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.