Skip to main content
Traditional CDC-to-lakehouse architectures often repeat the same infrastructure for every pipeline: a database log reader, an intermediate message broker, and an independent catalog writer. This works, but operational cost and pressure on source databases and catalogs increase as more pipelines are added. Ingest Engine separates these responsibilities into shared platform capabilities. Database capture, durable ingestion, stream processing, and Iceberg commit coordination can be managed independently while pipelines retain their own configuration, processing progress, and destination tables.

​
Shared CDC capture

Ingest Engine’s shared CDC architecture manages connector sessions for multiple source databases. Each session reads its database transaction log, such as the MySQL binlog or PostgreSQL WAL, and maintains its own source position and schema history. Customers configure CDC sources without operating a separate Debezium deployment for every pipeline. When multiple pipelines need changes from the same database, the captured stream can be reused instead of opening another replication path to the source. Shared CDC capture reuses database changes across independent pipelines Shared capture manages an independent connector session for each database while allowing downstream pipelines to reuse captured changes. This can provide:
  • Less CDC infrastructure to deploy, monitor, and upgrade
  • Fewer duplicate replication connections and replication slots
  • Lower capture overhead on source databases
  • Consistent handling of source positions and schema changes
  • Independent downstream pipelines built from the same captured changes

​
Durable ingestion without mandatory Kafka

Ingest Engine’s durable ingestion log separates database capture from pipeline processing. The layer persists captured changes once so downstream pipelines can consume or replay them independently. For direct CDC-to-Iceberg workloads, this provides the buffering and replay boundary for which Kafka is often introduced. With the durable ingestion log, Kafka is optional when it would otherwise serve only as intermediate CDC plumbing. A direct architecture becomes possible: Durable ingestion makes Kafka optional for direct CDC-to-Iceberg pipelines For direct CDC-to-Iceberg workloads, durable ingestion provides the buffering and replay boundary. Kafka remains an option when other systems also consume the stream. Kafka remains valuable when it provides capabilities outside the Ingest Engine, including:
  • An enterprise event stream shared with other applications
  • Topics consumed directly by systems outside the Ingest Engine
  • Organization-wide retention and governance requirements
  • Integration with existing Kafka producers and consumers
Ingest Engine allows teams to choose Kafka for these capabilities rather than requiring it for every CDC pipeline.

​
Coordinated Iceberg commits

Iceberg uses optimistic concurrency when writers update table metadata. When many pipelines write frequently, overlapping catalog updates can produce conflicts, retries, rate limiting, and large numbers of small snapshots. Ingest Engine can coordinate prepared commits from multiple pipelines before publishing them to the catalog. Coordinated Iceberg commits reduce overlapping catalog updates Commit coordination serializes or combines compatible Ingest Engine commits to reduce catalog pressure. Iceberg concurrency still applies to external writers. For each destination table, it can:
  • Serialize overlapping Ingest Engine commits
  • Combine compatible commit fragments
  • Regulate catalog concurrency
  • Apply consistent retry and backpressure policies
  • Reduce the number of catalog updates and small snapshots
This reduces avoidable contention between Ingest Engine pipelines and limits catalog request pressure. It does not bypass Iceberg’s concurrency model, and writers outside the Ingest Engine can still introduce conflicts.

​
What this architecture unlocks

Shared ingestion and commit coordination make it practical to operate more CDC pipelines without repeating the surrounding infrastructure for every pipeline. Common uses include:
  • Capturing changes from multiple operational databases into Iceberg
  • Feeding history, current-state, and derived pipelines from one captured stream
  • Building database-to-Iceberg pipelines without a mandatory message broker
  • Replaying or restarting pipelines without reconnecting capture to the database
  • Scaling pipeline count while limiting source-database replication pressure
  • Coordinating frequent Iceberg writes to reduce catalog contention
The result is a simpler CDC-to-lakehouse architecture with fewer systems to operate and clearer boundaries between capture, replay, processing, and table publication. See Change data capture and Apache Iceberg for currently supported configuration and behavior.
Last modified on October 1, 2026