Traditional CDC-to-lakehouse architectures often repeat the same infrastructure for every pipeline: a database log reader, an intermediate message broker, and an independent catalog writer. This works, but operational cost and pressure on source databases and catalogs increase as more pipelines are added.Ingest Engine separates these responsibilities into shared platform capabilities. Database capture, durable ingestion, stream processing, and Iceberg commit coordination can be managed independently while pipelines retain their own configuration, processing progress, and destination tables.
Ingest Engine’s shared CDC architecture manages connector sessions for multiple source databases. Each session reads its database transaction log, such as the MySQL binlog or PostgreSQL WAL, and maintains its own source position and schema history.Customers configure CDC sources without operating a separate Debezium deployment for every pipeline. When multiple pipelines need changes from the same database, the captured stream can be reused instead of opening another replication path to the source.Shared capture manages an independent connector session for each database while allowing downstream pipelines to reuse captured changes.This can provide:
Less CDC infrastructure to deploy, monitor, and upgrade
Fewer duplicate replication connections and replication slots
Lower capture overhead on source databases
Consistent handling of source positions and schema changes
Independent downstream pipelines built from the same captured changes
Ingest Engine’s durable ingestion log separates database capture from pipeline processing. The layer persists captured changes once so downstream pipelines can consume or replay them independently.For direct CDC-to-Iceberg workloads, this provides the buffering and replay boundary for which Kafka is often introduced. With the durable ingestion log, Kafka is optional when it would otherwise serve only as intermediate CDC plumbing.A direct architecture becomes possible:For direct CDC-to-Iceberg workloads, durable ingestion provides the buffering and replay boundary. Kafka remains an option when other systems also consume the stream.Kafka remains valuable when it provides capabilities outside the Ingest Engine, including:
An enterprise event stream shared with other applications
Topics consumed directly by systems outside the Ingest Engine
Organization-wide retention and governance requirements
Integration with existing Kafka producers and consumers
Ingest Engine allows teams to choose Kafka for these capabilities rather than requiring it for every CDC pipeline.
Iceberg uses optimistic concurrency when writers update table metadata. When many pipelines write frequently, overlapping catalog updates can produce conflicts, retries, rate limiting, and large numbers of small snapshots.Ingest Engine can coordinate prepared commits from multiple pipelines before publishing them to the catalog.Commit coordination serializes or combines compatible Ingest Engine commits to reduce catalog pressure. Iceberg concurrency still applies to external writers.For each destination table, it can:
Serialize overlapping Ingest Engine commits
Combine compatible commit fragments
Regulate catalog concurrency
Apply consistent retry and backpressure policies
Reduce the number of catalog updates and small snapshots
This reduces avoidable contention between Ingest Engine pipelines and limits catalog request pressure. It does not bypass Iceberg’s concurrency model, and writers outside the Ingest Engine can still introduce conflicts.
Shared ingestion and commit coordination make it practical to operate more CDC pipelines without repeating the surrounding infrastructure for every pipeline.Common uses include:
Capturing changes from multiple operational databases into Iceberg
Feeding history, current-state, and derived pipelines from one captured stream
Building database-to-Iceberg pipelines without a mandatory message broker
Replaying or restarting pipelines without reconnecting capture to the database
Scaling pipeline count while limiting source-database replication pressure
Coordinating frequent Iceberg writes to reduce catalog contention
The result is a simpler CDC-to-lakehouse architecture with fewer systems to operate and clearer boundaries between capture, replay, processing, and table publication.See Change data capture and Apache Iceberg for currently supported configuration and behavior.
