> For the complete documentation index, see [llms.txt](https://docs.e6data.com/ingestion-engine/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.e6data.com/ingestion-engine/architecture/shared-data-services.md). # Shared data services Traditional CDC-to-lakehouse architectures often repeat the same infrastructure for every pipeline: a database log reader, an intermediate message broker, and an independent catalog writer. This works, but operational cost and pressure on source databases and catalogs increase as more pipelines are added. e6 Ingestion Engine separates these responsibilities into shared platform capabilities. Database capture, durable ingestion, stream processing, and Iceberg commit coordination can be managed independently while pipelines retain their own configuration, processing progress, and destination tables. ## Shared CDC capture e6 Ingestion Engine's shared CDC architecture manages connector sessions for multiple source databases. Each session reads its database transaction log, such as the MySQL binlog or PostgreSQL WAL, and maintains its own source position and schema history. Customers configure CDC sources without operating a separate Debezium deployment for every pipeline. When multiple pipelines need changes from the same database, the captured stream can be reused instead of opening another replication path to the source. ![Shared CDC capture reuses database changes across independent pipelines](https://1725648790-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FuztQ31u5OyLCkPgUTosp%2Fuploads%2Fgit-blob-84183efe423370b87b2d32cf07072dd0e639be0e%2Fshared-data-services-cdc-capture.svg?alt=media) *Shared capture manages an independent connector session for each database while allowing downstream pipelines to reuse captured changes.* This can provide: * Less CDC infrastructure to deploy, monitor, and upgrade * Fewer duplicate replication connections and replication slots * Lower capture overhead on source databases * Consistent handling of source positions and schema changes * Independent downstream pipelines built from the same captured changes ## Durable ingestion without mandatory Kafka e6 Ingestion Engine's durable ingestion log separates database capture from pipeline processing. The layer persists captured changes once so downstream pipelines can consume or replay them independently. For direct CDC-to-Iceberg workloads, this provides the buffering and replay boundary for which Kafka is often introduced. With the durable ingestion log, Kafka is optional when it would otherwise serve only as intermediate CDC plumbing. A direct architecture becomes possible: ![Durable ingestion makes Kafka optional for direct CDC-to-Iceberg pipelines](https://1725648790-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FuztQ31u5OyLCkPgUTosp%2Fuploads%2Fgit-blob-91987151918f4b6a9cf323d28f075dcd2c5a0010%2Fshared-data-services-durable-ingestion.svg?alt=media) *For direct CDC-to-Iceberg workloads, durable ingestion provides the buffering and replay boundary. Kafka remains an option when other systems also consume the stream.* Kafka remains valuable when it provides capabilities outside the e6 Ingestion Engine, including: * An enterprise event stream shared with other applications * Topics consumed directly by systems outside the e6 Ingestion Engine * Organization-wide retention and governance requirements * Integration with existing Kafka producers and consumers e6 Ingestion Engine allows teams to choose Kafka for these capabilities rather than requiring it for every CDC pipeline. ## Coordinated Iceberg commits Iceberg uses optimistic concurrency when writers update table metadata. When many pipelines write frequently, overlapping catalog updates can produce conflicts, retries, rate limiting, and large numbers of small snapshots. e6 Ingestion Engine can coordinate prepared commits from multiple pipelines before publishing them to the catalog. ![Coordinated Iceberg commits reduce overlapping catalog updates](https://1725648790-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FuztQ31u5OyLCkPgUTosp%2Fuploads%2Fgit-blob-ba6c1ed610a0cfeb008c1d38c646c20f077e1b8a%2Fshared-data-services-iceberg-commits.svg?alt=media) *Commit coordination serializes or combines compatible e6 Ingestion Engine commits to reduce catalog pressure. Iceberg concurrency still applies to external writers.* For each destination table, it can: * Serialize overlapping e6 Ingestion Engine commits * Combine compatible commit fragments * Regulate catalog concurrency * Apply consistent retry and backpressure policies * Reduce the number of catalog updates and small snapshots This reduces avoidable contention between e6 Ingestion Engine pipelines and limits catalog request pressure. It does not bypass Iceberg's concurrency model, and writers outside the e6 Ingestion Engine can still introduce conflicts. ## What this architecture unlocks Shared ingestion and commit coordination make it practical to operate more CDC pipelines without repeating the surrounding infrastructure for every pipeline. Common uses include: * Capturing changes from multiple operational databases into Iceberg * Feeding history, current-state, and derived pipelines from one captured stream * Building database-to-Iceberg pipelines without a mandatory message broker * Replaying or restarting pipelines without reconnecting capture to the database * Scaling pipeline count while limiting source-database replication pressure * Coordinating frequent Iceberg writes to reduce catalog contention The result is a simpler CDC-to-lakehouse architecture with fewer systems to operate and clearer boundaries between capture, replay, processing, and table publication. See [Change data capture](/ingestion-engine/connectors/databases/cdc.md) and [Apache Iceberg](/ingestion-engine/connectors/lakehouse/iceberg.md) for currently supported configuration and behavior. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.e6data.com/ingestion-engine/architecture/shared-data-services.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.