Skip to main content
How-to guideApplies to: All deployments
Connect your schema registry when setting up a Kafka or Confluent source. Ingest Engine reads the Avro schema for you and fills in the columns automatically.

​
Before you start

Have your topic, registry connection details and destination ready. This guide supports Avro events. Your deployment needs access to the topic, registry and destination. For AWS resources in another account, ask your administrator to complete Cross Account Access first.

​
1. Connect your source

Create a Kafka or Confluent source in the console. Enter your broker details, authentication and topic, then choose where to start reading. Select your schema registry provider:
Enter Glue region and Registry name.If your administrator provided a role for registry access, enter it in Registry role ARN. For help with access, see Cross Account Access.Enter Schema name if it differs from your topic name.
Create the source. Ingest Engine fills in its columns from the registry.

​
2. Choose your destination

Create a destination and enter its connection details. For Iceberg, select the catalog, warehouse, namespace and table you want to write to. Connect your source to the destination in a pipeline. Select the columns you want to send, or use SELECT * to send all columns.

​
3. Start moving data

Start the pipeline and check that records appear in your destination after a successful checkpoint. If you chose to read only new events, send new data after the source starts consuming.

​
Other workflows and help

Use this option when one topic contains events from several source tables.
  1. Enable Multi-schema source when creating your source.
  2. Create an Iceberg destination and enable Multi-table routing. Leave the fixed table name and destination schema unset.
  3. Use the generated query to send the source directly to the destination.
Each event must identify its destination table. Ingest Engine creates missing tables automatically and stores updates and deletes as new event rows, rather than applying them to existing rows.See Iceberg multi-table routing for the required event field, naming rules and supported schema changes.
Check the registry, schema or subject name, and read permissions. For Glue, the latest schema version must be AVAILABLE.Source columns are discovered when you create the source. If you add fields to the registry later, update the source schema and pipeline to use them.
Last modified on November 6, 2026