> ## Documentation Index
> Fetch the complete documentation index at: https://docs.e6data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Schema Registry

> Move Avro data from Kafka or Confluent to your destination using AWS Glue or Confluent Schema Registry.

How-to guideApplies to: All deployments

Connect your schema registry when setting up a Kafka or Confluent source. Ingest Engine reads the Avro schema for you and fills in the columns automatically.

## Before you start

Have your topic, registry connection details and destination ready. This guide supports Avro events. Your deployment needs access to the topic, registry and destination.

For AWS resources in another account, ask your administrator to complete [Cross Account Access](https://docs.e6data.com/ingest/setup/cross-account-access.md) first.

## 1. Connect your source

Create a **Kafka** or **Confluent** source in the console. Enter your broker details, authentication and topic, then choose where to start reading.

Select your schema registry provider:

> [tab] AWS Glue
>
> Enter **Glue region** and **Registry name**.
>
> If your administrator provided a role for registry access, enter it in **Registry role ARN**. For help with access, see [Cross Account Access](https://docs.e6data.com/ingest/setup/cross-account-access.md#glue-schema-registry).
>
> Enter **Schema name** if it differs from your topic name.

> [tab] Confluent
>
> Enter **Registry endpoint**, **API key** and **API secret**. Use your registry credentials, which are separate from your broker credentials.
>
> Enter **Subject name** if it differs from `<topic>-value`.

Create the source. Ingest Engine fills in its columns from the registry.

## 2. Choose your destination

Create a destination and enter its connection details. For Iceberg, select the catalog, warehouse, namespace and table you want to write to.

Connect your source to the destination in a pipeline. Select the columns you want to send, or use `SELECT *` to send all columns.

## 3. Start moving data

Start the pipeline and check that records appear in your destination after a successful checkpoint. If you chose to read only new events, send new data after the source starts consuming.

## Other workflows and help

> [accordion] Write to several Iceberg tables
>
> Use this option when one topic contains events from several source tables.
>
> 1. Enable **Multi-schema source** when creating your source.
> 2. Create an Iceberg destination and enable **Multi-table routing**. Leave the fixed table name and destination schema unset.
> 3. Use the generated query to send the source directly to the destination.
>
> Each event must identify its destination table. Ingest Engine creates missing tables automatically and stores updates and deletes as new event rows, rather than applying them to existing rows.
>
> See [Iceberg multi-table routing](https://docs.e6data.com/ingest/connectors/lakehouse/iceberg.md#multi-table-routing) for the required event field, naming rules and supported schema changes.

> [accordion] The schema cannot be loaded or columns are missing
>
> Check the registry, schema or subject name, and read permissions. For Glue, the latest schema version must be `AVAILABLE`.
>
> Source columns are discovered when you create the source. If you add fields to the registry later, update the source schema and pipeline to use them.

> [accordion] Configure a pipeline with YAML or the API
>
> For complete importable pipelines, see [Schema Registry to Iceberg (YAML)](https://docs.e6data.com/ingest/examples/schema-registry-to-iceberg.md).
>
> See [Kafka schema registry configuration](https://docs.e6data.com/ingest/connectors/message-queues/kafka.md#schema-registry), [Confluent](https://docs.e6data.com/ingest/connectors/message-queues/confluent.md), and the [Avro reference](https://docs.e6data.com/ingest/reference/data-formats.md#schema-registry-integration).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.