For the complete documentation index, see llms.txt. This page is also available as Markdown.

Choose your pipeline

Not sure where to start? Pick your data source and destination below to find the right guide and connector docs for your use case.

What's your source?

Source
Use when...
You need
Guide

Kafka

Streaming events from a message broker

Broker address, topic name, auth

Kafka to Iceberg example

Confluent

Confluent Cloud managed Kafka

Confluent cluster, API key

Confluent connector docs

Kinesis

AWS-native event streaming

Stream name, AWS region

Kinesis connector docs

CDC (Postgres/MySQL)

Replicate database tables in real-time

DB host, credentials, table names

CDC connector docs

HTTP

Receive webhook/push data

Just a port number

HTTP to Iceberg example

Filesystem (S3/GCS)

Batch ingest existing files

Bucket path, credentials

What's your sink?

Sink
Use when...
You need
Guide

Iceberg

Analytics lakehouse (most common)

Catalog URL, warehouse path

Iceberg connector docs

Delta Lake

Delta ecosystem

Storage path

Delta connector docs

Kafka

Forward processed events to another topic

Broker address, topic

Kafka sink docs

Common Combinations

Kafka → Iceberg - Stream events into a data lakehouse for analytics

HTTP → Iceberg - Ingest webhooks into queryable tables

S3 Files → Iceberg - Batch-load existing data files into Iceberg

CDC → Iceberg - Replicate a database table in real-time

Last updated

Was this helpful?