For the complete documentation index, see llms.txt. This page is also available as Markdown.

Build your first pipeline

This guide builds an end-to-end pipeline that accepts JSON events over HTTP and writes them to an Apache Iceberg table.

You will create:

  • an HTTP source table;

  • an Iceberg sink table; and

  • a SQL pipeline that connects them.

Prerequisites

You need:

  • access to a running e6 Ingestion Engine cluster;

  • the Kubernetes namespace and name of that cluster;

  • an Iceberg REST catalog;

  • an S3 or S3-compatible object store; and

  • kubectl access to the cluster.

The example uses an Iceberg REST catalog and S3-compatible storage. For other catalog and storage combinations, see Apache Iceberg.

Step 1: Create the HTTP Source

Save this complete manifest as events-source.yaml:

Apply it:

The important source settings are:

Setting
Purpose

route

Selects the external hostname configured by the cluster gateway. It must be a lowercase DNS label.

port

Port on which the source accepts requests.

path

HTTP path that accepts POST requests.

buffer_size

Maximum number of requests that can wait for processing.

max_body_size

Maximum decompressed request size in bytes.

auth

Requires clients to authenticate with a bearer token. Basic authentication is also supported.

framing.method.newline

Allows one JSON record or multiple newline-delimited JSON records per request.

bad_data.fail

Stops processing when an input record does not match the schema. Use drop: {} only when discarding malformed records is acceptable.

The source schema and format are required. Production clusters also require explicit capacity, authentication, and bad-data settings, all of which are included above.

Step 2: Create the Iceberg Sink

Save this complete manifest as events-sink.yaml:

If your REST catalog is anonymous, remove the auth block. When using AWS S3 with workload identity or the default credential chain, remove the static access keys and any custom endpoint settings that are not needed.

Apply the sink and wait until both tables report Ready:

The important sink settings are:

Setting
Purpose

connector: iceberg

Selects the current Apache Iceberg sink.

catalog.provider: direct

Connects the sink directly to the configured catalog.

catalog.format: iceberg

Selects the Iceberg catalog protocol.

catalog.type: rest

Uses an Iceberg REST catalog.

namespace and tableName

Identify the destination table as analytics.events.

appendOnly: true

Accepts insert-only output. CDC output requires primary keys and additional constraints.

partitioning

Partitions the table by hour using event_time.

storageOptions

Configures access to the object store used by the table.

fileRotation and parquet

Control output file sizing and Parquet encoding.

uploadChunkSizeBytes and uploadConcurrency

Control multipart upload behavior.

iceberg.targetFileSizeBytes

Sets the target data-file size used by Iceberg maintenance.

maintenance.enabled

Makes the maintenance policy explicit. This first pipeline disables scheduled maintenance.

The sink schema must match the columns and types produced by the pipeline query.

Step 3: Create the Pipeline

Save this complete manifest as events-pipeline.yaml:

The SQL names are the config.name values from the two table manifests: events_source and events_sink.

Apply the pipeline:

parallelism: 1 is a safe starting point for this HTTP ingestion pipeline. The checkpoint interval is expressed in microseconds, so 10000000 means 10 seconds.

Step 4: Send an Event

Build the public source URL from the hostname assigned to route: events and the configured path: /events:

Send an event whose fields match the source schema:

A successful request returns HTTP 200, which means the event was accepted for processing. It does not confirm that a downstream checkpoint has completed. Iceberg commits occur during checkpoints, so table visibility can lag behind the HTTP response.

Step 5: Verify the Pipeline

Check the resource phases:

For details about a resource that is not ready:

Common HTTP responses are:

Response
Meaning

200

The request was accepted.

400

The request body is empty.

401

The bearer token is missing or incorrect.

404

The request path does not match config.path.

413

The request exceeds max_body_size.

429

The source buffer is full; retry with backoff.

503

The source is temporarily unavailable; retry.

Clean Up

Delete the pipeline before deleting its tables:

Continue

Continue with Core Concepts to understand how profiles, tables, pipelines, and streaming execution fit together.

Last updated

Was this helpful?