> ## Documentation Index
> Fetch the complete documentation index at: https://docs.e6data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Streaming

> Define a source, a sink, and a transformation with SQL. Laminar keeps processing new records until you stop the pipeline. Create and manage it from Workbench or a Spark Connect SQL

ExplanationApplies to: Unified engine deployments

Define a source, a sink, and a transformation with SQL. Laminar keeps processing new records until you stop the pipeline. Create and manage it from Workbench or a Spark Connect SQL client.

## How a pipeline works

| Part | What it defines | Kafka-to-Iceberg example |
| - | - | - |
| Source | Where records arrive and how to read them | A Kafka topic with JSON order messages |
| Transformation | Which records and columns to write | Keep paid orders and calculate tax |
| Sink | Where transformed records go | An Iceberg table in an existing catalog |
| Pipeline | The transformation and its execution options | Parallelism and checkpoint interval |

Use `CREATE TABLE ... WITH (connector = ...)` to register each connection. Use `CREATE STREAMING PIPELINE ... AS INSERT INTO ... SELECT ...` to connect them.

> **Creating starts execution.** Running `CREATE STREAMING PIPELINE` starts the pipeline by default. Add `start_paused = true` to save a pipeline for later.

## Choose a starting point

| Your goal | Read |
| - | - |
| Check services, access, and example values | [Before you start](https://docs.e6data.com/transformation/streaming/before-you-start) |
| Run a complete Kafka-to-Iceberg flow | [Your first pipeline](https://docs.e6data.com/transformation/streaming/first-pipeline) |
| Define Kafka input and choose offsets | [Sources](https://docs.e6data.com/transformation/streaming/sources) |
| Define an Iceberg destination | [Sinks](https://docs.e6data.com/transformation/streaming/sinks) |
| Filter records, calculate values, or use a CTE | [Transformations](https://docs.e6data.com/transformation/streaming/transformations) |
| Inspect, stop, resume, or retry a request | [Manage pipelines](https://docs.e6data.com/transformation/streaming/manage-pipelines) |
| Look up command syntax and defaults | [Streaming SQL reference](https://docs.e6data.com/transformation/streaming/sql-reference) |

## Connections and catalog tables

A Streaming source or sink name is a saved connector definition. For example, `paid_orders_sink` can point to the Iceberg table `analytics.paid_orders`. Registering the sink does not create a Unified catalog connection for reading that table.

To query the output, connect Unified to the same Iceberg catalog using the relevant [catalog recipe](https://docs.e6data.com/transformation/recipes/overview.md#catalog-connections). Use that catalog's qualified table name for reads.

## Execution and visibility

A successful create response means Laminar accepted and saved the pipeline. Workers then start it and publish output through the Streaming execution flow. Closing Workbench or the SQL client leaves the pipeline running.

Pipeline acceptance, running workers, and visible Iceberg rows are separate stages. Use `DESCRIBE PIPELINE` for the pipeline state. Use the shared [Observability](https://docs.e6data.com/transformation/operations/observability.md) docs for metrics and operational visibility.

These pages describe the Streaming SQL commands. Streaming transformations use the Streaming planner's SQL rules. The general SQL reference covers Unified queries; its full set of functions and commands does not apply to every Streaming transformation.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.