Streaming

ExplanationApplies to: Unified engine deployments

Define a source, a sink, and a transformation with SQL. Laminar keeps processing new records until you stop the pipeline. Create and manage it from Workbench or a Spark Connect SQL client.

How a pipeline works

Part	What it defines	Kafka-to-Iceberg example
Source	Where records arrive and how to read them	A Kafka topic with JSON order messages
Transformation	Which records and columns to write	Keep paid orders and calculate tax
Sink	Where transformed records go	An Iceberg table in an existing catalog
Pipeline	The transformation and its execution options	Parallelism and checkpoint interval

Use CREATE TABLE ... WITH (connector = ...) to register each connection. Use CREATE STREAMING PIPELINE ... AS INSERT INTO ... SELECT ... to connect them.

Creating starts execution. Running CREATE STREAMING PIPELINE starts the pipeline by default. Add start_paused = true to save a pipeline for later.

Choose a starting point

Your goal	Read
Check services, access, and example values	Before you start
Run a complete Kafka-to-Iceberg flow	Your first pipeline
Define Kafka input and choose offsets	Sources
Define an Iceberg destination	Sinks
Filter records, calculate values, or use a CTE	Transformations
Inspect, stop, resume, or retry a request	Manage pipelines
Look up command syntax and defaults	Streaming SQL reference

Connections and catalog tables

A Streaming source or sink name is a saved connector definition. For example, paid_orders_sink can point to the Iceberg table analytics.paid_orders. Registering the sink does not create a Unified catalog connection for reading that table. To query the output, connect Unified to the same Iceberg catalog using the relevant catalog recipe. Use that catalog’s qualified table name for reads.

Execution and visibility

A successful create response means Laminar accepted and saved the pipeline. Workers then start it and publish output through the Streaming execution flow. Closing Workbench or the SQL client leaves the pipeline running. Pipeline acceptance, running workers, and visible Iceberg rows are separate stages. Use DESCRIBE PIPELINE for the pipeline state. Use the shared Observability docs for metrics and operational visibility. These pages describe the Streaming SQL commands. Streaming transformations use the Streaming planner’s SQL rules. The general SQL reference covers Unified queries; its full set of functions and commands does not apply to every Streaming transformation.

Transformation Engine

Learn Laminar

Workloads

SQL reference

HTTP API

Recipes

Operations

How a pipeline works

Choose a starting point

Connections and catalog tables

Execution and visibility

​How a pipeline works

​Choose a starting point

​Connections and catalog tables

​Execution and visibility

How a pipeline works

Choose a starting point

Connections and catalog tables

Execution and visibility