Manage pipelines

How-to guideApplies to: Unified engine deployments

Create, inspect, stop, and resume Streaming pipelines from Workbench or a Spark Connect SQL client. The pipeline continues independently of the SQL session that created it.

Create and start

CREATE STREAMING PIPELINE starts execution by default. The source and sink must already be registered. Your first pipeline provides the full source, sink, and pipeline script. A successful SQL operation means the definition was accepted. Workers start asynchronously. Check the pipeline’s state before checking output rows.

List and inspect

SHOW PIPELINES;
DESCRIBE PIPELINE orders_to_iceberg;

Both commands return a result column with JSON metadata. SHOW includes pipelines created through the existing pipeline API as well as SQL.

Field	Meaning
`pipeline_id`	Stable native pipeline ID
`name`	Name used by the SQL lifecycle commands
`job_id`	The pipeline’s native job ID
`state`	Current execution state, such as `Created`, `Running`, or `Stopped`
`stop_mode`	Requested stop behavior; `none` permits running and `checkpoint` requests a checkpoint stop
`query`	Saved `INSERT INTO ... SELECT ...` transformation
`checkpoint_interval_micros`	Checkpoint interval in microseconds

The SQL operation can succeed while the pipeline is still starting. Its state and requested stop mode describe different things. For example, a stop request can return stop_mode: "checkpoint" while the state is still Running. Use the shared Observability docs for metrics and service visibility.

Save without starting

This example reuses the walkthrough’s registered connections and creates a separate, paused pipeline.

CREATE STREAMING PIPELINE reviewed_orders
WITH (
  parallelism = 2,
  checkpoint_interval = '10 seconds',
  start_paused = true
)
AS INSERT INTO paid_orders_sink
SELECT
  order_id,
  customer_id,
  amount,
  amount * 1.10 AS amount_with_tax
FROM orders_source
WHERE status = 'paid' AND amount > 0;

The initial state is Stopped, with stop_mode set to checkpoint. Review its definition before starting it.

DESCRIBE PIPELINE reviewed_orders;
START PIPELINE reviewed_orders;

This is a second pipeline with its own source progress. Stop the walkthrough pipeline first if both point to the same destination and you do not want duplicate input processing.

Stop and resume

STOP PIPELINE orders_to_iceberg;
DESCRIBE PIPELINE orders_to_iceberg;

STOP requests a checkpoint stop through the existing Streaming controller. It returns before that stop finishes. Repeat DESCRIBE until the state is Stopped.

START PIPELINE orders_to_iceberg;
DESCRIBE PIPELINE orders_to_iceberg;

START clears the stop request. The same pipeline resumes through its existing Streaming path and saved checkpoints. It does not create a new pipeline. It also does not prove that the workers have reached Running yet. START does not resume jobs whose state is Failed or Finished. Use the existing native pipeline controls for those jobs. Closing the browser tab or SQL session does not stop any pipeline.

Retry a request

When a response is lost, retry the original operation with the same session and operation ID. Laminar saves a receipt with the native mutation. A matching retry returns the same resource. Different content with the same identity is rejected. An old STOP retry cannot apply the stop again after a later START has been accepted. It returns the same pipeline’s current metadata instead. Running the SQL again as a fresh operation is a new request. Creating an existing name this way reports a name conflict. Creating a new pipeline name creates a separate job, with its own source progress.

Change or remove a definition

The current SQL commands cover connector registration, pipeline creation, listing, inspection, stop, and start. They do not include ALTER STREAMING PIPELINE, DROP PIPELINE, or SQL connector deletion. Use the existing pipeline and table controls for changes and removal. Stop a pipeline before deleting it. Remove connector definitions after their pipeline users are removed. Deleting a definition does not remove the Kafka topic or the Iceberg destination data.

Clean up the paused example

If you started reviewed_orders, stop it and wait for Stopped before removing it through the existing controls.

STOP PIPELINE reviewed_orders;
DESCRIBE PIPELINE reviewed_orders;

Continue

Streaming SQL reference lists the command forms, defaults, and validation limits.

Transformation Engine

Learn Laminar

Workloads

SQL reference

HTTP API

Recipes

Operations

Create and start

List and inspect

Save without starting

Stop and resume

Retry a request

Change or remove a definition

Clean up the paused example

Continue

​Create and start

​List and inspect

​Save without starting

​Stop and resume

​Retry a request

​Change or remove a definition

​Clean up the paused example

​Continue

Create and start

List and inspect

Save without starting

Stop and resume

Retry a request

Change or remove a definition

Clean up the paused example

Continue