Incremental workloads

How-to guideApplies to: Unified engine deployments

Process changes from a tracked source table using saved progress. This is different from filtering rows by a time window or running a continuous streaming pipeline.

Choose the right model

Model	Progress is defined by
Ordinary batch workload	The SQL and parameters for each run.
Windowed batch workload	An explicit time interval used by the SQL.
Incremental workload	The source table’s saved change cursor.
Continuous streaming pipeline	The streaming source and its checkpointed progress.

Use Data windows when business time defines a period. Use Streaming for continuous pipeline definitions.

Configure an incremental source

Choose a supported source table and its main branch. An incremental workload uses one SQL task without additional graph branches or dependencies. The source relation identifies the catalog ID, namespace, and table name. A task’s source configuration has this shape:

{
  "incremental_source": {
    "table": {
      "catalog_id": "replace-with-catalog-id",
      "namespace": ["analytics"],
      "name": "orders"
    },
    "branch": "main"
  }
}

The source table must support the required change-reading behavior. A normal SQL relation or an inline WITH dataset does not provide an incremental cursor. This example therefore needs a real supported table.

Preserve progress semantics

Successful publication advances the saved source progress under the engine’s commit rules. A failed or uncertain run must not be treated as permission to skip forward manually. Use Wait or Skip for overlap. Allow is rejected because concurrent cursor movement would break ordered progress. Custom parameters need fixed defaults, and run overrides must match them. Ordinary run-value built-ins are not available for incremental SQL.

Verify with a small source

Create a private supported source and destination using your writable catalog.
Insert a small initial batch and run the incremental workload.
Verify the destination and confirmed commit.
Add another source batch and start another run.
Confirm that the new run follows saved source progress instead of intentionally re-reading the whole table.

Record source versions, run IDs, and destination commits. Exact change visibility depends on the source connector and supported table operations.

Edit and recover carefully

Changing filters, parameters, source identity, or publication rules changes what the cursor means. Review those changes as a migration rather than assuming old progress can be reused without consequences. Selective reruns and historical backfills do not operate on incremental cursors. A new ordinary Run again continues the saved incremental progress; it does not select an arbitrary old time interval. See Results and table commits for uncertain publication.

Transformation Engine

Learn Laminar

Workloads

SQL reference

HTTP API

Recipes

Operations

Incremental workloads

Choose the right model

Configure an incremental source

Preserve progress semantics

Verify with a small source

Edit and recover carefully

​Choose the right model

​Configure an incremental source

​Preserve progress semantics

​Verify with a small source

​Edit and recover carefully

Choose the right model

Configure an incremental source

Preserve progress semantics

Verify with a small source

Edit and recover carefully