Process new data

TutorialApplies to: Unified engine deployments

Copy an initial source batch, append one new order, and run again. Verify that the output grows by one row because the workload keeps its source position.

Before you start

You need incremental SQL workloads and a writable Iceberg catalog whose connector supports append snapshot intervals. A snapshot is a saved version of a table. This lesson uses append changes only. Choose an unused namespace. The example uses learn_incremental_01. Replace it everywhere if needed. Keep the same selected catalog throughout. Use a new workload and empty target for this exercise.

1. Create separate source and target tables

In Workbench, select your writable catalog. Run these statements separately in the same session:

SET spark.sql.caseSensitive = true;
CREATE NAMESPACE learn_incremental_01;
CREATE TABLE learn_incremental_01.orders_source (
  order_id BIGINT, amount BIGINT
) USING iceberg;
CREATE TABLE learn_incremental_01.orders_target (
  order_id BIGINT, amount BIGINT
) USING iceberg;
INSERT INTO learn_incremental_01.orders_source VALUES (1, 20), (2, 30);

Append the initial rows once. Expect two source rows and no target rows. Keep these tables separate from the scheduled summary lesson.

2. Define one incremental task

Create Lesson - incremental orders in Workloads.
Set the query catalog and namespace to your selected catalog and learn_incremental_01. Use case-sensitive names.
Add one SQL task named copy_orders with the SQL below.
Enable Track progress for a source table. Select the same catalog, namespace learn_incremental_01, table orders_source, and branch main.
Choose available compute, a Manual schedule, and Wait or Skip for overlap.
Validate and create the workload.

INSERT INTO learn_incremental_01.orders_target
SELECT order_id, amount FROM learn_incremental_01.orders_source;

Run this SQL through the configured incremental workload. Running it directly in Workbench would read the whole source and append it again without this workload’s progress tracking. The current incremental mode requires one SQL task, an existing different target, and supported source changes. Do not add a predecessor task. Do not select Allow for overlapping runs.

3. Check the initial publication

Start one run. Wait for confirmed publication, then query the target in Workbench:

SELECT order_id, amount
FROM learn_incremental_01.orders_target
ORDER BY order_id;

Expect exactly (1, 20) and (2, 30). The total amount is 50. Inspect the run’s source progress and confirmed write details. Progress advances after the write is confirmed.

Change one thing

Append one new source row, once:

INSERT INTO learn_incremental_01.orders_source VALUES (3, 15);

Start the same workload again. Use the same definition and target. Query the target again. Expect exactly (1, 20), (2, 30), and (3, 15), with total amount 65.

4. Verify that old rows were not copied again

In Workbench, check:

SELECT order_id, COUNT(*) AS copies
FROM learn_incremental_01.orders_target
GROUP BY order_id
ORDER BY order_id;

Expect one copy of each order. Start the same workload again without changing the source. Once it settles, the target should still have three rows. Inspect the run’s progress information to distinguish new work from an unchanged source.

Check the result

The initial publication copied two rows. The next publication added only order 3. Repeating without new source rows did not add duplicates. Keep the source and workload identity stable. Creating another workload gives it a new progress identity. Dropping and recreating the source also changes its identity. Neither action continues the original saved position. This is not general row-level change data capture. Overwrite and delete snapshots may be unsupported. Do not replace a full aggregate with an aggregate of only the latest delta; that would lose earlier totals.

Clean up

Leave scheduling paused and wait for active work to finish. Delete the lesson workload if no longer needed. Then remove only its private tables:

DROP TABLE learn_incremental_01.orders_target;
DROP TABLE learn_incremental_01.orders_source;
DROP NAMESPACE learn_incremental_01;

Continue

The incremental workloads guide lists definition and source limits.

Transformation Engine

Learn Laminar

Workloads

SQL reference

HTTP API

Recipes

Operations

Before you start

1. Create separate source and target tables

2. Define one incremental task

3. Check the initial publication

Change one thing

4. Verify that old rows were not copied again

Check the result

Clean up

Continue

​Before you start

​1. Create separate source and target tables

​2. Define one incremental task

​3. Check the initial publication

​Change one thing

​4. Verify that old rows were not copied again

​Check the result

​Clean up

​Continue

Before you start

1. Create separate source and target tables

2. Define one incremental task

3. Check the initial publication

Change one thing

4. Verify that old rows were not copied again

Check the result

Clean up

Continue