> ## Documentation Index
> Fetch the complete documentation index at: https://docs.e6data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Process new data

> Copy an initial source batch, append one new order, and run again. Verify that the output grows by one row because the workload keeps its source position.

TutorialApplies to: Unified engine deployments

Copy an initial source batch, append one new order, and run again. Verify that the output grows by one row because the workload keeps its source position.

## Before you start

You need incremental SQL workloads and a writable Iceberg catalog whose connector supports append snapshot intervals. A **snapshot** is a saved version of a table. This lesson uses append changes only.

Choose an unused namespace. The example uses `learn_incremental_01`. Replace it everywhere if needed. Keep the same selected catalog throughout. Use a new workload and empty target for this exercise.

## 1. Create separate source and target tables

In Workbench, select your writable catalog. Run these statements separately in the same session:

```sql theme={"theme":{"light":"github-light","dark":"github-dark"}}
SET spark.sql.caseSensitive = true;
CREATE NAMESPACE learn_incremental_01;
CREATE TABLE learn_incremental_01.orders_source (
  order_id BIGINT, amount BIGINT
) USING iceberg;
CREATE TABLE learn_incremental_01.orders_target (
  order_id BIGINT, amount BIGINT
) USING iceberg;
INSERT INTO learn_incremental_01.orders_source VALUES (1, 20), (2, 30);
```

Append the initial rows once. Expect two source rows and no target rows. Keep these tables separate from the scheduled summary lesson.

## 2. Define one incremental task

1. Create `Lesson - incremental orders` in Workloads.
2. Set the query catalog and namespace to your selected catalog and `learn_incremental_01`. Use case-sensitive names.
3. Add one SQL task named `copy_orders` with the SQL below.
4. Enable **Track progress for a source table**. Select the same catalog, namespace `learn_incremental_01`, table `orders_source`, and branch `main`.
5. Choose available compute, a Manual schedule, and Wait or Skip for overlap.
6. Validate and create the workload.

```sql theme={"theme":{"light":"github-light","dark":"github-dark"}}
INSERT INTO learn_incremental_01.orders_target
SELECT order_id, amount FROM learn_incremental_01.orders_source;
```

Run this SQL through the configured incremental workload. Running it directly in Workbench would read the whole source and append it again without this workload's progress tracking.

The current incremental mode requires one SQL task, an existing different target, and supported source changes. Do not add a predecessor task. Do not select Allow for overlapping runs.

## 3. Check the initial publication

Start one run. Wait for confirmed publication, then query the target in Workbench:

```sql theme={"theme":{"light":"github-light","dark":"github-dark"}}
SELECT order_id, amount
FROM learn_incremental_01.orders_target
ORDER BY order_id;
```

Expect exactly **(1, 20)** and **(2, 30)**. The total amount is **50**. Inspect the run's source progress and confirmed write details. Progress advances after the write is confirmed.

## Change one thing

Append one new source row, once:

```sql theme={"theme":{"light":"github-light","dark":"github-dark"}}
INSERT INTO learn_incremental_01.orders_source VALUES (3, 15);
```

Start the same workload again. Use the same definition and target. Query the target again. Expect exactly **(1, 20), (2, 30), and (3, 15)**, with total amount **65**.

## 4. Verify that old rows were not copied again

In Workbench, check:

```sql theme={"theme":{"light":"github-light","dark":"github-dark"}}
SELECT order_id, COUNT(*) AS copies
FROM learn_incremental_01.orders_target
GROUP BY order_id
ORDER BY order_id;
```

Expect one copy of each order. Start the same workload again without changing the source. Once it settles, the target should still have three rows. Inspect the run's progress information to distinguish new work from an unchanged source.

## Check the result

The initial publication copied two rows. The next publication added only order 3. Repeating without new source rows did not add duplicates.

Keep the source and workload identity stable. Creating another workload gives it a new progress identity. Dropping and recreating the source also changes its identity. Neither action continues the original saved position.

This is not general row-level change data capture. Overwrite and delete snapshots may be unsupported. Do not replace a full aggregate with an aggregate of only the latest delta; that would lose earlier totals.

## Clean up

Leave scheduling paused and wait for active work to finish. Delete the lesson workload if no longer needed. Then remove only its private tables:

```sql theme={"theme":{"light":"github-light","dark":"github-dark"}}
DROP TABLE learn_incremental_01.orders_target;
DROP TABLE learn_incremental_01.orders_source;
DROP NAMESPACE learn_incremental_01;
```

## Continue

The incremental workloads guide lists definition and source limits.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.