> ## Documentation Index
> Fetch the complete documentation index at: https://docs.e6data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Incremental workloads

> Process changes from a tracked source table using saved progress. This is different from filtering rows by a time window or running a continuous streaming pipeline.

How-to guideApplies to: Unified engine deployments

Process changes from a tracked source table using saved progress. This is different from filtering rows by a time window or running a continuous streaming pipeline.

## Choose the right model

| Model | Progress is defined by |
| - | - |
| Ordinary batch workload | The SQL and parameters for each run. |
| Windowed batch workload | An explicit time interval used by the SQL. |
| Incremental workload | The source table's saved change cursor. |
| Continuous streaming pipeline | The streaming source and its checkpointed progress. |

Use [Data windows](https://docs.e6data.com/transformation/workloads/data-windows.md) when business time defines a period. Use [Streaming](https://docs.e6data.com/transformation/streaming) for continuous pipeline definitions.

## Configure an incremental source

Choose a supported source table and its `main` branch. An incremental workload uses one SQL task without additional graph branches or dependencies. The source relation identifies the catalog ID, namespace, and table name.

A task's source configuration has this shape:

```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
  "incremental_source": {
    "table": {
      "catalog_id": "replace-with-catalog-id",
      "namespace": ["analytics"],
      "name": "orders"
    },
    "branch": "main"
  }
}
```

The source table must support the required change-reading behavior. A normal SQL relation or an inline `WITH` dataset does not provide an incremental cursor. This example therefore needs a real supported table.

## Preserve progress semantics

Successful publication advances the saved source progress under the engine's commit rules. A failed or uncertain run must not be treated as permission to skip forward manually.

Use Wait or Skip for overlap. Allow is rejected because concurrent cursor movement would break ordered progress. Custom parameters need fixed defaults, and run overrides must match them. Ordinary run-value built-ins are not available for incremental SQL.

## Verify with a small source

1. Create a private supported source and destination using your writable catalog.
2. Insert a small initial batch and run the incremental workload.
3. Verify the destination and confirmed commit.
4. Add another source batch and start another run.
5. Confirm that the new run follows saved source progress instead of intentionally re-reading the whole table.

Record source versions, run IDs, and destination commits. Exact change visibility depends on the source connector and supported table operations.

## Edit and recover carefully

Changing filters, parameters, source identity, or publication rules changes what the cursor means. Review those changes as a migration rather than assuming old progress can be reused without consequences.

Selective reruns and historical backfills do not operate on incremental cursors. A new ordinary Run again continues the saved incremental progress; it does not select an arbitrary old time interval. See [Results and table commits](https://docs.e6data.com/transformation/workloads/results-and-commits.md) for uncertain publication.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.