> For the complete documentation index, see [llms.txt](https://docs.e6data.com/ingestion-engine/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.e6data.com/ingestion-engine/architecture/checkpointing.md).

# Checkpointing

Streaming pipelines run continuously, so recovery must preserve both processing state and progress through each source. A restart must not lose accepted data, repeat already-visible output, or combine state from different points in the stream.

e6 Ingestion Engine uses coordinated checkpoints to establish recovery points across a pipeline. For sinks that support staged, atomic commits, checkpoint completion can also define when a set of output becomes visible.

{% hint style="info" %}
Checkpointing is necessary for exactly-once processing, but it is not sufficient by itself. The end-to-end guarantee also depends on the source, sink, and connector configuration.
{% endhint %}

## What a checkpoint captures

A successful checkpoint represents one consistent point in the pipeline and may include:

* Replay positions for sources, such as queue offsets
* State for windows, joins, and aggregations
* Timers, watermarks, and other progress needed by stateful processing
* Metadata for sink output that has been staged but is not yet visible

Checkpoint data is recovery data. It is separate from the records written to the destination and should be stored on durable, highly available storage.

## Barriers and consistent cuts

e6 Ingestion Engine coordinates a checkpoint with barriers that flow through the same paths as records.

1. Each source establishes a boundary in its input and records the corresponding replay position.
2. The source emits a barrier after the records that belong to the checkpoint.
3. Operators process all records before that barrier and snapshot the resulting state.
4. The barrier continues downstream until every branch has reached the same logical boundary.
5. Sinks finish staging the output produced before the boundary.

This produces a **consistent cut**: records before the cut are reflected in the saved state, while records after it are not.

### Barrier alignment

An operator with multiple inputs may receive a checkpoint barrier on one input before another. It must not mix records from opposite sides of the checkpoint boundary.

During alignment, the operator waits for the same barrier on all required inputs and temporarily holds later records from inputs that have already reached it. Once all inputs are aligned, the operator snapshots one consistent state and resumes normal processing.

![Barrier alignment creates a consistent cut across inputs](https://1725648790-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FuztQ31u5OyLCkPgUTosp%2Fuploads%2Fgit-blob-4af8ec2ca789aa3aaa2127e87039ad70c740dd43%2Fcheckpoint-keyframes-barrier-alignment.svg?alt=media)

Alignment time grows when input rates are uneven or a branch is backpressured. Persistently slow alignment is therefore an important operational signal.

### Source and operator state

Replayable sources save positions from which they can resume. Stateful operators save the data required to reconstruct open windows, join buffers, aggregates, and their event-time progress.

On recovery, source positions and operator state must come from the same successful checkpoint. Combining a newer source position with older operator state could lose results; combining an older source position with newer state could double-count them.

## Staged sink output

For a sink with checkpoint-aligned commits, output is first written to a staging area or transaction that is not yet visible to consumers. The checkpoint records enough destination metadata to either complete or safely abandon that staged output during recovery.

![Output is staged before it becomes visible](https://1725648790-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FuztQ31u5OyLCkPgUTosp%2Fuploads%2Fgit-blob-5d90fe3c936a34804b3e10f325aa21fdef775e16%2Ftwo-phase-commit-keyframes-starting-state.svg?alt=media)

Closing a file or sending a batch does not necessarily mean the checkpoint is committed. A checkpoint becomes a usable recovery point only after all required source, operator, and sink state has been durably recorded.

## Conceptual two-phase commit

Sinks that can separate preparation from publication can align visibility with checkpoints:

1. **Prepare** - Sink tasks flush output and stage the destination changes associated with the checkpoint. Consumers cannot yet observe those changes.
2. **Commit** - After the checkpoint is durable, the destination publishes the prepared changes atomically.

Atomic publication means consumers see either the complete checkpoint output or none of it. They should not observe only a subset of parallel sink output.

![A successful commit makes the complete checkpoint output visible atomically](https://1725648790-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FuztQ31u5OyLCkPgUTosp%2Fuploads%2Fgit-blob-fab9f6b484460293e19a3d82ab3878afdfdc085a%2Ftwo-phase-commit-keyframes-commit-confirmed.svg?alt=media)

The destination-specific mechanism may be a catalog transaction, transactional producer, atomic rename, or another commit protocol. Not every sink supports this separation or atomic visibility.

## Exactly-once prerequisites

End-to-end exactly-once behavior requires all of the following:

* **Replayable source** - The source can resume from the position saved in a checkpoint.
* **Durable state** - Checkpoint storage preserves the complete recovery point.
* **Consistent restore** - Source progress, operator state, and sink state are restored from the same checkpoint.
* **Recoverable sink commit** - A prepared commit can be retried, completed, or identified as already committed without publishing duplicates.
* **Atomic visibility** - Parallel output for one checkpoint becomes visible as one destination change when the sink contract promises exactly-once visibility.
* **Stable processing contract** - Schema and connector changes remain compatible with the saved state.

Nondeterministic SQL, calls to mutable external systems, or lookup data that changes during replay can produce different values even when source records are processed exactly once from the checkpoint boundary. Treat those dependencies as part of the pipeline's delivery contract.

## Failure scenarios

| Failure point                                      | Recovery behavior                                                                                                        | Visibility                                                                           |
| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------ |
| Before the checkpoint completes                    | Discard the incomplete attempt and restore the latest successful checkpoint. Records after that checkpoint are replayed. | Prepared output from the failed attempt must remain invisible or be cleaned up.      |
| After checkpoint completion but before sink commit | Restore the completed checkpoint and finish or retry its prepared commit.                                                | Output becomes visible once, provided the sink commit is recoverable and idempotent. |
| While the commit outcome is unknown                | Reconcile with the destination using its transaction or commit identity before retrying.                                 | The destination must distinguish an already-applied commit from a new one.           |
| After a successful commit                          | Restore from the matching checkpoint and continue after its saved source positions.                                      | Previously committed output remains visible and is not republished.                  |
| Checkpoint storage is unavailable or unusable      | Recovery from that checkpoint cannot proceed safely.                                                                     | Operators must not infer success from partially available state.                     |

An incomplete checkpoint never replaces the last successful one. Recovery always starts from a checkpoint whose required state was fully persisted.

## Connector and sink caveats

Delivery guarantees are connector-specific:

* A non-replayable source cannot reproduce records lost with a failed worker.
* A sink without transactions, atomic publication, or idempotent writes may provide at-least-once output even when operator state is checkpointed.
* Some sinks commit independently per partition; others require one destination-wide atomic transaction.
* A transactional sink may require extra configuration, such as unique transaction identities or destination permissions.
* External side effects performed outside the coordinated sink protocol are not covered by the checkpoint.

Use each connector's documentation as the authoritative source for replay, commit, and visibility guarantees. Do not infer exactly-once behavior from checkpointing alone.

## Choosing a checkpoint interval

The checkpoint interval balances recovery objectives against steady-state overhead.

| Shorter interval                                | Longer interval                                   |
| ----------------------------------------------- | ------------------------------------------------- |
| Less data replay after failure                  | More data replay after failure                    |
| More frequent state writes and sink preparation | Lower coordination and storage overhead           |
| Can increase pressure on large-state pipelines  | Allows more state and staged output to accumulate |
| Reduces time between durable recovery points    | Increases the recovery-point gap                  |

Window size and checkpoint interval are independent. A long-running window may span many checkpoints; each checkpoint must still preserve enough state to resume that open window correctly.

Checkpoint duration should normally stay below the configured interval. If checkpoint work consistently approaches that interval, times out, or spends most of its time aligning barriers, increase capacity, reduce state pressure, tune the interval, or investigate a slow source or sink.

## Operating checkpointed pipelines

Monitor at least:

* Time since the last successful checkpoint
* Checkpoint duration and failures
* Barrier-alignment time
* State size and checkpoint-storage latency
* Sink prepare and commit latency
* Recovery count and replay volume
* Abandoned or expired staged output

Apply storage lifecycle policies carefully so active recovery points are retained while obsolete checkpoint data and abandoned staging data do not grow without bound.

## What checkpointing does not replace

Checkpointing is not a backup of destination data, a substitute for source retention, or a guarantee that every connector is exactly once. It provides coordinated recovery for pipeline progress and state. The final delivery guarantee remains the intersection of the source, processing logic, checkpoint storage, and sink commit contract.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.e6data.com/ingestion-engine/architecture/checkpointing.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.