Checkpointing

Streaming pipelines run continuously, so recovery must preserve both processing state and progress through each source. A restart must not lose accepted data, repeat already-visible output, or combine state from different points in the stream. Ingest Engine uses coordinated checkpoints to establish recovery points across a pipeline. For sinks that support staged, atomic commits, checkpoint completion can also define when a set of output becomes visible.

​

What a checkpoint captures

A successful checkpoint represents one consistent point in the pipeline and may include:

• Replay positions for sources, such as queue offsets
• State for windows, joins, and aggregations
• Timers, watermarks, and other progress needed by stateful processing
• Metadata for sink output that has been staged but is not yet visible

Checkpoint data is recovery data. It is separate from the records written to the destination and should be stored on durable, highly available storage.

​

Barriers and consistent cuts

Ingest Engine coordinates a checkpoint with barriers that flow through the same paths as records.

1. Each source establishes a boundary in its input and records the corresponding replay position.
2. The source emits a barrier after the records that belong to the checkpoint.
3. Operators process all records before that barrier and snapshot the resulting state.
4. The barrier continues downstream until every branch has reached the same logical boundary.
5. Sinks finish staging the output produced before the boundary.

This produces a consistent cut: records before the cut are reflected in the saved state, while records after it are not.

​

Barrier alignment

An operator with multiple inputs may receive a checkpoint barrier on one input before another. It must not mix records from opposite sides of the checkpoint boundary. During alignment, the operator waits for the same barrier on all required inputs and temporarily holds later records from inputs that have already reached it. Once all inputs are aligned, the operator snapshots one consistent state and resumes normal processing. Alignment time grows when input rates are uneven or a branch is backpressured. Persistently slow alignment is therefore an important operational signal.

​

Source and operator state

Replayable sources save positions from which they can resume. Stateful operators save the data required to reconstruct open windows, join buffers, aggregates, and their event-time progress. On recovery, source positions and operator state must come from the same successful checkpoint. Combining a newer source position with older operator state could lose results; combining an older source position with newer state could double-count them.

​

Staged sink output

For a sink with checkpoint-aligned commits, output is first written to a staging area or transaction that is not yet visible to consumers. The checkpoint records enough destination metadata to either complete or safely abandon that staged output during recovery. Closing a file or sending a batch does not necessarily mean the checkpoint is committed. A checkpoint becomes a usable recovery point only after all required source, operator, and sink state has been durably recorded.

​

Conceptual two-phase commit

Sinks that can separate preparation from publication can align visibility with checkpoints:

1. Prepare - Sink tasks flush output and stage the destination changes associated with the checkpoint. Consumers cannot yet observe those changes.
2. Commit - After the checkpoint is durable, the destination publishes the prepared changes atomically.

Atomic publication means consumers see either the complete checkpoint output or none of it. They should not observe only a subset of parallel sink output. The destination-specific mechanism may be a catalog transaction, transactional producer, atomic rename, or another commit protocol. Not every sink supports this separation or atomic visibility.

​

Exactly-once prerequisites

End-to-end exactly-once behavior requires all of the following:

• Replayable source - The source can resume from the position saved in a checkpoint.
• Durable state - Checkpoint storage preserves the complete recovery point.
• Consistent restore - Source progress, operator state, and sink state are restored from the same checkpoint.
• Recoverable sink commit - A prepared commit can be retried, completed, or identified as already committed without publishing duplicates.
• Atomic visibility - Parallel output for one checkpoint becomes visible as one destination change when the sink contract promises exactly-once visibility.
• Stable processing contract - Schema and connector changes remain compatible with the saved state.

Nondeterministic SQL, calls to mutable external systems, or lookup data that changes during replay can produce different values even when source records are processed exactly once from the checkpoint boundary. Treat those dependencies as part of the pipeline’s delivery contract.

​

Failure scenarios

An incomplete checkpoint never replaces the last successful one. Recovery always starts from a checkpoint whose required state was fully persisted.

​

Connector and sink caveats

Delivery guarantees are connector-specific:

• A non-replayable source cannot reproduce records lost with a failed worker.
• A sink without transactions, atomic publication, or idempotent writes may provide at-least-once output even when operator state is checkpointed.
• Some sinks commit independently per partition; others require one destination-wide atomic transaction.
• A transactional sink may require extra configuration, such as unique transaction identities or destination permissions.
• External side effects performed outside the coordinated sink protocol are not covered by the checkpoint.

Use each connector’s documentation as the authoritative source for replay, commit, and visibility guarantees. Do not infer exactly-once behavior from checkpointing alone.

​

Choosing a checkpoint interval

The checkpoint interval balances recovery objectives against steady-state overhead. Window size and checkpoint interval are independent. A long-running window may span many checkpoints; each checkpoint must still preserve enough state to resume that open window correctly. Checkpoint duration should normally stay below the configured interval. If checkpoint work consistently approaches that interval, times out, or spends most of its time aligning barriers, increase capacity, reduce state pressure, tune the interval, or investigate a slow source or sink.

​

Operating checkpointed pipelines

Monitor at least:

• Time since the last successful checkpoint
• Checkpoint duration and failures
• Barrier-alignment time
• State size and checkpoint-storage latency
• Sink prepare and commit latency
• Recovery count and replay volume
• Abandoned or expired staged output

Apply storage lifecycle policies carefully so active recovery points are retained while obsolete checkpoint data and abandoned staging data do not grow without bound.

​

What checkpointing does not replace

Checkpointing is not a backup of destination data, a substitute for source retention, or a guarantee that every connector is exactly once. It provides coordinated recovery for pipeline progress and state. The final delivery guarantee remains the intersection of the source, processing logic, checkpoint storage, and sink commit contract.