Event-time assignment
Declare event time withWATERMARK FOR in the source table:
CREATE TABLE form.
If a table does not declare WATERMARK FOR, Ingest Engine uses the source-provided _timestamp, such as a Kafka record timestamp.
Choose a field whose values reflect when events occurred. Processing-time fields can produce misleading results during source delays or replays.
Generating watermarks
Sources periodically derive a watermark from the timestamps they have observed. An emitted watermark is a progress boundary: downstream operators may treat an older event as late. Configure watermark expressions so progress does not move backward for a source partition. Without an explicitWATERMARK FOR ... AS ... expression, Ingest Engine uses _timestamp - 1 second. A custom expression controls the allowed out-of-order delay before a record is considered late.
The subtraction in the watermark expression is an intentional tradeoff:
- A smaller delay closes windows sooner but tolerates less disorder.
- A larger delay accepts later events but increases result latency and state retention.
_timestamp - 30 seconds allows approximately 30 seconds of event-time disorder while delaying window completion by the same amount.
Propagation through a pipeline
Using the minimum prevents an operator from closing time-based state while another input may still provide earlier records.
Quiet inputs
The current public SQL configuration does not provide an idle-input timeout. A quiet partition or join input can therefore hold back watermark progress for a multi-input operator. Design source partitioning so every active input can make progress, and monitor inputs that stop producing timestamps unexpectedly.Windows and sessions
Watermarks determine when event-time results become eligible for emission:- Tumbling windows close after the watermark passes the fixed window end.
- Sliding windows emit each completed slide whose covered interval is behind the watermark.
- Session windows close after the watermark passes the configured inactivity gap for a key.
- SQL window functions can emit completed event-time buckets once their boundary has passed.
Late events
A record with an event timestamp earlier than the current watermark is late. Ingest Engine does not provide a universal grace period or late-event side output, and late records are not guaranteed to update results that have already been finalized. To tolerate more out-of-order arrival, configure a larger delay in the watermark expression. This makes the watermark policy part of the pipeline’s data-quality contract. Choose it using observed source delay rather than assuming records arrive in order.State retention
Watermarks also bound time-based state. Conceptually:expiration boundary = watermark - retention
Records older than that boundary can be removed from joins, windows, and other retained state. Longer retention accommodates more delayed matches but increases checkpoint size, storage use, and recovery time.
Retention must be at least as long as the time relationship the query needs to preserve.
Checkpoints and recovery
Completed checkpoints preserve event-time progress together with operator state. On recovery, Ingest Engine restores both from the same checkpoint so windows and retention decisions remain consistent with restored data. Records replayed after the checkpoint are evaluated against the restored watermark policy in the same way as newly arriving records.Operational guidance
Watch for:- Growing difference between source event time and the current watermark
- A partition or join input that stops advancing its watermark
- Records arriving behind the current watermark
- Increasing state size or checkpoint duration
- A single input holding back multi-input operators