Watermarks
e6 Ingestion Engine processes time-based queries using event time: the timestamp carried by a record, rather than the wall-clock time at which the engine happens to process it.
A watermark is the pipeline's declaration of event-time progress. When the watermark passes the end of a window, e6 Ingestion Engine can close that window, emit its result, and eventually release state that is no longer needed.
Event-time assignment
Declare event time with WATERMARK FOR in the source table:
CREATE TABLE events (
event_id BIGINT,
event_time TIMESTAMP NOT NULL,
WATERMARK FOR event_time AS event_time - INTERVAL '5 seconds'
);The selected field must be a timestamp, and the watermark expression must produce a non-null timestamp. e6 Ingestion Engine uses the selected field as the record's event time.
See the SQL DDL reference for the complete CREATE TABLE form.
If a table does not declare WATERMARK FOR, e6 Ingestion Engine uses the source-provided _timestamp, such as a Kafka record timestamp.
Choose a field whose values reflect when events occurred. Processing-time fields can produce misleading results during source delays or replays.
Generating watermarks
Sources periodically derive a watermark from the timestamps they have observed. An emitted watermark is a progress boundary: downstream operators may treat an older event as late. Configure watermark expressions so progress does not move backward for a source partition.
Without an explicit WATERMARK FOR ... AS ... expression, e6 Ingestion Engine uses _timestamp - 1 second. A custom expression controls the allowed out-of-order delay before a record is considered late.
The subtraction in the watermark expression is an intentional tradeoff:
A smaller delay closes windows sooner but tolerates less disorder.
A larger delay accepts later events but increases result latency and state retention.
For example, _timestamp - 30 seconds allows approximately 30 seconds of event-time disorder while delaying window completion by the same amount.
Propagation through a pipeline
Most transformations preserve the watermark. Time-aware operators use it to close windows, expire join state, or finish sessions.
For an operator with multiple inputs, progress is limited by the slowest input:
Every input has reported event time
Minimum reported timestamp
Any input has not reported a watermark
No combined watermark is emitted yet
Using the minimum prevents an operator from closing time-based state while another input may still provide earlier records.
Quiet inputs
The current public SQL configuration does not provide an idle-input timeout. A quiet partition or join input can therefore hold back watermark progress for a multi-input operator.
Design source partitioning so every active input can make progress, and monitor inputs that stop producing timestamps unexpectedly.
Windows and sessions
Watermarks determine when event-time results become eligible for emission:
Tumbling windows close after the watermark passes the fixed window end.
Sliding windows emit each completed slide whose covered interval is behind the watermark.
Session windows close after the watermark passes the configured inactivity gap for a key.
SQL window functions can emit completed event-time buckets once their boundary has passed.
The watermark controls completeness, not the amount of data in a window. A high-volume and a low-volume window with the same time boundary close at the same event-time point.
Late events
A record with an event timestamp earlier than the current watermark is late.
e6 Ingestion Engine does not provide a universal grace period or late-event side output, and late records are not guaranteed to update results that have already been finalized. To tolerate more out-of-order arrival, configure a larger delay in the watermark expression.
This makes the watermark policy part of the pipeline's data-quality contract. Choose it using observed source delay rather than assuming records arrive in order.
State retention
Watermarks also bound time-based state. Conceptually:
expiration boundary = watermark - retention
Records older than that boundary can be removed from joins, windows, and other retained state. Longer retention accommodates more delayed matches but increases checkpoint size, storage use, and recovery time.
Retention must be at least as long as the time relationship the query needs to preserve.
Checkpoints and recovery
Completed checkpoints preserve event-time progress together with operator state. On recovery, e6 Ingestion Engine restores both from the same checkpoint so windows and retention decisions remain consistent with restored data.
Records replayed after the checkpoint are evaluated against the restored watermark policy in the same way as newly arriving records.
Operational guidance
Watch for:
Growing difference between source event time and the current watermark
A partition or join input that stops advancing its watermark
Records arriving behind the current watermark
Increasing state size or checkpoint duration
A single input holding back multi-input operators
These signals usually indicate source skew, a quiet input, or a watermark delay that does not match real arrival patterns.
Last updated
Was this helpful?