Overlap, missed runs, and queues

How-to guideApplies to: Unified engine deployments

Schedules can produce work faster than it finishes. Choose what should happen when a run is still active or the scheduler finds overdue occurrences.

Choose an overlap policy

Policy	Behavior	Useful when
Wait	Hold later work until earlier work permits it to proceed.	Runs write the same summary or must remain ordered.
Skip	Skip an occurrence that overlaps an earlier run.	A stale refresh is less useful than the next fresh one.
Allow	Permit overlapping runs, subject to capacity.	Runs can execute independently.

Wait does not turn the whole workflow into a transaction. Allow does not guarantee that workers are available. Incremental workloads require Wait or Skip to preserve cursor order.

Read a concrete timeline

Suppose the schedule is every five minutes and the 10:00 run takes twelve minutes.

Due boundary	Wait	Skip	Allow
10:00	Starts when capacity permits.	Starts when capacity permits.	Starts when capacity permits.
10:05	Waits behind active work.	Can be skipped because work is active.	Can run alongside 10:00.
10:10	Waits behind earlier work.	Can be skipped because work is active.	Can run alongside earlier runs.

Actual start times also depend on admission, allocation, and execution capacity. Inspect occurrences and runs to see which policy decision occurred.

Choose a missed-run policy

An occurrence is late when it exceeds the instance’s lateness tolerance. Catch-up behavior is separate from overlap:

Policy	Behavior
Run once	Coalesce overdue recurring work into one catch-up occurrence.
Run all	Retain missed boundaries for catch-up in order, within scheduler limits.
Skip	Skip overdue occurrences under the lateness rules.

For example, after a 10:00–10:20 outage, Run once does not mean a separate run for every five-minute boundary. Run all can build a backlog. Skip can leave gaps that require an explicit backfill. A catch-up run does not automatically cover every missed data period. If windows are enabled, inspect the selected run’s window. Use a backfill when you need an explicit historical range.

Understand queue limits

A scheduled occurrence can wait before a task reaches a worker. Task attempts can also wait for admission or compute. Check the saved queue reason and deadline instead of treating every wait as slow SQL. The scheduler bounds pending runs per workload. Queue timeout bounds an attempt’s queue wait under the execution rules. Neither setting increases capacity. If arrivals keep exceeding completion, increase the interval, reduce run duration, add appropriate capacity, or choose a policy that skips stale work. Do not select Allow for a shared append target without considering duplicate or competing writes.

Inspect before changing settings

Compare the due time, run creation time, admission, task queue phases, and execution duration. For the inline orders example, SQL is small; most delay should be explained by scheduling or available capacity. See Inspect runs and attempts, Backfills, and Scheduler configuration.

Transformation Engine

Learn Laminar

Workloads

SQL reference

HTTP API

Recipes

Operations

Overlap, missed runs, and queues

Choose an overlap policy

Read a concrete timeline

Choose a missed-run policy

Understand queue limits

Inspect before changing settings

​Choose an overlap policy

​Read a concrete timeline

​Choose a missed-run policy

​Understand queue limits

​Inspect before changing settings

Choose an overlap policy

Read a concrete timeline

Choose a missed-run policy

Understand queue limits

Inspect before changing settings