Submission defaults

[submission_defaults]

​

Settings

​

worker_scope

Chooses whether a worker is allocated for one task or reused within a workload run.

• Default: "Task"
• Type: choice: Task, Workload

Example "Task" gives tasks their own allocations. "Workload" lets tasks in the same run reuse a worker pool.

​

parallelism

Limits how many tasks in one workload run may execute at the same time.

• Default: Unset
• Type: integer; positive; optional

Example 2 allows up to two concurrent tasks. If unset, workload-scoped workers default to one; task-scoped workers have no additional cap here.

​

execution_target

Chooses the engine expected to execute the task.

• Default: "DuckDb"
• Type: choice: DataFusion, DuckDb

Example "DuckDb" selects Native Engine. DataFusion is another serialized target, but it needs a compatible executor; the current native worker starts Native Engine.

​

execution_timeout

Limits how long a task is allowed to execute.

• Default: Unset
• Type: duration; optional

Example { secs = 300, nanos = 0 } gives a task five minutes. Leaving this optional field unset gives the task no execution time limit.

​

compute_requirements.cpu_millicores

Sets the requested CPU capacity. A value of 1000 means one CPU.

• Default: 2000
• Type: integer; positive

Example 2000 requests two CPUs; 500 requests half a CPU. This is the allocation request, not a command to resize the controller.

​

compute_requirements.memory_bytes

Sets the memory requested for one worker allocation.

• Default: 2147483648
• Type: integer (bytes); positive

Example 2147483648 requests 2 GiB. For Kubernetes, the provider uses the allocation when setting the worker container’s memory request and limit.

​

max_attempts

Limits how many execution attempts may be made for a task.

• Default: 3
• Type: integer; positive

Example 3 allows up to three task attempts. An uncertain write still needs its saved outcome checked before another write attempt can be made.

​

missed_run_policy

Chooses what happens to scheduled occurrences that were missed.

• Default: "RunOnce"
• Type: choice: RunOnce, RunAll, Skip

Example If three scheduled times passed while the service was unavailable, RunOnce catches up once, RunAll considers all three, and Skip ignores them.

​

overlap_policy

Chooses what happens when a scheduled run is due while an earlier run is still active.

• Default: "Wait"
• Type: choice: Wait, Skip, Allow

Example Wait queues the next run, Skip ignores that overlapping occurrence, and Allow permits both runs to be active.

​

Execution tuning

​

compute_requirements.tuning

Sets detailed execution budgets within an allocation’s CPU and memory.

• Default: Unset
• Type: object; optional

Example For a two-CPU worker, set threads = 2 and explicit query, buffer, and spill budgets here. Omitted tuning values are calculated from the allocation.

​

compute_requirements.tuning.threads

Chooses how many execution threads one task may use within its assigned CPU allocation.

• Default: Unset
• Type: integer; positive; optional

Example For a task allocated 2000 millicores, 2 allows two execution threads. Leave it unset to derive a value from the allocation; setting more threads does not allocate more CPUs.

​

compute_requirements.tuning.query_memory_bytes

Sets how much of a worker’s allocated memory may be used by the query engine.

• Default: Unset
• Type: integer (bytes); positive; optional

Example On a 2 GiB worker, 1073741824 reserves a 1 GiB query budget. Leave room for connector buffers and process overhead; this budget cannot exceed 80% of the allocation.

​

compute_requirements.tuning.buffer_bytes

Sets the memory budget used for task data buffers.

• Default: Unset
• Type: integer (bytes); positive; optional

Example 67108864 gives a 64 MiB buffer budget. It must fit alongside query memory and the configured number of output buffers.

​

compute_requirements.tuning.batch_rows

Sets how many rows may be handled in one reader batch.

• Default: Unset
• Type: integer; positive; optional

Example 65536 reads up to 65,536 rows per batch when this setting is omitted. Smaller batches can reduce memory held at once; the maximum is 1048576.

​

compute_requirements.tuning.write_batch_rows

Sets the maximum rows per sink batch independently of reader batches. The executor also limits batching by its byte budget. One oversized input value may need a larger buffer.

• Default: Unset; resolves to 65536
• Type: integer; positive; at most 1048576; optional

​

compute_requirements.tuning.write_parallelism

Sets the number of worker-local Iceberg file writers. Writers use bounded queues and share a memory budget. All writers must finish before one Iceberg commit publishes their files.

• Default: Unset; resolves to execution threads
• Type: integer; positive; must not exceed execution threads; optional

Example On a four-core worker, write_parallelism: 4 allows four file writers. Each writer can encode and upload while the other writers continue. A one-core worker defaults to one writer.

​

compute_requirements.tuning.spill_bytes

Limits task data temporarily written to disk when execution needs it.

• Default: Unset
• Type: integer (bytes); optional

Example 8589934592 gives an 8 GiB task spill budget. Zero disables task spill, while an omitted value is calculated from allocation memory.

​

compute_requirements.tuning.output_buffer_batches

Limits how many output batches wait for the next consumer.

• Default: Unset
• Type: integer; positive; optional

Example 2 allows two batches to wait. Raising this can keep more data in memory; task tuning allows at most 32.

​

Resource budgets

Tuning is optional under each task’s compute_requirements object. Unset tuning derives values from the task allocation. Set task defaults here; tuning in workers.provisioners.<name>.capacity does not set task execution budgets. To make budgets explicit, set every tuning field. Threads cannot exceed rounded-up allocated CPUs. Query memory and the configured buffers must leave runtime overhead; individually valid limits can still be rejected together.