Skip to main content
ReferenceApplies to: Unified engine deployments
Size the services and task workers separately. Services keep planning and durable execution state. Workers need memory for source buffers, the Native Engine, result buffers, and process overhead. Python notebook runtimes add a separate resource demand.

​
Task requests and enforcement

Task compute requirements contain cpu_millicores and memory_bytes. The default task request is 2000 millicores and 2147483648 bytes, or 2 CPUs and 2 GiB. These are engine defaults, not measured minimum requirements for a full deployment or the entire TPC-DS workload. Increasing in-process capacity does not reserve that memory on the host. Several concurrent workers still share the host’s actual resources.

​
Execution tuning

Compute requirements can also contain execution tuning. It controls such values as Native Engine threads, query memory, connector buffers, batch size, output buffering, and spill. The engine validates tuning against the allocation budget. Task tuning can override Native Engine base settings. Raising only the worker’s total memory does not necessarily raise a particular connector buffer or query-memory limit. Use the effective task settings when comparing a failure with its available capacity. The complete Native Engine base section below uses 2 threads, a 1 GiB memory setting, and a 4 GiB spill setting:
Spill uses the temporary directory derived from storage_root. It must be writable in the worker environment. Task tuning may change the effective limits and may disable spill. The Native Engine memory setting is not a limit for the entire process.

​
Service capacity

Allow service memory for active sessions, query plans, catalog requests, task state, result responses, and embedded features. A combined process shares these costs. Splitting services changes placement; it does not remove them. Internal RPC concurrency, message sizes, source request concurrency, result page sizes, and retained Spark Connect responses have separate limits. Use the schema in Settings to inspect the current release. Large defaults are not a reason to admit more simultaneous work than the host can support.

​
Disk and container placement

Provide persistent space for metadata, the secrets key, local results, and any local table data. Provide writable scratch space for spill and notebook execution. Scratch files are not a substitute for durable table or result storage. On Kubernetes, include image pulls, worker startup, volume access, and scheduler placement in capacity planning. A Pod with no available placement is not ready compute. On Lambda, size memory and /tmp against the published function settings and task lifetime.

​
Choose initial values

Use one known query or ETL task first, then increase input and concurrency while measuring memory, CPU, spill, and startup time. Keep the generated TPC-DS SF1 data fixed when comparing deployments. SF1 describes dataset scale; it does not mean every query fits in 1 GiB of memory. Use a file-size target only where the output volume supports it. A roughly 100 MB Parquet target can suit larger writes, while small result tables may correctly produce a smaller file. File-size targets are separate from task memory, storage transfer parts, and commit behavior. See Iceberg write options.
Last modified on November 4, 2026