> ## Documentation Index
> Fetch the complete documentation index at: https://docs.e6data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Resources and capacity

> Size the services and task workers separately. Services keep planning and durable execution state. Workers need memory for source buffers, the Native Engine, result buffers, and pr

ReferenceApplies to: Unified engine deployments

Size the services and task workers separately. Services keep planning and durable execution state. Workers need memory for source buffers, the Native Engine, result buffers, and process overhead. Python notebook runtimes add a separate resource demand.

## Task requests and enforcement

Task compute requirements contain `cpu_millicores` and `memory_bytes`. The default task request is 2000 millicores and 2147483648 bytes, or 2 CPUs and 2 GiB. These are engine defaults, not measured minimum requirements for a full deployment or the entire TPC-DS workload.

| Worker backend | How resources are applied |
| - | - |
| In process | Checks each worker request against configured per-worker capacity; no OS isolation |
| Local process | Sends requirements to the child worker; the provider does not impose container resource limits |
| Docker | Sets container CPU and memory limits from the allocation |
| Kubernetes | Sets worker container CPU and memory requests and limits from the allocation |
| Lambda | Uses the published function's capacity and validates compatible task requests |

Increasing in-process capacity does not reserve that memory on the host. Several concurrent workers still share the host's actual resources.

## Execution tuning

Compute requirements can also contain execution tuning. It controls such values as Native Engine threads, query memory, connector buffers, batch size, output buffering, and spill. The engine validates tuning against the allocation budget.

Task tuning can override Native Engine base settings. Raising only the worker's total memory does not necessarily raise a particular connector buffer or query-memory limit. Use the effective task settings when comparing a failure with its available capacity.

The complete Native Engine base section below uses 2 threads, a 1 GiB memory setting, and a 4 GiB spill setting:

```toml theme={"theme":{"light":"github-light","dark":"github-dark"}}
[workers.runtime.duckdb]
extension_paths = []
threads = 2
memory_bytes = 1073741824
temporary_storage_bytes = 4294967296
output_buffer_batches = 2
```

Spill uses the temporary directory derived from `storage_root`. It must be writable in the worker environment. Task tuning may change the effective limits and may disable spill. The Native Engine memory setting is not a limit for the entire process.

## Service capacity

Allow service memory for active sessions, query plans, catalog requests, task state, result responses, and embedded features. A combined process shares these costs. Splitting services changes placement; it does not remove them.

Internal RPC concurrency, message sizes, source request concurrency, result page sizes, and retained Spark Connect responses have separate limits. Use the schema in Settings to inspect the current release. Large defaults are not a reason to admit more simultaneous work than the host can support.

## Disk and container placement

Provide persistent space for metadata, the secrets key, local results, and any local table data. Provide writable scratch space for spill and notebook execution. Scratch files are not a substitute for durable table or result storage.

On Kubernetes, include image pulls, worker startup, volume access, and scheduler placement in capacity planning. A Pod with no available placement is not ready compute. On Lambda, size memory and `/tmp` against the published function settings and task lifetime.

## Choose initial values

Use one known query or ETL task first, then increase input and concurrency while measuring memory, CPU, spill, and startup time. Keep the generated TPC-DS SF1 data fixed when comparing deployments. SF1 describes dataset scale; it does not mean every query fits in 1 GiB of memory.

Use a file-size target only where the output volume supports it. A roughly 100 MB Parquet target can suit larger writes, while small result tables may correctly produce a smaller file. File-size targets are separate from task memory, storage transfer parts, and commit behavior. See [Iceberg write options](https://docs.e6data.com/transformation/sql/writes.md#iceberg-write-options).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.