> ## Documentation Index > Fetch the complete documentation index at: https://docs.e6data.com/llms.txt > Use this file to discover all available pages before exploring further. # Workers > workers ReferenceApplies to: Unified engine deployments `[workers]` ## Named provisioners and profiles Define provisioners and profiles in the controller file. A provisioner chooses where a worker runs. A profile names its CPU, memory, and execution tuning. ```toml theme={"theme":{"light":"github-light","dark":"github-dark"}} [workers] default_provisioner = "local" default_profile = "small" [workers.provisioners.local] provider = "local_process" working_directory = "/var/lib/laminar" [workers.provisioners.cluster] provider = "kubernetes" namespace = "analytics" image = "laminar:your-version" [workers.profiles.small] cpu_millicores = 1000 memory_bytes = 2147483648 [workers.profiles.large] cpu_millicores = 8000 memory_bytes = 17179869184 ``` Each `workers.provisioners.` object starts with `provider`. Its other fields are the settings listed below for that provider. For example, use `workers.provisioners.cluster.namespace` for a Kubernetes namespace. Several names can use the same provider type. The console shows Provisioner and Compute profile in the workload's Compute section and in each task editor. Each setting uses the task value first, then the workload value, then the instance default. Leaving a selector unset keeps inheritance. For the API, set `options.provisioner` and `options.profile` on a workload. Set `provisioner` and `profile` on an individual task to override them. `GET /api/v1/execution-options` lists available names and resource limits. In the combined console host, the path starts with `/api/v1/unified`. A profile and custom compute requirements cannot be set at the same level. A task's custom resources replace an inherited profile. A task's profile replaces inherited custom resources. If no profile or custom value is selected, `submission_defaults.compute_requirements` supplies the resource values. The controller saves each task's resolved settings when its run starts. Retries and worker reuse keep those values. Configuration edits require a controller restart and affect new runs. Keep a provisioner definition unchanged while it owns active allocations. Changing the default does not change their owner. Removing or changing their owner blocks recovery and cleanup until the original definition is restored. All external workers must reach the shared controller endpoints and the configured result and table storage. Local-process workers share local paths. Docker, Kubernetes, and Lambda deployments need storage that their workers can reach. In-process capacity and Lambda capacity are checked before launch. Docker and Kubernetes enforce the requested container resources. Local-process and in-process resources are execution budgets rather than operating-system limits. Older single-provider files remain readable. Saving them writes the named format. ## Deployment compute pools A deployment profile owns a pool of workers. It can keep a fixed count or grow with queued work within autoscaling limits. The controller maintains every configured pool. Ready workers can serve tasks from different workloads. Each worker runs one task at a time. Tasks wait when their selected pool is busy. ```toml theme={"theme":{"light":"github-light","dark":"github-dark"}} [workers.deployment] default_profile = "standard" [workers.deployment.profiles.standard] provisioner = "cluster" sizing = { mode = "fixed", workers = 2 } [workers.deployment.profiles.standard.resources] cpu_millicores = 1000 memory_bytes = 2147483648 [workers.deployment.profiles.large] provisioner = "cluster" sizing = { mode = "fixed", workers = 1 } [workers.deployment.profiles.large.resources] cpu_millicores = 2000 memory_bytes = 4294967296 [submission_defaults] worker_scope = "Deployment" ``` This starts two standard workers and one large worker. Resources apply to each worker. The provisioner names refer to `workers.provisioners`. Different profiles can use different provider types. Local process, in-process, Docker, and Kubernetes support all three modes. Lambda supports Task compute only. Deployment pools use `submission_defaults.execution_target`. A workload that selects deployment compute must use that target. The default deployment profile is required when pools are configured. It selects a pool when `deployment_profile` is omitted. `submission_defaults.worker_scope` sets the default compute mode for new work. Existing configurations keep Task mode unless this setting changes. Workload and task settings support three modes: | `worker_scope` | Ownership and reuse | | - | - | | `Deployment` | Share a deployment pool across workloads and runs. | | `Workload` | Reuse workers within one workload run. | | `Task` | Create compute for each task attempt. | Use `options.worker_scope` and `options.deployment_profile` for a workload. Use `worker_scope` and `deployment_profile` on a task to override that choice. Unset task values inherit the workload selection. Switching between Deployment and dedicated compute drops settings inherited from the previous mode. Workload and Task modes share provisioner and resource settings. Deployment mode accepts only a deployment profile. Its provisioner comes from configuration. Its resource sizes come from the deployment profile. Set `provisioner`, `profile`, or `compute_requirements` only for Workload or Task mode. `parallelism` still limits active tasks in a workload; it does not resize deployment pools. The controller saves each task's resolved choice with its run. Retries keep that choice. Completing or cancelling a task releases its worker back to the pool after cleanup. It does not delete deployment compute. ### Change deployment compute at runtime On **Workers**, choose **Add profile** to create a deployment profile. Enter a unique name, choose an existing provisioner, and set CPU, memory, and capacity. Names accept up to 128 letters, numbers, underscores, and hyphens. The new profile appears in Workbench and workload execution settings without a restart. Provisioner definitions, such as a Kubernetes namespace or worker image, still come from the startup file. On **Workers**, open a profile's action menu and choose **Edit compute**. Change its fixed worker count or autoscaling minimum, maximum, and idle timeout. Set CPU in cores and memory in GiB for each worker. A minimum of zero allows an autoscaling pool to become empty when idle. Applying changes does not restart the controller. The controller saves them in its database. They survive controller restarts and override that profile's sizing and resources from the startup file. The provisioner remains unchanged. Increasing the worker count adds capacity to the current pool. Reducing the fixed count or autoscaling maximum retires idle workers first. Surplus busy workers finish their current task and receive no new tasks. Normal autoscaling reductions respect the idle timeout. CPU and memory changes apply to new runs. Accepted runs, including their queued tasks and repairs, keep their saved resources. Old and new worker sizes can overlap until accepted runs finish. Old pools keep capacity for assigned and runnable queued tasks. They have no warm minimum and release surplus idle workers after the current profile's idle timeout. Old fixed pools release surplus idle workers after one second. The capacity table shows how many workers still use previous resources. A larger worker can remain pending if its provisioner lacks capacity; startup failures remain visible in the table. Create a profile with `POST /api/v1/workers/profiles`: ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "name": "analytics", "provisioner": "cluster", "sizing": { "mode": "autoscaling", "min_workers": 0, "max_workers": 8, "idle_timeout_seconds": 30 }, "resources": { "cpu_millicores": 2000, "memory_bytes": 4294967296 } } ``` Use a provisioner name from `GET /api/v1/execution-options`. Creation returns HTTP 201 and revision 1. Duplicate names return HTTP 409. The controller checks worker scope support and resource limits before saving. New profiles are stored in the controller database and survive restarts. `GET /api/v1/workers/capacity` returns each profile's current revision, sizing, resources, and actual worker counts. Update any existing profile with `PUT /api/v1/workers/profiles/{name}`: ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "expected_revision": 0, "sizing": { "mode": "fixed", "workers": 3 }, "resources": { "cpu_millicores": 2000, "memory_bytes": 4294967296 } } ``` Use the revision from the latest capacity response. Concurrent edits return HTTP 409. Reload the profile before applying another change. In the combined host, these paths start with `/api/v1/unified`. ## Shared runtime Use `runtime.duckdb` for Native Engine settings, `runtime.executor` for execution I/O limits, and `runtime.connection` for worker-control limits. The controller uses these values for in-process workers and sends them to external workers. `batch_size` defaults to `100`. ## Worker startup Users maintain one controller file. The provisioner generates standard worker arguments and supplies two JSON environment values for process, Docker, and Kubernetes workers: * `LAMINAR_WORKER_CONFIG_JSON`: only the resolved settings needed by the worker. * `LAMINAR_ALLOCATION_JSON`: allocation identity, task and attempt identity, resources, controller endpoints, and registration token. Each value is limited to 64 KiB. Invalid input is rejected before execution. Kubernetes uses container environment values. Docker sets container environment values at creation. Local processes receive the same values in their environment. Allocation bootstrap does not create a Kubernetes Secret or mount a file. In-process workers receive typed values directly. Lambda workers receive the generated configuration and allocation in their invocation event. ### Local processes Set `workers.provisioners..executable` to override the running executable, `working_directory` to select its working directory, and `environment` for additional environment values. Standard worker arguments are generated automatically. ### Docker Set `docker.image` and an absolute `docker.executable` path. `docker.environment` and `docker.working_directory` apply to the provider command. Worker configuration and allocation are passed automatically. ### Kubernetes Set `kubernetes.namespace`, `kubernetes.image`, and optional `kubernetes.service_account`. An optional native `kubernetes.pod_template` supports placement, scratch volumes, identity, and other Pod settings. The provider supplies allocation identity, environment input, CPU and memory, and standard worker arguments. ### Lambda Set `lambda` fields for the published function, version, region, capacity, and invocation limits. The invocation carries worker startup data. It does not change the function environment for each allocation. ## In-process workers ### `in_process.capacity.tuning` Holds optional tuning fields in the capacity object. The current in-process host checks only its CPU and memory capacity. * **Default:** Unset * **Type:** object; optional > **Example** > > Set task execution budgets under submission\_defaults.compute\_requirements.tuning instead. Adding tuning here does not make the host apply those settings to every task. ### `in_process.capacity.tuning.threads` Chooses how many execution threads one task may use within its assigned CPU allocation. In this capacity object, it does not set the tuning used by tasks. * **Default:** Unset * **Type:** integer; positive; optional > **Example** > > Set threads under submission\_defaults.compute\_requirements.tuning to change the default task setting. For a task allocated 2000 millicores, `2` allows two execution threads. Leave it unset to derive a value from the allocation; setting more threads does not allocate more CPUs. ### `in_process.capacity.tuning.query_memory_bytes` Sets how much of a worker’s allocated memory may be used by the query engine. In this capacity object, it does not set the tuning used by tasks. * **Default:** Unset * **Type:** integer (bytes); positive; optional > **Example** > > Set query\_memory\_bytes under submission\_defaults.compute\_requirements.tuning to change the default task setting. On a 2 GiB worker, `1073741824` reserves a 1 GiB query budget. Leave room for connector buffers and process overhead; this budget cannot exceed 80% of the allocation. ### `in_process.capacity.tuning.buffer_bytes` Sets the memory budget used for task data buffers. In this capacity object, it does not set the tuning used by tasks. * **Default:** Unset * **Type:** integer (bytes); positive; optional > **Example** > > Set buffer\_bytes under submission\_defaults.compute\_requirements.tuning to change the default task setting. `67108864` gives a 64 MiB buffer budget. It must fit alongside query memory and the configured number of output buffers. ### `in_process.capacity.tuning.batch_rows` Sets how many rows may be handled in one reader batch. In this capacity object, it does not set the tuning used by tasks. * **Default:** Unset * **Type:** integer; positive; optional > **Example** > > Set batch\_rows under submission\_defaults.compute\_requirements.tuning to change the default task setting. `65536` reads up to 65,536 rows per batch when this setting is omitted. Smaller batches can reduce memory held at once; the maximum is 1048576. ### `in_process.capacity.tuning.write_batch_rows` Sets the maximum rows per sink batch independently of reader batches. The executor also limits batching by its byte budget. One oversized input value may need a larger buffer. * **Default:** Unset; resolves to 65536 * **Type:** integer; positive; at most 1048576; optional ### `in_process.capacity.tuning.write_parallelism` Sets the number of worker-local Iceberg file writers. Writers use bounded queues and share a memory budget. All writers must finish before one Iceberg commit publishes their files. * **Default:** Unset; resolves to execution threads * **Type:** integer; positive; must not exceed execution threads; optional > **Example** > > On a four-core worker, `write_parallelism: 4` allows four file writers. Each writer can encode and upload while the other writers continue. A one-core worker defaults to one writer. ### `in_process.capacity.tuning.spill_bytes` Limits task data temporarily written to disk when execution needs it. In this capacity object, it does not set the tuning used by tasks. * **Default:** Unset * **Type:** integer (bytes); optional > **Example** > > Set spill\_bytes under submission\_defaults.compute\_requirements.tuning to change the default task setting. `8589934592` gives an 8 GiB task spill budget. Zero disables task spill, while an omitted value is calculated from allocation memory. ### `in_process.capacity.tuning.output_buffer_batches` Limits how many output batches wait for the next consumer. In this capacity object, it does not set the tuning used by tasks. * **Default:** Unset * **Type:** integer; positive; optional > **Example** > > Set output\_buffer\_batches under submission\_defaults.compute\_requirements.tuning to change the default task setting. `2` allows two batches to wait. Raising this can keep more data in memory; task tuning allows at most 32. ### `in_process.capacity.cpu_millicores` Sets the largest CPU request that an in-process worker allocation may ask for. * **Default:** `2000` * **Type:** integer; positive > **Example** > > `2000` admits an allocation requesting up to two CPUs. A request for 4000 millicores is rejected as unavailable. This does not reserve CPU for the controller process. ### `in_process.capacity.memory_bytes` Sets the largest memory request that an in-process worker allocation may ask for. * **Default:** `2147483648` * **Type:** integer (bytes); positive > **Example** > > `2147483648` admits an allocation requesting up to 2 GiB. It does not set an operating-system memory limit or resize the controller container. ## Local process workers ### `local_process.executable` Chooses the Laminar program started for each separate local worker. * **Default:** Set when used * **Type:** string > **Example** > > `"/usr/local/bin/laminar"` starts that installed binary. Supply its worker config and role through process.arguments. ### `environment` Supplies the environment passed to the worker launch command. The command does not inherit the host environment. Values are strings. Sensitive values are hidden in Settings. * **Default:** Set when used * **Type:** map > **Example** > > Include the variables required by the launch program. An empty map passes no extra variables. For example, `environment = { APP_MODE = "demo" }` sets APP\_MODE to demo. ### `working_directory` Chooses the local folder from which the launch command runs. * **Default:** Set when used * **Type:** string > **Example** > > `"/var/lib/laminar"` makes relative paths used by the command resolve from that folder. ## Docker workers ### `docker.executable` Chooses the Docker command used by the provisioner to manage worker containers. * **Default:** Set when used * **Type:** string > **Example** > > `"/usr/local/bin/docker"` selects Docker on the provisioner host. The host must also be able to reach its Docker daemon. ### `docker.image` Chooses the container image used when starting a separate worker. * **Default:** Set when used * **Type:** string > **Example** > > Pin the same Laminar release used by the controller, so the worker understands the assignment format sent to it. ### `docker.environment` Supplies the environment passed to the worker launch command. The command does not inherit the host environment. Values are strings. Sensitive values are hidden in Settings. * **Default:** Set when used * **Type:** map > **Example** > > Include the variables required by the launch program. An empty map passes no extra variables. For example, `environment = { APP_MODE = "demo" }` sets APP\_MODE to demo. ### `docker.working_directory` Chooses the local folder from which the launch command runs. * **Default:** Set when used * **Type:** string > **Example** > > `"/var/lib/laminar"` makes relative paths used by the command resolve from that folder. ## Kubernetes workers ### `kubernetes.namespace` Chooses the Kubernetes namespace in which Unified creates its worker pods. * **Default:** Set when used * **Type:** string > **Example** > > `"analytics"` places Unified worker pods and the current provider’s bootstrap resources in namespace `analytics`. The provisioner needs permission in that namespace. ### `kubernetes.pod_template` Describes the pod that Unified creates for a worker. Sensitive values are hidden in Settings. * **Default:** Set when used * **Type:** json > **Example** > > Include one container named `worker` and its image. The provider supplies worker arguments, allocation identity, runtime environment values, CPU, and memory. Native Pod settings remain available. No allocation Secret or bootstrap mount is created. ## Lambda workers ### `lambda.function_name` Chooses the deployed Lambda function used to execute a task. * **Default:** Set when used * **Type:** string > **Example** > > Set it to the function name or ARN you deployed for Unified workers. This setting invokes an existing function; it does not create that function. ### `lambda.qualifier` Pins the Lambda function version used for worker invocations. * **Default:** Unset * **Type:** string; optional > **Example** > > `"7"` selects published version 7. Omit it only when the function identifier already includes a published numeric version. ### `lambda.region` Chooses the AWS Region used by this connection. * **Default:** Set when used * **Type:** string > **Example** > > `"us-east-1"` directs requests to that Region. Use the Region containing your bucket or Lambda function. ### `lambda.aws_executable` Tells the provisioner where to find the AWS CLI used for Lambda invocation. * **Default:** Set when used * **Type:** string > **Example** > > `"/usr/local/bin/aws"` selects that executable on the provisioner host. Installing it inside an unrelated worker pod does not make it available to the provisioner. ### `lambda.memory_mib` Records the memory size configured on the deployed Lambda worker function. * **Default:** Set when used * **Type:** integer (MiB) > **Example** > > `2048` means the deployed function has 2048 MiB of memory. This field must match the function; it does not resize Lambda itself. ### `lambda.timeout_seconds` Records the execution timeout configured on the deployed Lambda worker function. * **Default:** Set when used * **Type:** integer (seconds) > **Example** > > `300` describes a function configured with a five-minute timeout. It does not change the function’s AWS configuration. ### `lambda.maximum_event_age_seconds` Records how long AWS may retain an asynchronous worker invocation before discarding it. * **Default:** Set when used * **Type:** integer (seconds) > **Example** > > `3600` describes a one-hour maximum event age. Match this to the deployed Lambda asynchronous invocation settings. ### Stop, resume, and delete The profile menu on **Workers** includes **Stop profile**, **Resume profile**, and **Delete profile**. Stop blocks new admissions and assignments. Queued tasks wait. Running tasks finish before their workers are released. Resume restores the saved capacity. A fixed profile can also have zero workers without losing its settings. Delete removes the profile from compute choices and releases idle workers. The configured default profile cannot be deleted. Finish or cancel unfinished runs, and change or delete saved workloads that use the profile first. Deletion keeps worker and task history. It also survives controller restarts, including profiles that were defined in the startup file. A deployment worker's menu includes **Drain and remove** and **Replace worker**. Both let the current task finish before release. Remove lowers fixed capacity by one, or lowers the autoscaling minimum and maximum by one. Removing the last autoscaling worker stops the profile. Replace keeps capacity unchanged. The controller starts replacement capacity when the profile is active. These changes use revision checks. If another update wins first, reload the profile before trying again. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.