Workers - e6data Docs

ReferenceApplies to: Unified engine deployments

[workers]

Named provisioners and profiles

Define provisioners and profiles in the controller file. A provisioner chooses where a worker runs. A profile names its CPU, memory, and execution tuning.

[workers]
default_provisioner = "local"
default_profile = "small"

[workers.provisioners.local]
provider = "local_process"
working_directory = "/var/lib/laminar"

[workers.provisioners.cluster]
provider = "kubernetes"
namespace = "analytics"
image = "laminar:your-version"

[workers.profiles.small]
cpu_millicores = 1000
memory_bytes = 2147483648

[workers.profiles.large]
cpu_millicores = 8000
memory_bytes = 17179869184

Each workers.provisioners.<name> object starts with provider. Its other fields are the settings listed below for that provider. For example, use workers.provisioners.cluster.namespace for a Kubernetes namespace. Several names can use the same provider type. The console shows Provisioner and Compute profile in the workload’s Compute section and in each task editor. Each setting uses the task value first, then the workload value, then the instance default. Leaving a selector unset keeps inheritance. For the API, set options.provisioner and options.profile on a workload. Set provisioner and profile on an individual task to override them. GET /api/v1/execution-options lists available names and resource limits. In the combined console host, the path starts with /api/v1/unified. A profile and custom compute requirements cannot be set at the same level. A task’s custom resources replace an inherited profile. A task’s profile replaces inherited custom resources. If no profile or custom value is selected, submission_defaults.compute_requirements supplies the resource values. The controller saves each task’s resolved settings when its run starts. Retries and worker reuse keep those values. Configuration edits require a controller restart and affect new runs. Keep a provisioner definition unchanged while it owns active allocations. Changing the default does not change their owner. Removing or changing their owner blocks recovery and cleanup until the original definition is restored. All external workers must reach the shared controller endpoints and the configured result and table storage. Local-process workers share local paths. Docker, Kubernetes, and Lambda deployments need storage that their workers can reach. In-process capacity and Lambda capacity are checked before launch. Docker and Kubernetes enforce the requested container resources. Local-process and in-process resources are execution budgets rather than operating-system limits. Older single-provider files remain readable. Saving them writes the named format.

Deployment compute pools

A deployment profile owns a pool of workers. It can keep a fixed count or grow with queued work within autoscaling limits. The controller maintains every configured pool. Ready workers can serve tasks from different workloads. Each worker runs one task at a time. Tasks wait when their selected pool is busy.

[workers.deployment]
default_profile = "standard"

[workers.deployment.profiles.standard]
provisioner = "cluster"
sizing = { mode = "fixed", workers = 2 }

[workers.deployment.profiles.standard.resources]
cpu_millicores = 1000
memory_bytes = 2147483648

[workers.deployment.profiles.large]
provisioner = "cluster"
sizing = { mode = "fixed", workers = 1 }

[workers.deployment.profiles.large.resources]
cpu_millicores = 2000
memory_bytes = 4294967296

[submission_defaults]
worker_scope = "Deployment"

This starts two standard workers and one large worker. Resources apply to each worker. The provisioner names refer to workers.provisioners. Different profiles can use different provider types. Local process, in-process, Docker, and Kubernetes support all three modes. Lambda supports Task compute only. Deployment pools use submission_defaults.execution_target. A workload that selects deployment compute must use that target. The default deployment profile is required when pools are configured. It selects a pool when deployment_profile is omitted. submission_defaults.worker_scope sets the default compute mode for new work. Existing configurations keep Task mode unless this setting changes. Workload and task settings support three modes:

`worker_scope`	Ownership and reuse
`Deployment`	Share a deployment pool across workloads and runs.
`Workload`	Reuse workers within one workload run.
`Task`	Create compute for each task attempt.

Use options.worker_scope and options.deployment_profile for a workload. Use worker_scope and deployment_profile on a task to override that choice. Unset task values inherit the workload selection. Switching between Deployment and dedicated compute drops settings inherited from the previous mode. Workload and Task modes share provisioner and resource settings. Deployment mode accepts only a deployment profile. Its provisioner comes from configuration. Its resource sizes come from the deployment profile. Set provisioner, profile, or compute_requirements only for Workload or Task mode. parallelism still limits active tasks in a workload; it does not resize deployment pools. The controller saves each task’s resolved choice with its run. Retries keep that choice. Completing or cancelling a task releases its worker back to the pool after cleanup. It does not delete deployment compute.

Change deployment compute at runtime

On Workers, choose Add profile to create a deployment profile. Enter a unique name, choose an existing provisioner, and set CPU, memory, and capacity. Names accept up to 128 letters, numbers, underscores, and hyphens. The new profile appears in Workbench and workload execution settings without a restart. Provisioner definitions, such as a Kubernetes namespace or worker image, still come from the startup file. On Workers, open a profile’s action menu and choose Edit compute. Change its fixed worker count or autoscaling minimum, maximum, and idle timeout. Set CPU in cores and memory in GiB for each worker. A minimum of zero allows an autoscaling pool to become empty when idle. Applying changes does not restart the controller. The controller saves them in its database. They survive controller restarts and override that profile’s sizing and resources from the startup file. The provisioner remains unchanged. Increasing the worker count adds capacity to the current pool. Reducing the fixed count or autoscaling maximum retires idle workers first. Surplus busy workers finish their current task and receive no new tasks. Normal autoscaling reductions respect the idle timeout. CPU and memory changes apply to new runs. Accepted runs, including their queued tasks and repairs, keep their saved resources. Old and new worker sizes can overlap until accepted runs finish. Old pools keep capacity for assigned and runnable queued tasks. They have no warm minimum and release surplus idle workers after the current profile’s idle timeout. Old fixed pools release surplus idle workers after one second. The capacity table shows how many workers still use previous resources. A larger worker can remain pending if its provisioner lacks capacity; startup failures remain visible in the table. Create a profile with POST /api/v1/workers/profiles:

{
  "name": "analytics",
  "provisioner": "cluster",
  "sizing": {
    "mode": "autoscaling",
    "min_workers": 0,
    "max_workers": 8,
    "idle_timeout_seconds": 30
  },
  "resources": { "cpu_millicores": 2000, "memory_bytes": 4294967296 }
}

Use a provisioner name from GET /api/v1/execution-options. Creation returns HTTP 201 and revision 1. Duplicate names return HTTP 409. The controller checks worker scope support and resource limits before saving. New profiles are stored in the controller database and survive restarts. GET /api/v1/workers/capacity returns each profile’s current revision, sizing, resources, and actual worker counts. Update any existing profile with PUT /api/v1/workers/profiles/{name}:

{
  "expected_revision": 0,
  "sizing": { "mode": "fixed", "workers": 3 },
  "resources": { "cpu_millicores": 2000, "memory_bytes": 4294967296 }
}

Use the revision from the latest capacity response. Concurrent edits return HTTP 409. Reload the profile before applying another change. In the combined host, these paths start with /api/v1/unified.

Shared runtime

Use runtime.duckdb for Native Engine settings, runtime.executor for execution I/O limits, and runtime.connection for worker-control limits. The controller uses these values for in-process workers and sends them to external workers. batch_size defaults to 100.

Worker startup

Users maintain one controller file. The provisioner generates standard worker arguments and supplies two JSON environment values for process, Docker, and Kubernetes workers:

LAMINAR_WORKER_CONFIG_JSON: only the resolved settings needed by the worker.
LAMINAR_ALLOCATION_JSON: allocation identity, task and attempt identity, resources, controller endpoints, and registration token.

Each value is limited to 64 KiB. Invalid input is rejected before execution. Kubernetes uses container environment values. Docker sets container environment values at creation. Local processes receive the same values in their environment. Allocation bootstrap does not create a Kubernetes Secret or mount a file. In-process workers receive typed values directly. Lambda workers receive the generated configuration and allocation in their invocation event.

Local processes

Set workers.provisioners.<name>.executable to override the running executable, working_directory to select its working directory, and environment for additional environment values. Standard worker arguments are generated automatically.

Docker

Set docker.image and an absolute docker.executable path. docker.environment and docker.working_directory apply to the provider command. Worker configuration and allocation are passed automatically.

Kubernetes

Set kubernetes.namespace, kubernetes.image, and optional kubernetes.service_account. An optional native kubernetes.pod_template supports placement, scratch volumes, identity, and other Pod settings. The provider supplies allocation identity, environment input, CPU and memory, and standard worker arguments.

Lambda

Set lambda fields for the published function, version, region, capacity, and invocation limits. The invocation carries worker startup data. It does not change the function environment for each allocation.

In-process workers

`in_process.capacity.tuning`

Holds optional tuning fields in the capacity object. The current in-process host checks only its CPU and memory capacity.

Default: Unset
Type: object; optional

Example Set task execution budgets under submission_defaults.compute_requirements.tuning instead. Adding tuning here does not make the host apply those settings to every task.

`in_process.capacity.tuning.threads`

Chooses how many execution threads one task may use within its assigned CPU allocation. In this capacity object, it does not set the tuning used by tasks.

Default: Unset
Type: integer; positive; optional

Example Set threads under submission_defaults.compute_requirements.tuning to change the default task setting. For a task allocated 2000 millicores, 2 allows two execution threads. Leave it unset to derive a value from the allocation; setting more threads does not allocate more CPUs.

`in_process.capacity.tuning.query_memory_bytes`

Sets how much of a worker’s allocated memory may be used by the query engine. In this capacity object, it does not set the tuning used by tasks.

Default: Unset
Type: integer (bytes); positive; optional

Example Set query_memory_bytes under submission_defaults.compute_requirements.tuning to change the default task setting. On a 2 GiB worker, 1073741824 reserves a 1 GiB query budget. Leave room for connector buffers and process overhead; this budget cannot exceed 80% of the allocation.

`in_process.capacity.tuning.buffer_bytes`

Sets the memory budget used for task data buffers. In this capacity object, it does not set the tuning used by tasks.

Default: Unset
Type: integer (bytes); positive; optional

Example Set buffer_bytes under submission_defaults.compute_requirements.tuning to change the default task setting. 67108864 gives a 64 MiB buffer budget. It must fit alongside query memory and the configured number of output buffers.

`in_process.capacity.tuning.batch_rows`

Sets how many rows may be handled in one reader batch. In this capacity object, it does not set the tuning used by tasks.

Default: Unset
Type: integer; positive; optional

Example Set batch_rows under submission_defaults.compute_requirements.tuning to change the default task setting. 65536 reads up to 65,536 rows per batch when this setting is omitted. Smaller batches can reduce memory held at once; the maximum is 1048576.

`in_process.capacity.tuning.write_batch_rows`

Sets the maximum rows per sink batch independently of reader batches. The executor also limits batching by its byte budget. One oversized input value may need a larger buffer.

Default: Unset; resolves to 65536
Type: integer; positive; at most 1048576; optional

`in_process.capacity.tuning.write_parallelism`

Sets the number of worker-local Iceberg file writers. Writers use bounded queues and share a memory budget. All writers must finish before one Iceberg commit publishes their files.

Default: Unset; resolves to execution threads
Type: integer; positive; must not exceed execution threads; optional

Example On a four-core worker, write_parallelism: 4 allows four file writers. Each writer can encode and upload while the other writers continue. A one-core worker defaults to one writer.

`in_process.capacity.tuning.spill_bytes`

Limits task data temporarily written to disk when execution needs it. In this capacity object, it does not set the tuning used by tasks.

Default: Unset
Type: integer (bytes); optional

Example Set spill_bytes under submission_defaults.compute_requirements.tuning to change the default task setting. 8589934592 gives an 8 GiB task spill budget. Zero disables task spill, while an omitted value is calculated from allocation memory.

`in_process.capacity.tuning.output_buffer_batches`

Limits how many output batches wait for the next consumer. In this capacity object, it does not set the tuning used by tasks.

Default: Unset
Type: integer; positive; optional

Example Set output_buffer_batches under submission_defaults.compute_requirements.tuning to change the default task setting. 2 allows two batches to wait. Raising this can keep more data in memory; task tuning allows at most 32.

`in_process.capacity.cpu_millicores`

Sets the largest CPU request that an in-process worker allocation may ask for.

Default: 2000
Type: integer; positive

Example 2000 admits an allocation requesting up to two CPUs. A request for 4000 millicores is rejected as unavailable. This does not reserve CPU for the controller process.

`in_process.capacity.memory_bytes`

Sets the largest memory request that an in-process worker allocation may ask for.

Default: 2147483648
Type: integer (bytes); positive

Example 2147483648 admits an allocation requesting up to 2 GiB. It does not set an operating-system memory limit or resize the controller container.

Local process workers

`local_process.executable`

Chooses the Laminar program started for each separate local worker.

Default: Set when used
Type: string

Example "/usr/local/bin/laminar" starts that installed binary. Supply its worker config and role through process.arguments.

`environment`

Supplies the environment passed to the worker launch command. The command does not inherit the host environment. Values are strings. Sensitive values are hidden in Settings.

Default: Set when used
Type: map

Example Include the variables required by the launch program. An empty map passes no extra variables. For example, environment = { APP_MODE = "demo" } sets APP_MODE to demo.

`working_directory`

Chooses the local folder from which the launch command runs.

Default: Set when used
Type: string

Example "/var/lib/laminar" makes relative paths used by the command resolve from that folder.

Docker workers

`docker.executable`

Chooses the Docker command used by the provisioner to manage worker containers.

Default: Set when used
Type: string

Example "/usr/local/bin/docker" selects Docker on the provisioner host. The host must also be able to reach its Docker daemon.

`docker.image`

Chooses the container image used when starting a separate worker.

Default: Set when used
Type: string

Example Pin the same Laminar release used by the controller, so the worker understands the assignment format sent to it.

`docker.environment`

Supplies the environment passed to the worker launch command. The command does not inherit the host environment. Values are strings. Sensitive values are hidden in Settings.

Default: Set when used
Type: map

Example Include the variables required by the launch program. An empty map passes no extra variables. For example, environment = { APP_MODE = "demo" } sets APP_MODE to demo.

`docker.working_directory`

Chooses the local folder from which the launch command runs.

Default: Set when used
Type: string

Example "/var/lib/laminar" makes relative paths used by the command resolve from that folder.

Kubernetes workers

`kubernetes.namespace`

Chooses the Kubernetes namespace in which Unified creates its worker pods.

Default: Set when used
Type: string

Example "analytics" places Unified worker pods and the current provider’s bootstrap resources in namespace analytics. The provisioner needs permission in that namespace.

`kubernetes.pod_template`

Describes the pod that Unified creates for a worker. Sensitive values are hidden in Settings.

Default: Set when used
Type: json

Example Include one container named worker and its image. The provider supplies worker arguments, allocation identity, runtime environment values, CPU, and memory. Native Pod settings remain available. No allocation Secret or bootstrap mount is created.

Lambda workers

`lambda.function_name`

Chooses the deployed Lambda function used to execute a task.

Default: Set when used
Type: string

Example Set it to the function name or ARN you deployed for Unified workers. This setting invokes an existing function; it does not create that function.

`lambda.qualifier`

Pins the Lambda function version used for worker invocations.

Default: Unset
Type: string; optional

Example "7" selects published version 7. Omit it only when the function identifier already includes a published numeric version.

`lambda.region`

Chooses the AWS Region used by this connection.

Default: Set when used
Type: string

Example "us-east-1" directs requests to that Region. Use the Region containing your bucket or Lambda function.

`lambda.aws_executable`

Tells the provisioner where to find the AWS CLI used for Lambda invocation.

Default: Set when used
Type: string

Example "/usr/local/bin/aws" selects that executable on the provisioner host. Installing it inside an unrelated worker pod does not make it available to the provisioner.

`lambda.memory_mib`

Records the memory size configured on the deployed Lambda worker function.

Default: Set when used
Type: integer (MiB)

Example 2048 means the deployed function has 2048 MiB of memory. This field must match the function; it does not resize Lambda itself.

`lambda.timeout_seconds`

Records the execution timeout configured on the deployed Lambda worker function.

Default: Set when used
Type: integer (seconds)

Example 300 describes a function configured with a five-minute timeout. It does not change the function’s AWS configuration.

`lambda.maximum_event_age_seconds`

Records how long AWS may retain an asynchronous worker invocation before discarding it.

Default: Set when used
Type: integer (seconds)

Example 3600 describes a one-hour maximum event age. Match this to the deployed Lambda asynchronous invocation settings.

Stop, resume, and delete

The profile menu on Workers includes Stop profile, Resume profile, and Delete profile. Stop blocks new admissions and assignments. Queued tasks wait. Running tasks finish before their workers are released. Resume restores the saved capacity. A fixed profile can also have zero workers without losing its settings. Delete removes the profile from compute choices and releases idle workers. The configured default profile cannot be deleted. Finish or cancel unfinished runs, and change or delete saved workloads that use the profile first. Deletion keeps worker and task history. It also survives controller restarts, including profiles that were defined in the startup file. A deployment worker’s menu includes Drain and remove and Replace worker. Both let the current task finish before release. Remove lowers fixed capacity by one, or lowers the autoscaling minimum and maximum by one. Removing the last autoscaling worker stops the profile. Replace keeps capacity unchanged. The controller starts replacement capacity when the profile is active. These changes use revision checks. If another update wins first, reload the profile before trying again.

​Named provisioners and profiles

​Deployment compute pools

​Change deployment compute at runtime

​Shared runtime

​Worker startup

​Local processes

​Docker

​Kubernetes

​Lambda

​In-process workers

​in_process.capacity.tuning

​in_process.capacity.tuning.threads

​in_process.capacity.tuning.query_memory_bytes

​in_process.capacity.tuning.buffer_bytes

​in_process.capacity.tuning.batch_rows

​in_process.capacity.tuning.write_batch_rows

​in_process.capacity.tuning.write_parallelism

​in_process.capacity.tuning.spill_bytes

​in_process.capacity.tuning.output_buffer_batches

​in_process.capacity.cpu_millicores

​in_process.capacity.memory_bytes

​Local process workers

​local_process.executable

​environment

​working_directory

​Docker workers

​docker.executable

​docker.image

​docker.environment

​docker.working_directory

​Kubernetes workers

​kubernetes.namespace

​kubernetes.pod_template

​Lambda workers

​lambda.function_name

​lambda.qualifier

​lambda.region

​lambda.aws_executable

​lambda.memory_mib

​lambda.timeout_seconds

​lambda.maximum_event_age_seconds

​Stop, resume, and delete