Skip to main content
How-to guideApplies to: Unified engine deployments
Choose where task attempts execute and how much work can proceed together. A graph defines dependencies; compute settings determine whether ready tasks have capacity to run.

Choose a compute lifetime

Choose the workload default in the compute step. Tasks can inherit it or use an override. Deployment compute selects a deployment profile; its resources and provisioner come from that profile. Workload and task compute can use supported provisioners, profiles, or custom resources. The examples leave instance-specific compute choices unset. Select an available mode before saving. Profile names are deployment-specific; there is no universal standard profile name required by these docs.

Size CPU and memory

CPU is measured in millicores: 1,000 millicores is one CPU. Memory is measured in bytes. A compute profile and custom resource requirements are alternative ways to select resources. More requested resources can increase allocation waits if capacity is limited. First check whether time is spent waiting for a worker, executing SQL, reading storage, or committing output.

Control parallel work

parallelism limits concurrent task work within a run. Independent tasks also need available worker capacity. Overlap controls concurrency between runs, which is a separate decision. For the four-task example, parallelism of one makes the two summaries execute in sequence. More capacity lets both become active when their shared predecessor succeeds.

Set time and attempt limits

Retries create new attempts and can add delay. A schedule interval does not bound execution duration. Read Retry policies and Sensors for their separate clocks.

Tune query execution

Advanced options include query threads, query memory, connector buffer memory, batch rows, spill space, output buffering, file read concurrency, and input prefetch. These limits operate inside the allocated compute. Start with instance defaults. Change a setting to address observed evidence, such as a connector buffer limit or query memory pressure. A larger worker does not automatically remove a smaller explicit buffer limit.

Inspect the saved choices

Compare requested options with effective options and attempt execution details. Defaults can fill fields that the editor leaves unset. Accepted work records the settings used for execution; later profile changes should not be read as the settings of an earlier run. For profile administration and instance defaults, use Compute configuration and Submission defaults.
Last modified on November 4, 2026