Tasks and dependencies

How-to guideApplies to: Unified engine deployments

Use tasks to give each step a clear purpose. Connect them when one step needs another step’s successful completion or returned rows.

Choose a task type

Type	Use it for	Output for dependent tasks
SQL	Read, transform, or publish data.	Retained query rows can be passed through a result alias.
Batch Python	Run a Python program once per workload run.	Use ordering dependencies and explicit storage for shared data.
Sensor	Wait for a named external condition.	Readiness, not a tabular result.

Give tasks stable names such as summarize_orders and publish_summary. Their names appear in dependencies, failures, and selective rerun selections.

Control execution order

In the downstream task, choose predecessors under Run after. Leave result sharing off when only order matters. A task with several predecessors waits for all of them to succeed. A failed predecessor can cause dependent tasks to be skipped. Independent tasks can become ready in parallel. The graph must not contain cycles, missing tasks, or duplicate connections. Adding a task after another task creates a dependency. Adding a parallel task gives it the same predecessors. Neither action creates condition-based execution.

Pass rows with result aliases

Enable Share upstream results and assign a Result alias to the incoming connection. Use that alias as a relation in the downstream SQL. A connection is represented as:

{
  "upstream": "orders",
  "downstream": "by_category",
  "input_name": "order_rows"
}

Set input_name to null for an ordering-only dependency. Incoming aliases must be unique within a downstream task, using its case-sensitivity rules. Aliases are scoped to the downstream task and run. They do not register permanent tables. Keep column names and types explicit: a changed upstream schema can break downstream SQL even when the graph still validates. See Build a multi-task workflow for a complete runnable graph.

Use Python tasks

Choose Batch Python for a program that runs once per workload run. A task can read the parameters dictionary:

region = parameters.get('region', 'North')
print(f'Preparing analysis for {region}')

Each attempt has its own Python kernel. Python variables do not carry across tasks. SQL result aliases are not inputs to batch Python tasks. Use a table or explicit storage when data must cross that boundary. The worker needs the configured Python runtime and any imported packages. Saving and validation do not execute Python. Keep automatic replay off for programs whose external effects are not safe to repeat; see Retry policies.

Keep writes explicit

A successful write-only task may have no rows to share. Use an ordering dependency, then read its destination table when that is the intended flow. Tasks do not form one transaction. A committed write can remain visible if a later task fails. See Publish a table result.

Edit a graph safely

Renaming a task updates dependency references, but it does not rewrite SQL aliases. Removing a task removes its connections, but downstream SQL may still use the removed input name. Review connections, validate, and run the changed workflow before scheduling it.

Transformation Engine

Learn Laminar

Workloads

SQL reference

HTTP API

Recipes

Operations

Tasks and dependencies

Choose a task type

Control execution order

Pass rows with result aliases

Use Python tasks

Keep writes explicit

Edit a graph safely

​Choose a task type

​Control execution order

​Pass rows with result aliases

​Use Python tasks

​Keep writes explicit

​Edit a graph safely

Choose a task type

Control execution order

Pass rows with result aliases

Use Python tasks

Keep writes explicit

Edit a graph safely