Automate an analysis

TutorialApplies to: Unified engine deployments

Create a two-task SQL workload. The first task groups sample orders. The second writes those results to a private table. Verify the output, then add a schedule.

Before you start

You need SQL workloads and a writable Iceberg catalog. The catalog must allow namespace and table creation. The six input orders are included in the task SQL. Choose an unused namespace. This lesson uses learn_batch_01. Replace that name everywhere if it already exists. Record the selected catalog name. Use the same catalog in Workbench and in the workload. A task is one part of a workload. A run is one execution of the saved workload. A result alias gives a dependent task a name for rows produced upstream.

1. Prepare the output

In Workbench, select your writable catalog. Run these statements separately, in order, in the same session:

SET spark.sql.caseSensitive = true;
CREATE NAMESPACE learn_batch_01;
CREATE TABLE learn_batch_01.category_sales (
  category STRING,
  sales BIGINT
) USING iceberg;

Expect a new empty table with two columns. If namespace creation reports that the name is already in use, choose a new name before continuing. Do not reuse an unknown table.

2. Create the source task

Open Workloads and create Lesson - category summary. In Details, use the selected catalog, namespace learn_batch_01, and case-sensitive names. In Tasks, add a SQL task named summarize. Use this complete query:

WITH orders(order_id, category, region, amount) AS (
  VALUES
    (1, 'Books',  'North', 20),
    (2, 'Games',  'South', 30),
    (3, 'Games',  'North', 15),
    (4, 'Books',  'North', 40),
    (5, 'Garden', 'South', 25),
    (6, 'Garden', 'North', 10)
)
SELECT category, SUM(amount) AS sales
FROM orders
GROUP BY category
ORDER BY category;

This task has no dependencies. Its result should contain Books 60, Games 45, and Garden 35. The SQL is copied into the workload; it is not a live link to a saved query.

3. Add the publication task

Add a second SQL task named publish. Under Run after, select summarize. Set the dependency’s Result alias to category_totals.

INSERT OVERWRITE TABLE learn_batch_01.category_sales
SELECT category, sales FROM category_totals;

category_totals exists as an upstream result input in this workload. This statement cannot run by itself in Workbench. In Compute, choose an available execution target. In Schedule, keep the workload Manual. In Review, validate the definition and create it.

4. Run and verify publication

Select Start run.
Open the run and confirm that summarize finishes before publish.
Inspect the publication task’s final state and confirmed write details.
Return to Workbench with the same catalog and run:

SELECT category, sales
FROM learn_batch_01.category_sales
ORDER BY category;

category	sales
Books	60
Games	45
Garden	35

Start the workload once more and repeat the query. Expect the same three rows, not six. Overwrite replaces the table contents. A workload is not one transaction across all its tasks.

5. Schedule the workload

Edit the workload and open Schedule. Choose Fixed interval, such as five minutes, if suitable for your instance. Choose Wait for overlapping runs so two runs do not write this output at once. Review the time zone, timeout, and missed-run policy. Save through Review. Check the next due time. After a scheduled run completes, inspect its trigger and the output table. Do not use the existence of a schedule as proof that a run finished.

Change one thing

Pause recurring execution. Add WHERE region = 'North' between FROM and GROUP BY in the summarize task. Save the revised workload and start a manual run. Expect Books = 60, Games = 15, and Garden = 10 in the output table. Its total is now 85. Inspect the run’s saved revision to confirm which SQL produced it.

Check the result

You have verified task order, a confirmed table write, repeat-run overwrite behavior, a scheduled run, and a changed definition. Query success in summarize alone does not prove that publish committed.

Clean up

Pause the schedule and wait for active runs to finish. Keep or delete the lesson workload. After removing its consumers, drop only the table and namespace created here:

DROP TABLE learn_batch_01.category_sales;
DROP NAMESPACE learn_batch_01;

Use the same catalog. The namespace drop should succeed only when it is empty.

Continue

Process new data to track a changing source table. See task dependencies and schedules for reference.

Transformation Engine

Learn Laminar

Workloads

SQL reference

HTTP API

Recipes

Operations

Before you start

1. Prepare the output

2. Create the source task

3. Add the publication task

4. Run and verify publication

5. Schedule the workload

Change one thing

Check the result

Clean up

Continue

​Before you start

​1. Prepare the output

​2. Create the source task

​3. Add the publication task

​4. Run and verify publication

​5. Schedule the workload

​Change one thing

​Check the result

​Clean up

​Continue

Before you start

1. Prepare the output

2. Create the source task

3. Add the publication task

4. Run and verify publication

5. Schedule the workload

Change one thing

Check the result

Clean up

Continue