Optimize manifests

ReferenceApplies to: Unified engine deployments

An Iceberg manifest lists data files or delete files and their metadata. Manifest optimization rewrites that metadata layout. Use it when file metadata is fragmented across many manifests, rather than when the data files themselves need compaction.

Before you start

Choose a writable Iceberg catalog and an unused practice namespace in Workbench. Native manifest optimization currently requires an existing format-v2 table. This example replaces only maintenance_docs.manifest_demo and uses inline rows.

Create writes, optimize, and verify

SET spark.sql.caseSensitive = true;
CREATE NAMESPACE IF NOT EXISTS maintenance_docs;
DROP TABLE IF EXISTS maintenance_docs.manifest_demo;
CREATE TABLE maintenance_docs.manifest_demo (
  id BIGINT,
  region STRING,
  amount BIGINT
) USING iceberg
TBLPROPERTIES ('format-version' = '2');

INSERT INTO maintenance_docs.manifest_demo
VALUES (1, 'North', 10), (2, 'South', 20);
INSERT INTO maintenance_docs.manifest_demo
VALUES (3, 'North', 30), (4, 'South', 40);
INSERT INTO maintenance_docs.manifest_demo
VALUES (5, 'North', 50), (6, 'South', 60);

SELECT COUNT(*) AS row_count, SUM(amount) AS total_amount
FROM maintenance_docs.manifest_demo;

CALL system.optimize_metadata(
  table => 'maintenance_docs.manifest_demo',
  target_metadata_size_bytes => 1048576
);

SELECT COUNT(*) AS row_count, SUM(amount) AS total_amount
FROM maintenance_docs.manifest_demo;

The aggregate before and after the call should return six rows and total amount 210. The 1 MiB metadata target is a small practice value. It does not set the size of Parquet data files.

Read the result

The CALL result has two counters:

Result column	Meaning
`rewritten_manifests_count`	Manifests replaced by this operation
`added_manifests_count`	Replacement manifests added

A small table may have little to reorganize. Zero rewritten manifests can be a successful result. The number of data files and visible rows should not change because of a manifest-only rewrite. Record the snapshot and history before the call, wait for successful publication, then refresh the table inspection. Recheck the same logical rows. On an active table, separate changes from concurrent writers from the effect of this operation.

Use the familiar alias

The same metadata action is available as rewrite_manifests. This example reuses the table created above and uses the instance’s metadata-size default.

CALL system.rewrite_manifests(
  table => 'maintenance_docs.manifest_demo'
);

The two names select the same action. You do not need to run both to maintain a table. Use one call per intended operation.

Know the boundary

Manifest optimization does not merge small Parquet data files, remove row-delete records, expire old snapshots, delete orphan files, or compute query statistics. Choose data compaction or delete compaction when the underlying file layout is the issue.

Clean up

DROP TABLE IF EXISTS maintenance_docs.manifest_demo;

See the procedure reference for argument names and defaults.

Transformation Engine

Learn Laminar

Workloads

SQL reference

HTTP API

Recipes

Operations

Before you start

Create writes, optimize, and verify

Read the result

Use the familiar alias

Know the boundary

Clean up

​Before you start

​Create writes, optimize, and verify

​Read the result

​Use the familiar alias

​Know the boundary

​Clean up

Before you start

Create writes, optimize, and verify

Read the result

Use the familiar alias

Know the boundary

Clean up