Move a local secret to an external provider

How-to guideApplies to: Unified engine deployments

Keep a local name that consumers already use, but fetch its value from AWS Parameter Store instead of the encrypted local value.

Prepare the destination

Configure recipe_ssm and create the raw client-secret parameter as described in Use AWS Parameter Store. Start with the same valid client-secret value in both places so this exercise changes the source without also changing the remote client identity. Use an existing Polaris catalog with a retail warehouse and the SF1 item table in namespace tpcds_sf1. Replace the example URLs, client ID, warehouse, namespace, and Region with your values. The Polaris client needs access to the warehouse and table. These examples request PRINCIPAL_ROLE:ALL; use the scope granted to your client. For the final SELECT, Polaris must supply usable temporary S3 credentials for the table. Otherwise attach a separate storage credential as shown in Separate catalog and storage credentials. A successful login or table listing alone does not prove S3 access.

Create a consumer of the local value

CREATE SECRET recipe_moving_secret AS '<client-secret>';

CREATE CREDENTIAL recipe_moving_oauth USING oauth2
OPTIONS (
  'authentication' = 'client_credentials',
  'client_id' = '<client-id>',
  'token_endpoint' = 'https://polaris.example.com/api/catalog/v1/oauth/tokens'
)
SECRETS ('client_secret' = 'recipe_moving_secret');

CREATE CATALOG recipe_moving_retail USING `iceberg-rest`
OPTIONS (
  'catalog.type' = 'polaris',
  'uri' = 'https://polaris.example.com/api/catalog',
  'warehouse' = 'retail',
  'auth.type' = 'oauth2',
  'oauth2-server-uri' = 'https://polaris.example.com/api/catalog/v1/oauth/tokens',
  'scope' = 'PRINCIPAL_ROLE:ALL',
  's3.region' = 'us-east-2',
  's3.endpoint' = 'https://s3.us-east-2.amazonaws.com',
  's3.allow-http' = 'false'
)
CREDENTIALS (
  'auth' = 'recipe_moving_oauth'
);

SELECT i_item_sk, i_item_id
FROM recipe_moving_retail.tpcds_sf1.item
ORDER BY i_item_sk
LIMIT 5;

Change where the name resolves

ALTER SECRET recipe_moving_secret
USING 'recipe_ssm:///laminar/recipes/polaris-client-secret';

DESCRIBE SECRET recipe_moving_secret;

SELECT i_item_sk, i_item_id
FROM recipe_moving_retail.tpcds_sf1.item
ORDER BY i_item_sk
LIMIT 5;

DESCRIBE should now show an external reference. The credential still names recipe_moving_secret. A fresh query must be able to resolve that reference when credentials are next acquired. Already running work can retain access acquired before the change. When finished:

DROP CATALOG recipe_moving_retail;
DROP CREDENTIAL recipe_moving_oauth;
DROP SECRET recipe_moving_secret;

The parameter remains in AWS.

Transformation Engine

Learn Laminar

Workloads

SQL reference

HTTP API

Recipes

Operations

Move a local secret to an external provider

Prepare the destination

Create a consumer of the local value

Change where the name resolves

​Prepare the destination

​Create a consumer of the local value

​Change where the name resolves

Prepare the destination

Create a consumer of the local value

Change where the name resolves