> ## Documentation Index
> Fetch the complete documentation index at: https://docs.e6data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure registry, Kubernetes, and networking

> Build the Azure infrastructure (VNet, AKS, identities, ADLS storage) and install the Kubernetes platform components for an In-VPC e6data deployment on AKS.

This page builds the Azure infrastructure for an In-VPC deployment and installs the cluster-wide Kubernetes components the e6data workspace runs on. Run these steps with the Azure CLI (`az`), `kubectl`, and Helm.

Part 1 provisions Azure infrastructure (run once per cluster). Part 2 installs the Kubernetes platform components (also once per cluster). Per-workspace deployment is covered on the next page, [Deploy workspace and e6data](/platform/deploy/azure-cloudprem/deploy-workspace).

## Infrastructure requirements

Review these before you begin. They apply whether you create a new VNet (Part 1, Step 1) or reuse an existing one.

### VNet options

| Option | When to use |
| - | - |
| **New VNet** | Recommended for an isolated e6data deployment. Follow Part 1, Step 1. |
| **Existing VNet** | When e6data must share the network with other workloads. Skip Step 1 and meet the checklist below. |

### VNet and subnet sizing

With **Azure CNI overlay** (used here), pods get IPs from the **pod CIDR** (`10.244.0.0/16`), not the subnet - so the AKS subnet only needs addresses for **nodes**, not pods.

| Resource | Minimum | Recommended | Notes |
| - | - | - | - |
| **VNet CIDR** | /20 | /16 (`10.0.0.0/16`) | Must not overlap pod/service CIDRs or peered networks. |
| **AKS subnet** | /24 | /17 (`10.0.0.0/17`) | Nodes only (pods are on the overlay). |
| **Pod CIDR** | - | `10.244.0.0/16` | Virtual (overlay); must not overlap the VNet or peers. |
| **Service CIDR** | - | `172.20.0.0/16` | Cluster-internal; must not overlap the VNet or peers. |

### Availability zones

| Requirement | Minimum | Recommended |
| - | - | - |
| **Zones** | 2 | 3 (`1`, `2`, `3`) |

e6data spreads workloads across zones for high availability - the engine NodePool lists zones `1`, `2`, and `3`.

### Azure quotas (vCPU)

Quota is per VM family, per region. Check and request increases before deploying - increases can take 24–48 hours.

| Family | Used by |
| - | - |
| **Dpsv6 / Dpsv5 (ARM)** | System pool (`Standard_D2ps_v6`) and the operator pool. |
| **D / E / F / L (ARM)** | Engine nodes (Karpenter). |
| **Total Regional vCPUs** | All of the above. |

Sponsorship and free subscriptions often start with **0 quota for ARM families** - request an increase first if `az aks create` fails with a quota or SKU-not-available error.

### VM families

| Component | VM families | Architecture | Notes |
| - | - | - | - |
| **System pool** | `Standard_D2ps_v6` | arm64 | AKS system pods; use `Standard_D4as_v5` for x86. |
| **e6-operator** | `D`, `E` | arm64 | Operator and cert-manager (`e6operator` NodePool). |
| **Engine nodes** | `D`, `E`, `F`, `L` | arm64 | Query engine (Karpenter); the `L` family adds local NVMe. |

The e6data engine is optimized for **ARM64** for price-performance. Confirm your region has the chosen families available.

### Existing VNet checklist

If you reuse an existing VNet, confirm:

* [ ] No CIDR overlap between the VNet, pod CIDR (`10.244.0.0/16`), service CIDR (`172.20.0.0/16`), or any peered network.
* [ ] The AKS subnet has room for your node count (with CNI overlay, pods don't consume subnet IPs).
* [ ] Nodes have outbound internet (a NAT Gateway is recommended).
* [ ] `Microsoft.Storage` and `Microsoft.ContainerRegistry` service endpoints are enabled on the subnet.

### Existing AKS cluster checklist

If you already have an AKS cluster, **skip Part 1 Steps 1–3** and verify the cluster meets the requirements below first:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
az aks show -g <rg> -n <cluster> \
  --query "{k8s:kubernetesVersion, oidc:oidcIssuerProfile.enabled, wi:securityProfile.workloadIdentity.enabled, network:networkProfile.networkPlugin, outbound:networkProfile.outboundType}" -o yaml
```

| Requirement | Expected | If not met |
| - | - | - |
| **OIDC issuer + Workload Identity** | both `true` | `az aks update -g <rg> -n <cluster> --enable-oidc-issuer --enable-workload-identity` |
| **Kubernetes version** | 1.29+ (1.33 ideal) | Upgrade the cluster. |
| **Outbound egress** | reaches `e6labs.azurecr.io`, `mcr.microsoft.com`, `quay.io`, `app.e6.run` | Allow-list these (Azure Firewall / UDR). |
| **Deployer permissions** | User Access Administrator | Grant it - required for the role assignments. |

<Warning>
  OIDC issuer + Workload Identity is the setting that silently breaks everything if it's off. Check it first.
</Warning>

Then skip to **Step 4** (cluster identity) and continue through Parts 1 and 2 as written.

<Info>
  Steps 1 and 3 also export variables that later steps use. Run Step 0, then set these for your existing VNet and cluster before continuing:

  ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
  export VNET_ID=$(az network vnet show -g <vnet-rg> -n <vnet-name> --query id -o tsv)
  export AKS_SUBNET_ID=$(az network vnet subnet show -g <vnet-rg> --vnet-name <vnet-name> -n <subnet-name> --query id -o tsv)
  export AKS_OIDC_ISSUER=$(az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME --query "oidcIssuerProfile.issuerUrl" -o tsv)
  export NODE_RESOURCE_GROUP=$(az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME --query nodeResourceGroup -o tsv)
  ```

  Step 6C.2 also reads the VNet by `${PREFIX}-vnet`; substitute your VNet name there if it differs.
</Info>

## Part 1 - Azure infrastructure

By the end of this part you will have a VNet with a NAT Gateway, an AKS cluster with OIDC issuer and Workload Identity enabled, a system node pool and an engine node pool strategy, an ADLS Gen2 storage account, and a workspace Managed Identity federated to the e6data service accounts.

### Step 0: Set variables

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Naming and placement
export PREFIX="e6data"                  # resource name prefix (lowercase, no spaces)
export LOCATION="eastus"
export RESOURCE_GROUP="${PREFIX}-rg"
export CLUSTER_NAME="${PREFIX}-aks"

# Network
export VNET_CIDR="10.0.0.0/16"
export AKS_SUBNET_CIDR="10.0.0.0/17"     # first half of the VNet

# Cluster
export K8S_VERSION="1.33"

# Workspace (also used as the Kubernetes namespace)
export WORKSPACE_NAME="my-workspace"

# Subscription context
export SUBSCRIPTION_ID=$(az account show --query id -o tsv)
```

### Step 1: Create the resource group, VNet, and NAT Gateway

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
az group create --name $RESOURCE_GROUP --location $LOCATION

az network vnet create \
  --resource-group $RESOURCE_GROUP \
  --name ${PREFIX}-vnet \
  --address-prefix $VNET_CIDR \
  --location $LOCATION

# AKS subnet (nodes + overlay pods). Service endpoints keep Storage/ACR traffic on the Azure backbone.
az network vnet subnet create \
  --resource-group $RESOURCE_GROUP \
  --vnet-name ${PREFIX}-vnet \
  --name ${PREFIX}-aks \
  --address-prefixes $AKS_SUBNET_CIDR \
  --service-endpoints Microsoft.Storage Microsoft.ContainerRegistry

# NAT Gateway for stable egress (static public IP, 30-minute idle timeout)
az network public-ip create \
  --resource-group $RESOURCE_GROUP \
  --name ${PREFIX}-nat-pip \
  --sku Standard \
  --allocation-method Static \
  --location $LOCATION

az network nat gateway create \
  --resource-group $RESOURCE_GROUP \
  --name ${PREFIX}-nat \
  --public-ip-addresses ${PREFIX}-nat-pip \
  --idle-timeout 30 \
  --location $LOCATION

az network vnet subnet update \
  --resource-group $RESOURCE_GROUP \
  --vnet-name ${PREFIX}-vnet \
  --name ${PREFIX}-aks \
  --nat-gateway ${PREFIX}-nat

# Capture IDs for cluster creation
export AKS_SUBNET_ID=$(az network vnet subnet show \
  --resource-group $RESOURCE_GROUP --vnet-name ${PREFIX}-vnet --name ${PREFIX}-aks \
  --query id -o tsv)

export VNET_ID=$(az network vnet show \
  --resource-group $RESOURCE_GROUP --name ${PREFIX}-vnet --query id -o tsv)

echo "AKS_SUBNET_ID=$AKS_SUBNET_ID"
```

### Step 2: Create the AKS cluster

Create the cluster with OIDC issuer and Workload Identity enabled, Azure CNI overlay with Cilium, and an ARM64 system pool tainted `CriticalAddonsOnly`:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
az aks create \
  --resource-group $RESOURCE_GROUP \
  --name $CLUSTER_NAME \
  --location $LOCATION \
  --kubernetes-version $K8S_VERSION \
  --tier standard \
  --node-resource-group ${CLUSTER_NAME}-nodes \
  --enable-managed-identity \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --network-plugin azure \
  --network-plugin-mode overlay \
  --network-policy cilium \
  --network-dataplane cilium \
  --pod-cidr 10.244.0.0/16 \
  --service-cidr 172.20.0.0/16 \
  --dns-service-ip 172.20.0.10 \
  --outbound-type loadBalancer \
  --load-balancer-sku standard \
  --vnet-subnet-id $AKS_SUBNET_ID \
  --nodepool-name system \
  --node-vm-size Standard_D2ps_v6 \
  --node-count 2 \
  --enable-cluster-autoscaler \
  --min-count 2 \
  --max-count 4 \
  --node-osdisk-size 50 \
  --max-pods 110 \
  --nodepool-taints CriticalAddonsOnly=true:NoSchedule \
  --generate-ssh-keys
```

Notes:

* `Standard_D2ps_v6` is ARM64. All e6data system images are multi-arch; for x86 use `Standard_D4as_v5`.
* The NAT Gateway on the subnet takes precedence for egress even with `--outbound-type loadBalancer`.
* To use Entra ID (Azure AD) RBAC for cluster auth, append `--enable-aad --enable-azure-rbac --disable-local-accounts --aad-admin-group-object-ids <group-object-id>`.

### Step 3: Connect and capture cluster outputs

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME

# OIDC issuer URL - needed for every federated identity credential below
export AKS_OIDC_ISSUER=$(az aks show \
  --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME \
  --query "oidcIssuerProfile.issuerUrl" -o tsv)

# Node resource group (where AKS puts VMs/VMSS/LBs)
export NODE_RESOURCE_GROUP=$(az aks show \
  --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME \
  --query nodeResourceGroup -o tsv)

echo "AKS_OIDC_ISSUER=$AKS_OIDC_ISSUER"
kubectl get nodes
```

AKS hosts the OIDC issuer for you - there is nothing to create. You only reference its URL when creating federated credentials.

### Step 4: Grant the cluster identity network access

The cluster's control-plane identity needs **Network Contributor** on the VNet so the Azure cloud provider can manage load balancer backend pools for nodes in your subnet:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
CLUSTER_IDENTITY_PRINCIPAL_ID=$(az aks show \
  --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME \
  --query identity.principalId -o tsv)

az role assignment create \
  --assignee-object-id $CLUSTER_IDENTITY_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "Network Contributor" \
  --scope $VNET_ID
```

### Step 5: Container registry access

e6data images are pulled from `e6labs.azurecr.io`. Choose one option and confirm it with your onboarding engineer (they configure the matching side).

**Option A - AcrPull role** (if e6data grants your kubelet identity access, or you mirror images to your own ACR):

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
KUBELET_OBJECT_ID=$(az aks show \
  --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME \
  --query identityProfile.kubeletidentity.objectId -o tsv)

# Scope = your mirrored ACR resource ID, or the e6data-shared ACR ID from onboarding
az role assignment create \
  --assignee-object-id $KUBELET_OBJECT_ID \
  --assignee-principal-type ServicePrincipal \
  --role AcrPull \
  --scope "<acr-resource-id>"
```

**Option B - Pull secret** (e6data provides a scoped ACR token). The secret must exist in **both** the `e6operator` namespace and the workspace namespace before components install, so create the namespaces now:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
for NS in e6operator ${WORKSPACE_NAME}; do
  kubectl create namespace ${NS} --dry-run=client -o yaml | kubectl apply -f -
  kubectl create secret docker-registry e6data-acr-pull \
    --namespace ${NS} \
    --docker-server=e6labs.azurecr.io \
    --docker-username="<token-name-from-e6data>" \
    --docker-password="<token-password-from-e6data>"
done
```

With Option B, also reference `e6data-acr-pull` wherever the guide notes `imagePullSecrets` (operator values in Step 2.4, NamespaceConfig in the next page).

### Step 6: Engine node pool

Engine pods need a user node pool to scale on. Pick one provisioner. The `workspace-name` label and taint are mandatory - engine pods schedule onto nodes by exactly this label/taint pair.

**Option A - Static user node pool (simplest):**

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
az aks nodepool add \
  --resource-group $RESOURCE_GROUP \
  --cluster-name $CLUSTER_NAME \
  --name engine \
  --mode User \
  --node-vm-size Standard_D8ps_v5 \
  --enable-cluster-autoscaler \
  --min-count 0 --max-count 10 \
  --node-osdisk-size 100 \
  --zones 1 2 3 \
  --labels workspace-name=${WORKSPACE_NAME} \
  --node-taints workspace-name=${WORKSPACE_NAME}:NoSchedule
```

**Option B - Node Auto Provisioning (NAP, managed Karpenter):**

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
az aks update --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME \
  --node-provisioning-mode Auto
```

**Option C - Self-hosted Karpenter** (recommended for scale-to-zero engine nodes). This uses the AKS Karpenter provider published in Microsoft's container registry. It has five sub-steps: identity, cluster values, bootstrap token, Helm install, and the NodeClass/NodePool.

#### 6C.1 - Karpenter Managed Identity and roles

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
az identity create \
  --resource-group $RESOURCE_GROUP \
  --name ${CLUSTER_NAME}-karpenter-identity \
  --location $LOCATION

KARPENTER_PRINCIPAL_ID=$(az identity show -g $RESOURCE_GROUP \
  --name ${CLUSTER_NAME}-karpenter-identity --query principalId -o tsv)
export KARPENTER_IDENTITY_CLIENT_ID=$(az identity show -g $RESOURCE_GROUP \
  --name ${CLUSTER_NAME}-karpenter-identity --query clientId -o tsv)

# Federated credential -> kube-system/karpenter service account
az identity federated-credential create \
  --name ${CLUSTER_NAME}-karpenter-federated \
  --identity-name ${CLUSTER_NAME}-karpenter-identity \
  --resource-group $RESOURCE_GROUP \
  --issuer $AKS_OIDC_ISSUER \
  --subject "system:serviceaccount:kube-system:karpenter" \
  --audiences "api://AzureADTokenExchange"

# Role assignments
NODE_RG_ID="/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${NODE_RESOURCE_GROUP}"
RG_ID="/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}"

az role assignment create --assignee-object-id $KARPENTER_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal --role "Virtual Machine Contributor" --scope $NODE_RG_ID
az role assignment create --assignee-object-id $KARPENTER_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal --role "Network Contributor" --scope $VNET_ID
az role assignment create --assignee-object-id $KARPENTER_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal --role "Managed Identity Operator" --scope $NODE_RG_ID
az role assignment create --assignee-object-id $KARPENTER_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal --role "Reader" --scope $RG_ID
```

#### 6C.2 - Gather cluster values

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export KARPENTER_VERSION="1.7.1"
export CLUSTER_ENDPOINT="https://$(az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME --query fqdn -o tsv):443"
export NODE_IDENTITIES=$(az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME \
  --query identityProfile.kubeletidentity.resourceId -o tsv)
export VNET_GUID=$(az network vnet show -g $RESOURCE_GROUP -n ${PREFIX}-vnet --query resourceGuid -o tsv)
export SSH_PUBLIC_KEY="$(cat ~/.ssh/id_rsa.pub)"
```

#### 6C.3 - Bootstrap token

New nodes use this token to join the cluster. It must be in `<6-char-id>.<16-char-secret>` format:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
TOKEN_ID=$(LC_ALL=C tr -dc 'a-z0-9' < /dev/urandom | head -c 6)
TOKEN_SECRET=$(LC_ALL=C tr -dc 'a-z0-9' < /dev/urandom | head -c 16)
export BOOTSTRAP_TOKEN="${TOKEN_ID}.${TOKEN_SECRET}"

kubectl create secret generic "bootstrap-token-${TOKEN_ID}" \
  --namespace kube-system \
  --type=bootstrap.kubernetes.io/token \
  --from-literal=token-id="$TOKEN_ID" \
  --from-literal=token-secret="$TOKEN_SECRET" \
  --from-literal=usage-bootstrap-authentication=true \
  --from-literal=usage-bootstrap-signing=true
```

#### 6C.4 - Install Karpenter

Create `karpenter-values.yaml`, substituting the variables set above:

```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
replicas: 2
controller:
  env:
    - { name: LEADER_ELECT, value: "true" }
    - { name: CLUSTER_NAME, value: "${CLUSTER_NAME}" }
    - { name: CLUSTER_ENDPOINT, value: "${CLUSTER_ENDPOINT}" }
    - { name: KUBELET_BOOTSTRAP_TOKEN, value: "${BOOTSTRAP_TOKEN}" }
    - { name: NETWORK_PLUGIN, value: "azure" }
    - { name: NETWORK_POLICY, value: "cilium" }
    - { name: NODE_IDENTITIES, value: "${NODE_IDENTITIES}" }
    - { name: ARM_SUBSCRIPTION_ID, value: "${SUBSCRIPTION_ID}" }
    - { name: LOCATION, value: "${LOCATION}" }
    - { name: VNET_SUBNET_ID, value: "${AKS_SUBNET_ID}" }
    - { name: SSH_PUBLIC_KEY, value: "${SSH_PUBLIC_KEY}" }
    - { name: VNET_GUID, value: "${VNET_GUID}" }
    - { name: ARM_USE_CREDENTIAL_FROM_ENVIRONMENT, value: "true" }
    - { name: ARM_USE_MANAGED_IDENTITY_EXTENSION, value: "false" }
    - { name: ARM_USER_ASSIGNED_IDENTITY_ID, value: "" }
    - { name: AZURE_NODE_RESOURCE_GROUP, value: "${NODE_RESOURCE_GROUP}" }
  resources:
    requests: { cpu: 200m, memory: 256Mi }
    limits: { cpu: "1", memory: 1Gi }
serviceAccount:
  name: karpenter
  annotations:
    azure.workload.identity/client-id: "${KARPENTER_IDENTITY_CLIENT_ID}"
podLabels:
  azure.workload.identity/use: "true"
tolerations:
  - { key: CriticalAddonsOnly, operator: Exists, effect: NoSchedule }
logLevel: info
```

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
envsubst < karpenter-values.yaml | helm upgrade --install karpenter \
  oci://mcr.microsoft.com/aks/karpenter/karpenter \
  --version "${KARPENTER_VERSION}" \
  --namespace kube-system --wait --timeout 5m -f -

kubectl rollout status deployment/karpenter -n kube-system --timeout=120s
```

#### 6C.5 - AKSNodeClass and NodePool for engine nodes

The taint `workspace-name=<workspace>` is what the e6data engine pods tolerate, so nodes are dedicated to the workspace:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat << EOF | kubectl apply -f -
apiVersion: karpenter.azure.com/v1alpha2
kind: AKSNodeClass
metadata:
  name: ${WORKSPACE_NAME}-nodeclass
  labels:
    app: e6data
    e6data-workspace-name: ${WORKSPACE_NAME}
spec:
  imageFamily: Ubuntu2204
  osDiskSizeGB: 100
  tags:
    Name: ${WORKSPACE_NAME}
    app: e6data
    namespace: ${WORKSPACE_NAME}
    type: internal-compute
    ManagedBy: karpenter
EOF

cat << EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: ${WORKSPACE_NAME}-nodepool
  labels:
    workspace-name: ${WORKSPACE_NAME}
    type: ${WORKSPACE_NAME}
spec:
  disruption:
    budgets:
      - nodes: 100%
        reasons: [Empty]
      - nodes: "0"
        reasons: [Drifted]
    consolidateAfter: 30s
    consolidationPolicy: WhenEmpty
  limits:
    cpu: "5000"
  template:
    metadata:
      labels:
        workspace-name: ${WORKSPACE_NAME}
        type: ${WORKSPACE_NAME}
    spec:
      expireAfter: 720h
      nodeClassRef:
        group: karpenter.azure.com
        kind: AKSNodeClass
        name: ${WORKSPACE_NAME}-nodeclass
      requirements:
        - key: karpenter.azure.com/sku-family
          operator: In
          values: [D, E, F, L]
        - key: karpenter.sh/capacity-type
          operator: In
          values: [on-demand, spot]
        - key: kubernetes.io/arch
          operator: In
          values: [arm64]
      taints:
        - { effect: NoSchedule, key: workspace-name, value: ${WORKSPACE_NAME} }
EOF

kubectl get aksnodeclasses.karpenter.azure.com
kubectl get nodepools.karpenter.sh
```

Notes:

* `kubernetes.io/arch: arm64` with sku-families `D/E/F/L` makes Karpenter pick ARM variants. For x86, change arch to `amd64`.
* `capacity-type` includes `spot`; remove it for on-demand only.
* A `Warning: use v1beta1.AKSNodeClass instead` message is a Karpenter 1.7.x deprecation notice - `v1alpha2` still works and matches the e6data production templates. Safe to ignore.

### Step 7: Create the metadata storage (ADLS Gen2)

e6data stores workspace metadata in an ADLS Gen2 account (hierarchical namespace enabled) with one private container per workspace:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Storage account names: lowercase alphanumeric only, max 24 chars
export STORAGE_ACCOUNT=$(echo "${PREFIX}workspaces" | tr -d '-' | tr '[:upper:]' '[:lower:]' | cut -c1-24)

az storage account create \
  --name $STORAGE_ACCOUNT \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --kind StorageV2 --sku Standard_LRS \
  --min-tls-version TLS1_2 --https-only true \
  --allow-blob-public-access false \
  --enable-hierarchical-namespace true

# Metadata container. --auth-mode login requires your user to hold "Storage Blob Data
# Contributor" on the account (Owner alone is not enough for data-plane operations).
az storage container create \
  --account-name $STORAGE_ACCOUNT \
  --name ${WORKSPACE_NAME}-metadata \
  --auth-mode login

export STORAGE_ACCOUNT_ID=$(az storage account show \
  --name $STORAGE_ACCOUNT --resource-group $RESOURCE_GROUP --query id -o tsv)
```

<Warning>
  Do **not** enable blob versioning or soft delete on this account - both break ABFS/Hadoop compatibility, which the engine relies on.
</Warning>

The storage backend URI used later in the workspace config is `abfss://${WORKSPACE_NAME}-metadata@${STORAGE_ACCOUNT}.dfs.core.windows.net/`.

### Step 8: Create the workspace Managed Identity and federated credentials

All e6data workspace components share **one** user-assigned Managed Identity, with one federated credential per e6data service account that needs Azure access. The core components are **engine** and **console**:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
az identity create \
  --resource-group $RESOURCE_GROUP \
  --name ${PREFIX}-${WORKSPACE_NAME}-engine-identity \
  --location $LOCATION

export ENGINE_IDENTITY_CLIENT_ID=$(az identity show -g $RESOURCE_GROUP \
  --name ${PREFIX}-${WORKSPACE_NAME}-engine-identity --query clientId -o tsv)
export ENGINE_IDENTITY_PRINCIPAL_ID=$(az identity show -g $RESOURCE_GROUP \
  --name ${PREFIX}-${WORKSPACE_NAME}-engine-identity --query principalId -o tsv)

# One federated credential per core service account
for SA in engine console; do
  az identity federated-credential create \
    --name ${PREFIX}-${WORKSPACE_NAME}-${SA}-federated \
    --identity-name ${PREFIX}-${WORKSPACE_NAME}-engine-identity \
    --resource-group $RESOURCE_GROUP \
    --issuer $AKS_OIDC_ISSUER \
    --subject "system:serviceaccount:${WORKSPACE_NAME}:${WORKSPACE_NAME}-${SA}" \
    --audiences "api://AzureADTokenExchange"
done

echo "ENGINE_IDENTITY_CLIENT_ID=$ENGINE_IDENTITY_CLIENT_ID"
```

<Info>
  When you later enable add-ons (Laminar, Copilot, Metriq), add a federated credential for each of their service accounts against this same identity. The monitoring service account intentionally has no identity binding - it pushes to GreptimeDB with username and password.
</Info>

### Step 9: Role assignments for storage access

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Read/write on the metadata storage account
az role assignment create \
  --assignee-object-id $ENGINE_IDENTITY_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" \
  --scope $STORAGE_ACCOUNT_ID

# Read-only on each storage account that holds your data lake (repeat per account)
az role assignment create \
  --assignee-object-id $ENGINE_IDENTITY_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Reader" \
  --scope "/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/<data-rg>/providers/Microsoft.Storage/storageAccounts/<data-storage-account>"
```

### Part 1 verification

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME \
  --query "{k8s:kubernetesVersion, oidc:oidcIssuerProfile.enabled, wi:securityProfile.workloadIdentity.enabled, network:networkProfile.networkPlugin}" -o table

kubectl get nodes -o wide

az identity federated-credential list \
  --identity-name ${PREFIX}-${WORKSPACE_NAME}-engine-identity \
  --resource-group $RESOURCE_GROUP \
  --query "[].{name:name, subject:subject}" -o table
```

Record `AKS_OIDC_ISSUER`, `NODE_RESOURCE_GROUP`, `ENGINE_IDENTITY_CLIENT_ID`, `STORAGE_ACCOUNT`, and the storage backend URI - the workspace deployment uses them.

## Part 2 - Kubernetes platform components

This part installs the cluster-level components: an `e6operator` NodePool, cert-manager, the e6-operator CRDs, the e6-operator controller, and the service-token verifier secret. Karpenter itself was installed in Part 1 Step 6C. e6data provides the operator chart and CRDs during onboarding.

<Info>
  The recommended layout pins cert-manager and the operator onto dedicated `e6operator` nodes (label `app: e6operator`, taint `workload=e6operator`). Step 2.1 creates that NodePool if you run Karpenter. If you use static or NAP node pools instead, omit the `nodeSelector`/`tolerations` blocks in Steps 2.2 and 2.4 and let the components run on the system pool.
</Info>

### Step 2.1: e6-operator NodePool (Karpenter only)

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat << 'EOF' | kubectl apply -f -
apiVersion: karpenter.azure.com/v1alpha2
kind: AKSNodeClass
metadata:
  name: e6operator
spec:
  imageFamily: Ubuntu2204
  osDiskSizeGB: 50
  tags:
    ManagedBy: karpenter
    Name: e6operator
    app: e6data
    workload: e6operator
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: e6operator
spec:
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30s
  limits:
    cpu: "100"
    memory: 100Gi
  template:
    metadata:
      labels:
        workload: e6operator
        app: e6operator
    spec:
      expireAfter: Never
      nodeClassRef:
        group: karpenter.azure.com
        kind: AKSNodeClass
        name: e6operator
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: [on-demand]
        - key: kubernetes.io/arch
          operator: In
          values: [arm64]
        - key: karpenter.azure.com/sku-family
          operator: In
          values: [D, E]
      taints:
        - { effect: NoSchedule, key: workload, value: e6operator }
EOF
```

### Step 2.2: Install cert-manager

The operator's admission webhooks get their serving certificates from cert-manager, so it is required:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export CERT_MANAGER_VERSION="v1.17.2"

cat > cert-manager-values.yaml << 'EOF'
crds:
  enabled: true
prometheus:
  enabled: false
webhook:
  timeoutSeconds: 4
  tolerations:
    - { key: workload, operator: Equal, value: e6operator, effect: NoSchedule }
  nodeSelector:
    app: e6operator
tolerations:
  - { key: workload, operator: Equal, value: e6operator, effect: NoSchedule }
nodeSelector:
  app: e6operator
cainjector:
  tolerations:
    - { key: workload, operator: Equal, value: e6operator, effect: NoSchedule }
  nodeSelector:
    app: e6operator
startupapicheck:
  tolerations:
    - { key: workload, operator: Equal, value: e6operator, effect: NoSchedule }
  nodeSelector:
    app: e6operator
EOF

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version "${CERT_MANAGER_VERSION}" \
  --namespace cert-manager --create-namespace --wait \
  -f cert-manager-values.yaml

kubectl get pods -n cert-manager
```

### Step 2.3: Install the e6-operator CRDs

The CRDs are applied with `kubectl`, **not** Helm, because the combined payload exceeds Helm's 1 MB release-secret limit. `--server-side` is required because client-side apply hits the 256 KB annotation limit:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Extract the chart provided by e6data
tar -xzf e6-operator-crds.tgz

kubectl apply --server-side --force-conflicts -f ./e6-operator-crds/crds/

# Expect ~25 CRDs in the e6data.io group
kubectl get crds | grep e6data.io
```

Key CRDs used in workspace deployment: `namespaceconfigs`, `queryrouters`, `monitoringservices`, `metadataservices`, `queryservices`, `accesstokens`, `e6catalogs`, `governances`.

### Step 2.4: Install the e6-operator controller

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Extract the chart; the helm path must point to the folder containing Chart.yaml
tar -xzf e6-operator.tgz

export E6_OPERATOR_IMAGE_TAG="<VERSION_PROVIDED_BY_E6DATA>"

cat > operator-values.yaml << EOF
replicaCount: 2
image:
  repository: e6labs.azurecr.io/e6data/e6-operator
  tag: "${E6_OPERATOR_IMAGE_TAG}"
  pullPolicy: Always
# Option B (pull secret): uncomment the next two lines.
# imagePullSecrets:
#   - name: e6data-acr-pull
controllerManager:
  logLevel: info
  leaderElection:
    enabled: true
env:
  - name: LOG_MODE
    value: "production"
# The chart key is `webhook` (singular)
webhook:
  enabled: true
  certManager:
    enabled: true
resources:
  requests: { cpu: 200m, memory: 256Mi }
  limits: { cpu: "1", memory: 1Gi }
priorityClassName: system-cluster-critical
tolerations:
  - { key: workload, operator: Equal, value: e6operator, effect: NoSchedule }
nodeSelector:
  app: e6operator
karpenter:
  enabled: true        # set false if using static pools or NAP
# clusterMonitoring activates only when greptime.endpoint is set. In-VPC tenants run
# their own cluster monitoring, so leave it unset (stays off).
EOF

helm upgrade --install e6operator ./e6-operator \
  --namespace e6operator --create-namespace --wait --timeout 8m \
  -f operator-values.yaml

kubectl rollout status deployment/e6operator -n e6operator --timeout=300s
kubectl get pods -n e6operator
```

### Step 2.5: Create the service-token verifier secret

The console authenticates `e6_svc_` service tokens against the e6data control plane using a shared verifier secret. The e6-operator reflects this secret into every workspace namespace as `<workspace>-cp-verifier`.

<Warning>
  If this secret is missing, service-token authentication silently fails with `401`. Create it now with the value from your onboarding engineer.
</Warning>

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export CP_VERIFIER_SECRET_VALUE="<value-provided-by-e6data>"

kubectl create secret generic cp-verifier-shared \
  --namespace e6operator \
  --from-file=SERVICE_CREDENTIAL_VERIFIER_SECRET=<(printf '%s' "$CP_VERIFIER_SECRET_VALUE") \
  --dry-run=client -o yaml | kubectl apply -f -
```

## Next

Continue to [Deploy workspace and e6data](/platform/deploy/azure-cloudprem/deploy-workspace).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.