Configure registry, Kubernetes, and networking

Build the Azure infrastructure (VNet, AKS, identities, ADLS storage) and install the Kubernetes platform components for an In-VPC e6data deployment on AKS.

This page builds the Azure infrastructure for an In-VPC deployment and installs the cluster-wide Kubernetes components the e6data workspace runs on. Run these steps with the Azure CLI (az), kubectl, and Helm.

Part 1 provisions Azure infrastructure (run once per cluster). Part 2 installs the Kubernetes platform components (also once per cluster). Per-workspace deployment is covered on the next page, Deploy workspace and e6data.

Infrastructure requirements

Review these before you begin. They apply whether you create a new VNet (Part 1, Step 1) or reuse an existing one.

VNet options

Option	When to use
New VNet	Recommended for an isolated e6data deployment. Follow Part 1, Step 1.
Existing VNet	When e6data must share the network with other workloads. Skip Step 1 and meet the checklist below.

VNet and subnet sizing

With Azure CNI overlay (used here), pods get IPs from the pod CIDR (10.244.0.0/16), not the subnet - so the AKS subnet only needs addresses for nodes, not pods.

Resource	Minimum	Recommended	Notes
VNet CIDR	/20	/16 (10.0.0.0/16)	Must not overlap pod/service CIDRs or peered networks.
AKS subnet	/24	/17 (10.0.0.0/17)	Nodes only (pods are on the overlay).
Pod CIDR	-	10.244.0.0/16	Virtual (overlay); must not overlap the VNet or peers.
Service CIDR	-	172.20.0.0/16	Cluster-internal; must not overlap the VNet or peers.

Availability zones

Requirement	Minimum	Recommended
Zones	2	3 (1, 2, 3)

e6data spreads workloads across zones for high availability - the engine NodePool lists zones 1, 2, and 3.

Azure quotas (vCPU)

Quota is per VM family, per region. Check and request increases before deploying - increases can take 24–48 hours.

Family	Used by
Dpsv6 / Dpsv5 (ARM)	System pool (Standard_D2ps_v6) and the operator pool.
D / E / F / L (ARM)	Engine nodes (Karpenter).
Total Regional vCPUs	All of the above.

Sponsorship and free subscriptions often start with 0 quota for ARM families - request an increase first if az aks create fails with a quota or SKU-not-available error.

VM families

Component	VM families	Architecture	Notes
System pool	Standard_D2ps_v6	arm64	AKS system pods; use Standard_D4as_v5 for x86.
e6-operator	D, E	arm64	Operator and cert-manager (e6operator NodePool).
Engine nodes	D, E, F, L	arm64	Query engine (Karpenter); the L family adds local NVMe.

The e6data engine is optimized for ARM64 for price-performance. Confirm your region has the chosen families available.

Existing VNet checklist

If you reuse an existing VNet, confirm:

• No CIDR overlap between the VNet, pod CIDR (10.244.0.0/16), service CIDR (172.20.0.0/16), or any peered network.
• The AKS subnet has room for your node count (with CNI overlay, pods don't consume subnet IPs).
• Nodes have outbound internet (a NAT Gateway is recommended).
• Microsoft.Storage and Microsoft.ContainerRegistry service endpoints are enabled on the subnet.

Existing AKS cluster checklist

If you already have an AKS cluster, skip Part 1 Steps 1–3 and verify the cluster meets the requirements below first:

az aks show -g <rg> -n <cluster> \
  --query "{k8s:kubernetesVersion, oidc:oidcIssuerProfile.enabled, wi:securityProfile.workloadIdentity.enabled, network:networkProfile.networkPlugin, outbound:networkProfile.outboundType}" -o yaml

Requirement	Expected	If not met
OIDC issuer + Workload Identity	both true	az aks update -g <rg> -n <cluster> --enable-oidc-issuer --enable-workload-identity
Kubernetes version	1.29+ (1.33 ideal)	Upgrade the cluster.
Outbound egress	reaches e6labs.azurecr.io, mcr.microsoft.com, quay.io, app.e6.run	Allow-list these (Azure Firewall / UDR).
Deployer permissions	User Access Administrator	Grant it - required for the role assignments.

OIDC issuer + Workload Identity is the setting that silently breaks everything if it's off. Check it first.

Then skip to Step 4 (cluster identity) and continue through Parts 1 and 2 as written.

Part 1 - Azure infrastructure

By the end of this part you will have a VNet with a NAT Gateway, an AKS cluster with OIDC issuer and Workload Identity enabled, a system node pool and an engine node pool strategy, an ADLS Gen2 storage account, and a workspace Managed Identity federated to the e6data service accounts.

Step 0: Set variables

# Naming and placement
export PREFIX="e6data"                  # resource name prefix (lowercase, no spaces)
export LOCATION="eastus"
export RESOURCE_GROUP="${PREFIX}-rg"
export CLUSTER_NAME="${PREFIX}-aks"

# Network
export VNET_CIDR="10.0.0.0/16"
export AKS_SUBNET_CIDR="10.0.0.0/17"     # first half of the VNet

# Cluster
export K8S_VERSION="1.33"

# Workspace (also used as the Kubernetes namespace)
export WORKSPACE_NAME="my-workspace"

# Subscription context
export SUBSCRIPTION_ID=$(az account show --query id -o tsv)

Step 1: Create the resource group, VNet, and NAT Gateway

az group create --name $RESOURCE_GROUP --location $LOCATION

az network vnet create \
  --resource-group $RESOURCE_GROUP \
  --name ${PREFIX}-vnet \
  --address-prefix $VNET_CIDR \
  --location $LOCATION

# AKS subnet (nodes + overlay pods). Service endpoints keep Storage/ACR traffic on the Azure backbone.
az network vnet subnet create \
  --resource-group $RESOURCE_GROUP \
  --vnet-name ${PREFIX}-vnet \
  --name ${PREFIX}-aks \
  --address-prefixes $AKS_SUBNET_CIDR \
  --service-endpoints Microsoft.Storage Microsoft.ContainerRegistry

# NAT Gateway for stable egress (static public IP, 30-minute idle timeout)
az network public-ip create \
  --resource-group $RESOURCE_GROUP \
  --name ${PREFIX}-nat-pip \
  --sku Standard \
  --allocation-method Static \
  --location $LOCATION

az network nat gateway create \
  --resource-group $RESOURCE_GROUP \
  --name ${PREFIX}-nat \
  --public-ip-addresses ${PREFIX}-nat-pip \
  --idle-timeout 30 \
  --location $LOCATION

az network vnet subnet update \
  --resource-group $RESOURCE_GROUP \
  --vnet-name ${PREFIX}-vnet \
  --name ${PREFIX}-aks \
  --nat-gateway ${PREFIX}-nat

# Capture IDs for cluster creation
export AKS_SUBNET_ID=$(az network vnet subnet show \
  --resource-group $RESOURCE_GROUP --vnet-name ${PREFIX}-vnet --name ${PREFIX}-aks \
  --query id -o tsv)

export VNET_ID=$(az network vnet show \
  --resource-group $RESOURCE_GROUP --name ${PREFIX}-vnet --query id -o tsv)

echo "AKS_SUBNET_ID=$AKS_SUBNET_ID"

Step 2: Create the AKS cluster

Create the cluster with OIDC issuer and Workload Identity enabled, Azure CNI overlay with Cilium, and an ARM64 system pool tainted CriticalAddonsOnly:

az aks create \
  --resource-group $RESOURCE_GROUP \
  --name $CLUSTER_NAME \
  --location $LOCATION \
  --kubernetes-version $K8S_VERSION \
  --tier standard \
  --node-resource-group ${CLUSTER_NAME}-nodes \
  --enable-managed-identity \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --network-plugin azure \
  --network-plugin-mode overlay \
  --network-policy cilium \
  --network-dataplane cilium \
  --pod-cidr 10.244.0.0/16 \
  --service-cidr 172.20.0.0/16 \
  --dns-service-ip 172.20.0.10 \
  --outbound-type loadBalancer \
  --load-balancer-sku standard \
  --vnet-subnet-id $AKS_SUBNET_ID \
  --nodepool-name system \
  --node-vm-size Standard_D2ps_v6 \
  --node-count 2 \
  --enable-cluster-autoscaler \
  --min-count 2 \
  --max-count 4 \
  --node-osdisk-size 50 \
  --max-pods 110 \
  --nodepool-taints CriticalAddonsOnly=true:NoSchedule \
  --generate-ssh-keys

Notes:

• Standard_D2ps_v6 is ARM64. All e6data system images are multi-arch; for x86 use Standard_D4as_v5.
• The NAT Gateway on the subnet takes precedence for egress even with --outbound-type loadBalancer.
• To use Entra ID (Azure AD) RBAC for cluster auth, append --enable-aad --enable-azure-rbac --disable-local-accounts --aad-admin-group-object-ids <group-object-id>.

Step 3: Connect and capture cluster outputs

az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME

# OIDC issuer URL - needed for every federated identity credential below
export AKS_OIDC_ISSUER=$(az aks show \
  --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME \
  --query "oidcIssuerProfile.issuerUrl" -o tsv)

# Node resource group (where AKS puts VMs/VMSS/LBs)
export NODE_RESOURCE_GROUP=$(az aks show \
  --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME \
  --query nodeResourceGroup -o tsv)

echo "AKS_OIDC_ISSUER=$AKS_OIDC_ISSUER"
kubectl get nodes

AKS hosts the OIDC issuer for you - there is nothing to create. You only reference its URL when creating federated credentials.

Step 4: Grant the cluster identity network access

The cluster's control-plane identity needs Network Contributor on the VNet so the Azure cloud provider can manage load balancer backend pools for nodes in your subnet:

CLUSTER_IDENTITY_PRINCIPAL_ID=$(az aks show \
  --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME \
  --query identity.principalId -o tsv)

az role assignment create \
  --assignee-object-id $CLUSTER_IDENTITY_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "Network Contributor" \
  --scope $VNET_ID

Step 5: Container registry access

e6data images are pulled from e6labs.azurecr.io. Choose one option and confirm it with your onboarding engineer (they configure the matching side).

Option A - AcrPull role (if e6data grants your kubelet identity access, or you mirror images to your own ACR):

KUBELET_OBJECT_ID=$(az aks show \
  --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME \
  --query identityProfile.kubeletidentity.objectId -o tsv)

# Scope = your mirrored ACR resource ID, or the e6data-shared ACR ID from onboarding
az role assignment create \
  --assignee-object-id $KUBELET_OBJECT_ID \
  --assignee-principal-type ServicePrincipal \
  --role AcrPull \
  --scope "<acr-resource-id>"

Option B - Pull secret (e6data provides a scoped ACR token). The secret must exist in both the e6operator namespace and the workspace namespace before components install, so create the namespaces now:

for NS in e6operator ${WORKSPACE_NAME}; do
  kubectl create namespace ${NS} --dry-run=client -o yaml | kubectl apply -f -
  kubectl create secret docker-registry e6data-acr-pull \
    --namespace ${NS} \
    --docker-server=e6labs.azurecr.io \
    --docker-username="<token-name-from-e6data>" \
    --docker-password="<token-password-from-e6data>"
done

With Option B, also reference e6data-acr-pull wherever the guide notes imagePullSecrets (operator values in Step 2.4, NamespaceConfig in the next page).

Step 6: Engine node pool

Engine pods need a user node pool to scale on. Pick one provisioner. The workspace-name label and taint are mandatory - engine pods schedule onto nodes by exactly this label/taint pair.

Option A - Static user node pool (simplest):

az aks nodepool add \
  --resource-group $RESOURCE_GROUP \
  --cluster-name $CLUSTER_NAME \
  --name engine \
  --mode User \
  --node-vm-size Standard_D8as_v5 \
  --enable-cluster-autoscaler \
  --min-count 0 --max-count 10 \
  --node-osdisk-size 100 \
  --zones 1 2 3 \
  --labels workspace-name=${WORKSPACE_NAME} \
  --node-taints workspace-name=${WORKSPACE_NAME}:NoSchedule

Option B - Node Auto Provisioning (NAP, managed Karpenter):

az aks update --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME \
  --node-provisioning-mode Auto

Option C - Self-hosted Karpenter (recommended for scale-to-zero engine nodes). This uses the AKS Karpenter provider published in Microsoft's container registry. It has five sub-steps: identity, cluster values, bootstrap token, Helm install, and the NodeClass/NodePool.

6C.1 - Karpenter Managed Identity and roles

az identity create \
  --resource-group $RESOURCE_GROUP \
  --name ${CLUSTER_NAME}-karpenter-identity \
  --location $LOCATION

KARPENTER_PRINCIPAL_ID=$(az identity show -g $RESOURCE_GROUP \
  --name ${CLUSTER_NAME}-karpenter-identity --query principalId -o tsv)
export KARPENTER_IDENTITY_CLIENT_ID=$(az identity show -g $RESOURCE_GROUP \
  --name ${CLUSTER_NAME}-karpenter-identity --query clientId -o tsv)

# Federated credential -> kube-system/karpenter service account
az identity federated-credential create \
  --name ${CLUSTER_NAME}-karpenter-federated \
  --identity-name ${CLUSTER_NAME}-karpenter-identity \
  --resource-group $RESOURCE_GROUP \
  --issuer $AKS_OIDC_ISSUER \
  --subject "system:serviceaccount:kube-system:karpenter" \
  --audiences "api://AzureADTokenExchange"

# Role assignments
NODE_RG_ID="/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${NODE_RESOURCE_GROUP}"
RG_ID="/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}"

az role assignment create --assignee-object-id $KARPENTER_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal --role "Virtual Machine Contributor" --scope $NODE_RG_ID
az role assignment create --assignee-object-id $KARPENTER_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal --role "Network Contributor" --scope $VNET_ID
az role assignment create --assignee-object-id $KARPENTER_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal --role "Managed Identity Operator" --scope $NODE_RG_ID
az role assignment create --assignee-object-id $KARPENTER_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal --role "Reader" --scope $RG_ID

6C.2 - Gather cluster values

export KARPENTER_VERSION="1.7.1"
export CLUSTER_ENDPOINT="https://$(az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME --query fqdn -o tsv):443"
export NODE_IDENTITIES=$(az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME \
  --query identityProfile.kubeletidentity.resourceId -o tsv)
export VNET_GUID=$(az network vnet show -g $RESOURCE_GROUP -n ${PREFIX}-vnet --query resourceGuid -o tsv)
export SSH_PUBLIC_KEY="$(cat ~/.ssh/id_rsa.pub)"

6C.3 - Bootstrap token

New nodes use this token to join the cluster. It must be in <6-char-id>.<16-char-secret> format:

TOKEN_ID=$(LC_ALL=C tr -dc 'a-z0-9' < /dev/urandom | head -c 6)
TOKEN_SECRET=$(LC_ALL=C tr -dc 'a-z0-9' < /dev/urandom | head -c 16)
export BOOTSTRAP_TOKEN="${TOKEN_ID}.${TOKEN_SECRET}"

kubectl create secret generic "bootstrap-token-${TOKEN_ID}" \
  --namespace kube-system \
  --type=bootstrap.kubernetes.io/token \
  --from-literal=token-id="$TOKEN_ID" \
  --from-literal=token-secret="$TOKEN_SECRET" \
  --from-literal=usage-bootstrap-authentication=true \
  --from-literal=usage-bootstrap-signing=true

6C.4 - Install Karpenter

Create karpenter-values.yaml, substituting the variables set above:

replicas: 2
controller:
  env:
    - { name: LEADER_ELECT, value: "true" }
    - { name: CLUSTER_NAME, value: "${CLUSTER_NAME}" }
    - { name: CLUSTER_ENDPOINT, value: "${CLUSTER_ENDPOINT}" }
    - { name: KUBELET_BOOTSTRAP_TOKEN, value: "${BOOTSTRAP_TOKEN}" }
    - { name: NETWORK_PLUGIN, value: "azure" }
    - { name: NETWORK_POLICY, value: "cilium" }
    - { name: NODE_IDENTITIES, value: "${NODE_IDENTITIES}" }
    - { name: ARM_SUBSCRIPTION_ID, value: "${SUBSCRIPTION_ID}" }
    - { name: LOCATION, value: "${LOCATION}" }
    - { name: VNET_SUBNET_ID, value: "${AKS_SUBNET_ID}" }
    - { name: SSH_PUBLIC_KEY, value: "${SSH_PUBLIC_KEY}" }
    - { name: VNET_GUID, value: "${VNET_GUID}" }
    - { name: ARM_USE_CREDENTIAL_FROM_ENVIRONMENT, value: "true" }
    - { name: ARM_USE_MANAGED_IDENTITY_EXTENSION, value: "false" }
    - { name: ARM_USER_ASSIGNED_IDENTITY_ID, value: "" }
    - { name: AZURE_NODE_RESOURCE_GROUP, value: "${NODE_RESOURCE_GROUP}" }
  resources:
    requests: { cpu: 200m, memory: 256Mi }
    limits: { cpu: "1", memory: 1Gi }
serviceAccount:
  name: karpenter
  annotations:
    azure.workload.identity/client-id: "${KARPENTER_IDENTITY_CLIENT_ID}"
podLabels:
  azure.workload.identity/use: "true"
tolerations:
  - { key: CriticalAddonsOnly, operator: Exists, effect: NoSchedule }
logLevel: info

envsubst < karpenter-values.yaml | helm upgrade --install karpenter \
  oci://mcr.microsoft.com/aks/karpenter/karpenter \
  --version "${KARPENTER_VERSION}" \
  --namespace kube-system --wait --timeout 5m -f -

kubectl rollout status deployment/karpenter -n kube-system --timeout=120s

6C.5 - AKSNodeClass and NodePool for engine nodes

The taint workspace-name=<workspace> is what the e6data engine pods tolerate, so nodes are dedicated to the workspace:

cat << EOF | kubectl apply -f -
apiVersion: karpenter.azure.com/v1alpha2
kind: AKSNodeClass
metadata:
  name: ${WORKSPACE_NAME}-nodeclass
  labels:
    app: e6data
    e6data-workspace-name: ${WORKSPACE_NAME}
spec:
  imageFamily: Ubuntu2204
  osDiskSizeGB: 100
  tags:
    Name: ${WORKSPACE_NAME}
    app: e6data
    namespace: ${WORKSPACE_NAME}
    type: internal-compute
    ManagedBy: karpenter
EOF

cat << EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: ${WORKSPACE_NAME}-nodepool
  labels:
    workspace-name: ${WORKSPACE_NAME}
    type: ${WORKSPACE_NAME}
spec:
  disruption:
    budgets:
      - nodes: 100%
        reasons: [Empty]
      - nodes: "0"
        reasons: [Drifted]
    consolidateAfter: 30s
    consolidationPolicy: WhenEmpty
  limits:
    cpu: "5000"
  template:
    metadata:
      labels:
        workspace-name: ${WORKSPACE_NAME}
        type: ${WORKSPACE_NAME}
    spec:
      expireAfter: 720h
      nodeClassRef:
        group: karpenter.azure.com
        kind: AKSNodeClass
        name: ${WORKSPACE_NAME}-nodeclass
      requirements:
        - key: karpenter.azure.com/sku-family
          operator: In
          values: [D, E, F, L]
        - key: karpenter.sh/capacity-type
          operator: In
          values: [on-demand, spot]
        - key: kubernetes.io/arch
          operator: In
          values: [arm64]
      taints:
        - { effect: NoSchedule, key: workspace-name, value: ${WORKSPACE_NAME} }
EOF

kubectl get aksnodeclasses.karpenter.azure.com
kubectl get nodepools.karpenter.sh

Notes:

• kubernetes.io/arch: arm64 with sku-families D/E/F/L makes Karpenter pick ARM variants. For x86, change arch to amd64.
• capacity-type includes spot; remove it for on-demand only.
• A Warning: use v1beta1.AKSNodeClass instead message is a Karpenter 1.7.x deprecation notice - v1alpha2 still works and matches the e6data production templates. Safe to ignore.

Step 7: Create the metadata storage (ADLS Gen2)

e6data stores workspace metadata in an ADLS Gen2 account (hierarchical namespace enabled) with one private container per workspace:

# Storage account names: lowercase alphanumeric only, max 24 chars
export STORAGE_ACCOUNT=$(echo "${PREFIX}workspaces" | tr -d '-' | tr '[:upper:]' '[:lower:]' | cut -c1-24)

az storage account create \
  --name $STORAGE_ACCOUNT \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --kind StorageV2 --sku Standard_LRS \
  --min-tls-version TLS1_2 --https-only true \
  --allow-blob-public-access false \
  --enable-hierarchical-namespace true

# Metadata container. --auth-mode login requires your user to hold "Storage Blob Data
# Contributor" on the account (Owner alone is not enough for data-plane operations).
az storage container create \
  --account-name $STORAGE_ACCOUNT \
  --name ${WORKSPACE_NAME}-metadata \
  --auth-mode login

export STORAGE_ACCOUNT_ID=$(az storage account show \
  --name $STORAGE_ACCOUNT --resource-group $RESOURCE_GROUP --query id -o tsv)

Do not enable blob versioning or soft delete on this account - both break ABFS/Hadoop compatibility, which the engine relies on.

The storage backend URI used later in the workspace config is abfss://${WORKSPACE_NAME}-metadata@${STORAGE_ACCOUNT}.dfs.core.windows.net/.

Step 8: Create the workspace Managed Identity and federated credentials

All e6data workspace components share one user-assigned Managed Identity, with one federated credential per e6data service account that needs Azure access. The core components are engine and console:

az identity create \
  --resource-group $RESOURCE_GROUP \
  --name ${PREFIX}-${WORKSPACE_NAME}-engine-identity \
  --location $LOCATION

export ENGINE_IDENTITY_CLIENT_ID=$(az identity show -g $RESOURCE_GROUP \
  --name ${PREFIX}-${WORKSPACE_NAME}-engine-identity --query clientId -o tsv)
export ENGINE_IDENTITY_PRINCIPAL_ID=$(az identity show -g $RESOURCE_GROUP \
  --name ${PREFIX}-${WORKSPACE_NAME}-engine-identity --query principalId -o tsv)

# One federated credential per core service account
for SA in engine console; do
  az identity federated-credential create \
    --name ${PREFIX}-${WORKSPACE_NAME}-${SA}-federated \
    --identity-name ${PREFIX}-${WORKSPACE_NAME}-engine-identity \
    --resource-group $RESOURCE_GROUP \
    --issuer $AKS_OIDC_ISSUER \
    --subject "system:serviceaccount:${WORKSPACE_NAME}:${WORKSPACE_NAME}-${SA}" \
    --audiences "api://AzureADTokenExchange"
done

echo "ENGINE_IDENTITY_CLIENT_ID=$ENGINE_IDENTITY_CLIENT_ID"

When you later enable add-ons (Laminar, Copilot, Metriq), add a federated credential for each of their service accounts against this same identity. The monitoring service account intentionally has no identity binding - it pushes to GreptimeDB with username and password.

Step 9: Role assignments for storage access

# Read/write on the metadata storage account
az role assignment create \
  --assignee-object-id $ENGINE_IDENTITY_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" \
  --scope $STORAGE_ACCOUNT_ID

# Read-only on each storage account that holds your data lake (repeat per account)
az role assignment create \
  --assignee-object-id $ENGINE_IDENTITY_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Reader" \
  --scope "/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/<data-rg>/providers/Microsoft.Storage/storageAccounts/<data-storage-account>"

Part 1 verification

az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME \
  --query "{k8s:kubernetesVersion, oidc:oidcIssuerProfile.enabled, wi:securityProfile.workloadIdentity.enabled, network:networkProfile.networkPlugin}" -o table

kubectl get nodes -o wide

az identity federated-credential list \
  --identity-name ${PREFIX}-${WORKSPACE_NAME}-engine-identity \
  --resource-group $RESOURCE_GROUP \
  --query "[].{name:name, subject:subject}" -o table

Record AKS_OIDC_ISSUER, NODE_RESOURCE_GROUP, ENGINE_IDENTITY_CLIENT_ID, STORAGE_ACCOUNT, and the storage backend URI - the workspace deployment uses them.

Part 2 - Kubernetes platform components

This part installs the cluster-level components: an e6operator NodePool, cert-manager, the e6-operator CRDs, the e6-operator controller, and the service-token verifier secret. Karpenter itself was installed in Part 1 Step 6C. e6data provides the operator chart and CRDs during onboarding.

The recommended layout pins cert-manager and the operator onto dedicated e6operator nodes (label app: e6operator, taint workload=e6operator). Step 2.1 creates that NodePool if you run Karpenter. If you use static or NAP node pools instead, omit the nodeSelector/tolerations blocks in Steps 2.2 and 2.4 and let the components run on the system pool.

Step 2.1: e6-operator NodePool (Karpenter only)

cat << 'EOF' | kubectl apply -f -
apiVersion: karpenter.azure.com/v1alpha2
kind: AKSNodeClass
metadata:
  name: e6operator
spec:
  imageFamily: Ubuntu2204
  osDiskSizeGB: 50
  tags:
    ManagedBy: karpenter
    Name: e6operator
    app: e6data
    workload: e6operator
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: e6operator
spec:
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30s
  limits:
    cpu: "100"
    memory: 100Gi
  template:
    metadata:
      labels:
        workload: e6operator
        app: e6operator
    spec:
      expireAfter: Never
      nodeClassRef:
        group: karpenter.azure.com
        kind: AKSNodeClass
        name: e6operator
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: [on-demand]
        - key: kubernetes.io/arch
          operator: In
          values: [arm64]
        - key: karpenter.azure.com/sku-family
          operator: In
          values: [D, E]
      taints:
        - { effect: NoSchedule, key: workload, value: e6operator }
EOF

Step 2.2: Install cert-manager

The operator's admission webhooks get their serving certificates from cert-manager, so it is required:

export CERT_MANAGER_VERSION="v1.17.2"

cat > cert-manager-values.yaml << 'EOF'
crds:
  enabled: true
prometheus:
  enabled: false
webhook:
  timeoutSeconds: 4
  tolerations:
    - { key: workload, operator: Equal, value: e6operator, effect: NoSchedule }
  nodeSelector:
    app: e6operator
tolerations:
  - { key: workload, operator: Equal, value: e6operator, effect: NoSchedule }
nodeSelector:
  app: e6operator
cainjector:
  tolerations:
    - { key: workload, operator: Equal, value: e6operator, effect: NoSchedule }
  nodeSelector:
    app: e6operator
startupapicheck:
  tolerations:
    - { key: workload, operator: Equal, value: e6operator, effect: NoSchedule }
  nodeSelector:
    app: e6operator
EOF

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version "${CERT_MANAGER_VERSION}" \
  --namespace cert-manager --create-namespace --wait \
  -f cert-manager-values.yaml

kubectl get pods -n cert-manager

Step 2.3: Install the e6-operator CRDs

The CRDs are applied with kubectl, not Helm, because the combined payload exceeds Helm's 1 MB release-secret limit. --server-side is required because client-side apply hits the 256 KB annotation limit:

# Extract the chart provided by e6data
tar -xzf e6-operator-crds.tgz

kubectl apply --server-side --force-conflicts -f ./e6-operator-crds/crds/

# Expect ~25 CRDs in the e6data.io group
kubectl get crds | grep e6data.io

Key CRDs used in workspace deployment: namespaceconfigs, queryrouters, monitoringservices, metadataservices, queryservices, accesstokens, e6catalogs, governances.

Step 2.4: Install the e6-operator controller

# Extract the chart; the helm path must point to the folder containing Chart.yaml
tar -xzf e6-operator.tgz

export E6_OPERATOR_IMAGE_TAG="<VERSION_PROVIDED_BY_E6DATA>"

cat > operator-values.yaml << EOF
replicaCount: 2
image:
  repository: e6labs.azurecr.io/e6data/e6-operator
  tag: "${E6_OPERATOR_IMAGE_TAG}"
  pullPolicy: Always
# Option B (pull secret): uncomment the next two lines.
# imagePullSecrets:
#   - name: e6data-acr-pull
controllerManager:
  logLevel: info
  leaderElection:
    enabled: true
env:
  - name: LOG_MODE
    value: "production"
# The chart key is `webhook` (singular)
webhook:
  enabled: true
  certManager:
    enabled: true
resources:
  requests: { cpu: 200m, memory: 256Mi }
  limits: { cpu: "1", memory: 1Gi }
priorityClassName: system-cluster-critical
tolerations:
  - { key: workload, operator: Equal, value: e6operator, effect: NoSchedule }
nodeSelector:
  app: e6operator
karpenter:
  enabled: true        # set false if using static pools or NAP
# clusterMonitoring activates only when greptime.endpoint is set. In-VPC tenants run
# their own cluster monitoring, so leave it unset (stays off).
EOF

helm upgrade --install e6operator ./e6-operator \
  --namespace e6operator --create-namespace --wait --timeout 8m \
  -f operator-values.yaml

kubectl rollout status deployment/e6operator -n e6operator --timeout=300s
kubectl get pods -n e6operator

Step 2.5: Create the service-token verifier secret

The console authenticates e6_svc_ service tokens against the e6data control plane using a shared verifier secret. The e6-operator reflects this secret into every workspace namespace as <workspace>-cp-verifier.

If this secret is missing, service-token authentication silently fails with 401. Create it now with the value from your onboarding engineer.

export CP_VERIFIER_SECRET_VALUE="<value-provided-by-e6data>"

kubectl create secret generic cp-verifier-shared \
  --namespace e6operator \
  --from-file=SERVICE_CREDENTIAL_VERIFIER_SECRET=<(printf '%s' "$CP_VERIFIER_SECRET_VALUE") \
  --dry-run=client -o yaml | kubectl apply -f -

Next

Continue to Deploy workspace and e6data.