Deploy workspace and e6data

Deploy an e6data workspace on AKS - namespace, TLS, service accounts and RBAC, NamespaceConfig, QueryRouter, and DNS, using explicit kubectl and YAML.

With the infrastructure and platform components in place, deploy a workspace. Run this part once per workspace, in order: namespace → TLS → service accounts and RBAC → NamespaceConfig → NVMe init (optional) → QueryRouter → AccessToken (optional) → DNS.

Step 0: Set workspace variables

These commands re-derive the Part 1 values, so this works in a fresh shell. Set PREFIX, RESOURCE_GROUP, and WORKSPACE_NAME to the same values you used in Part 1.

export PREFIX="e6data"
export RESOURCE_GROUP="${PREFIX}-rg"
export LOCATION="eastus"

export WORKSPACE_NAME="my-workspace"
export NAMESPACE="${WORKSPACE_NAME}"
export TENANT_NAME="<tenant-name-from-e6data>"
export CONSOLE_HOSTNAME="e6data.yourdomain.com"   # the workspace endpoint
export IMAGE_REPOSITORY="e6labs.azurecr.io"

# Metadata storage account (from Part 1 Step 7)
export STORAGE_ACCOUNT=$(echo "${PREFIX}workspaces" | tr -d '-' | tr '[:upper:]' '[:lower:]' | cut -c1-24)

# Workspace Managed Identity (from Part 1 Step 8). PREFIX, WORKSPACE_NAME, and RESOURCE_GROUP
# must match Part 1 exactly, or this returns "ResourceNotFound".
export WORKSPACE_IDENTITY_CLIENT_ID=$(az identity show \
  --resource-group $RESOURCE_GROUP \
  --name ${PREFIX}-${WORKSPACE_NAME}-engine-identity \
  --query clientId -o tsv)

# e6data control plane - the console polls this for cluster create/suspend/resume commands
export AGENT_API_BASE_URL="https://app.e6.run"
export AGENT_POLL_INTERVAL="30s"

# Console super-admin(s) - REQUIRED, comma-separated emails. Without this, logins land with
# no admin access. Use your own login email.
export AUTHZ_SUPER_ADMINS="[email protected]"

# JWT auth - REQUIRED. The load balancer is L4, so Envoy enforces auth itself.
# Values provided by e6data during onboarding.
export JWT_ISSUER="https://app.e6.run"
export JWT_JWKS_URI="https://app.e6.run/.well-known/jwks.json"
export JWT_AUDIENCE="e6-controlplane"

# Image tags - use the versions from your e6data release notes
export CONSOLE_IMAGE_TAG="1.0.4603"
export ENVOY_IMAGE_TAG="1.37.1-pgrouter-e6.36"
export XDS_IMAGE_TAG="1.0.350"
export ENVOY_INSTANCE_TYPE="Standard_D4ps_v5"

NAMESPACE must stay equal to WORKSPACE_NAME - the federated identity credentials from Part 1 Step 8 are bound to service accounts in exactly that namespace. If you change it, Workload Identity authentication fails.

Step 1: Create the namespace

kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

Step 2: Create the TLS certificate

The workspace endpoint is served over HTTPS by Envoy, which terminates TLS in-cluster using a Kubernetes secret named envoy-tls. Unlike AWS - where the NLB can terminate TLS with an ACM cert - the Azure load balancer is L4 and passes through, so the certificate always lives in the cluster. Every option below produces the same envoy-tls secret covering ${CONSOLE_HOSTNAME}. Choose one.

Option A: e6data-provided wildcard certificate

If your workspace endpoint is on an e6data domain (for example *.e6.run), e6data provides a CA-signed wildcard certificate. Load it into the secret:

kubectl create secret tls envoy-tls \
  --namespace ${NAMESPACE} \
  --cert=fullchain.pem \
  --key=privkey.pem

Option B: Customer-procured certificate

A certificate from your own CA (DigiCert, Namecheap, Sectigo, or a corporate CA). Works with any domain. Make sure fullchain.pem includes the full chain (leaf plus intermediates):

kubectl create secret tls envoy-tls \
  --namespace ${NAMESPACE} \
  --cert=fullchain.pem \
  --key=privkey.pem

If your certificate is stored in Azure Key Vault, download it first, split it into tls.crt and tls.key, then create the secret the same way.

Option C: Let's Encrypt via cert-manager

cert-manager obtains and auto-renews a free Let's Encrypt certificate using a DNS01 challenge through Azure DNS. This requires the letsencrypt-prod ClusterIssuer (a public domain in an Azure DNS zone you control, plus a cert-manager identity with DNS Zone Contributor):

cat << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: envoy-tls
  namespace: ${NAMESPACE}
spec:
  secretName: envoy-tls
  duration: 2160h    # 90 days
  renewBefore: 720h  # 30 days
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - ${CONSOLE_HOSTNAME}
EOF

kubectl wait certificate/envoy-tls -n ${NAMESPACE} --for=condition=Ready --timeout=300s

Option D: Azure Key Vault + Secrets Store CSI

The closest Azure analog to AWS ACM: Key Vault owns and auto-rotates the certificate, and the Secrets Store CSI driver syncs it into the cluster as the envoy-tls secret. Best for teams already standardized on Key Vault for certificate lifecycle. Enable the azure-keyvault-secrets-provider addon, then create a SecretProviderClass that targets your certificate and sets secretObjects to sync a kubernetes.io/tls secret named envoy-tls. The exact SecretProviderClass is environment-specific - ask your e6data onboarding engineer for a template.

Step 3: Create service accounts and RBAC

Create the core service accounts (engine, monitoring, console, query-router) - each annotated with the Workload Identity client ID from Part 1 - plus their Roles and Bindings. Save as rbac.yaml and apply with envsubst < rbac.yaml | kubectl apply -f -.

# --- Engine ServiceAccount + Role + RoleBinding ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${WORKSPACE_NAME}-engine
  namespace: ${NAMESPACE}
  annotations:
    azure.workload.identity/client-id: ${WORKSPACE_IDENTITY_CLIENT_ID}
  labels:
    azure.workload.identity/use: "true"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ${WORKSPACE_NAME}-engine-role
  namespace: ${NAMESPACE}
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/status", "endpoints", "services", "configmaps", "secrets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "create", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "deployments/status", "replicasets", "replicasets/status"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ${WORKSPACE_NAME}-engine-role-binding
  namespace: ${NAMESPACE}
subjects:
  - kind: ServiceAccount
    name: ${WORKSPACE_NAME}-engine
    namespace: ${NAMESPACE}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ${WORKSPACE_NAME}-engine-role
---
# --- Monitoring ServiceAccount + ClusterRole + ClusterRoleBinding ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${WORKSPACE_NAME}-monitoring
  namespace: ${NAMESPACE}
  annotations:
    azure.workload.identity/client-id: ${WORKSPACE_IDENTITY_CLIENT_ID}
  labels:
    azure.workload.identity/use: "true"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ${WORKSPACE_NAME}-monitoring-role
rules:
  - apiGroups: [""]
    resources: ["nodes", "nodes/proxy", "services", "endpoints", "pods", "events", "namespaces"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["/metrics"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ${WORKSPACE_NAME}-monitoring-role-binding
subjects:
  - kind: ServiceAccount
    name: ${WORKSPACE_NAME}-monitoring
    namespace: ${NAMESPACE}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ${WORKSPACE_NAME}-monitoring-role
---
# --- Console ServiceAccount + RoleBindings ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${WORKSPACE_NAME}-console
  namespace: ${NAMESPACE}
  annotations:
    azure.workload.identity/client-id: ${WORKSPACE_IDENTITY_CLIENT_ID}
  labels:
    azure.workload.identity/use: "true"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ${WORKSPACE_NAME}-console-role-binding
  namespace: ${NAMESPACE}
subjects:
  - kind: ServiceAccount
    name: ${WORKSPACE_NAME}-console
    namespace: ${NAMESPACE}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ${WORKSPACE_NAME}-engine-role
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ${WORKSPACE_NAME}-console-e6crds-role
  namespace: ${NAMESPACE}
rules:
  - apiGroups: ["e6data.io"]
    resources: ["metadataservices", "queryservices", "e6catalogs", "catalogrefreshschedules", "catalogrefreshes", "pools", "governances", "releasemanagers", "accesstokens", "queryrouters"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["e6data.io"]
    resources: ["monitoringservices", "authgateways"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["e6data.io"]
    resources: ["namespaceconfigs"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: ["e6data.io"]
    resources: ["queryservices/status", "metadataservices/status"]
    verbs: ["get", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ${WORKSPACE_NAME}-console-e6crds-role-binding
  namespace: ${NAMESPACE}
subjects:
  - kind: ServiceAccount
    name: ${WORKSPACE_NAME}-console
    namespace: ${NAMESPACE}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ${WORKSPACE_NAME}-console-e6crds-role
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ${WORKSPACE_NAME}-console-warmnodepools-role
rules:
  - apiGroups: ["e6data.io"]
    resources: ["warmnodepools"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ${WORKSPACE_NAME}-console-warmnodepools-role-binding
subjects:
  - kind: ServiceAccount
    name: ${WORKSPACE_NAME}-console
    namespace: ${NAMESPACE}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ${WORKSPACE_NAME}-console-warmnodepools-role
---
# --- Query Router ServiceAccount + Role + RoleBinding ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${WORKSPACE_NAME}-query-router
  namespace: ${NAMESPACE}
  annotations:
    azure.workload.identity/client-id: ${WORKSPACE_IDENTITY_CLIENT_ID}
  labels:
    azure.workload.identity/use: "true"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ${WORKSPACE_NAME}-query-router-role
  namespace: ${NAMESPACE}
rules:
  - apiGroups: [""]
    resources: ["endpoints", "services", "pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["e6data.io"]
    resources: ["queryservices", "queryservices/status"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ${WORKSPACE_NAME}-query-router-role-binding
  namespace: ${NAMESPACE}
subjects:
  - kind: ServiceAccount
    name: ${WORKSPACE_NAME}-query-router
    namespace: ${NAMESPACE}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ${WORKSPACE_NAME}-query-router-role

envsubst < rbac.yaml | kubectl apply -f -
kubectl get sa -n ${NAMESPACE}

This creates the core service accounts. When you enable add-ons (Laminar, Copilot, Metriq), add their service accounts (and Laminar's Role) here, and add a federated credential for each against the workspace identity (Part 1 Step 8).

Step 4: Apply the NamespaceConfig

The NamespaceConfig is the workspace's root custom resource. Once applied, the e6-operator creates the console, the compaction job, and (on demand) the MetadataServices:

cat << EOF | kubectl apply -f -
apiVersion: e6data.io/v1alpha1
kind: NamespaceConfig
metadata:
  name: ${WORKSPACE_NAME}-nsc
  namespace: ${NAMESPACE}
spec:
  cloud: AZURE
  workspaceType: cloud_prem
  region: ${LOCATION}
  workspace: ${WORKSPACE_NAME}
  tenant: ${TENANT_NAME}
  karpenterNodePool: ${WORKSPACE_NAME}-nodepool
  imageRepository: ${IMAGE_REPOSITORY}
  imagePullSecrets: []        # Option B (pull secret): ["e6data-acr-pull"]
  serviceAccounts:
    data: ${WORKSPACE_NAME}-engine
    monitoring: ${WORKSPACE_NAME}-monitoring
    console: ${WORKSPACE_NAME}-console
    queryRouter: ${WORKSPACE_NAME}-query-router
  storageBackend: wasbs://${WORKSPACE_NAME}-metadata@${STORAGE_ACCOUNT}.dfs.core.windows.net
  tolerations:
    - key: "workspace-name"
      operator: "Equal"
      value: "${WORKSPACE_NAME}"
      effect: "NoSchedule"
  nodeSelector:
    workspace-name: "${WORKSPACE_NAME}"
  compaction:
    image:
      name: "e6data/e6-console"
      tag: "${CONSOLE_IMAGE_TAG}"
    schedule: "0 * * * *"
    hoursAgo: 2
    resources:
      requests: { cpu: "500m", memory: "1Gi" }
      limits: { cpu: "2", memory: "4Gi" }
  console:
    enabled: true
    replicas: 2
    image:
      name: "e6data/e6-console"
      tag: "${CONSOLE_IMAGE_TAG}"
    resources:
      requests: { cpu: "8", memory: "8Gi" }
      limits: { cpu: "8", memory: "10Gi" }
    environmentVariables:
      AZURE_REGION: "${LOCATION}"
      CONSOLE_HOSTNAME: "${CONSOLE_HOSTNAME}"
      QUERY_ROUTER_URL: "https://${CONSOLE_HOSTNAME}"
      AGENT_API_BASE_URL: "${AGENT_API_BASE_URL}"
      AGENT_POLL_INTERVAL: "${AGENT_POLL_INTERVAL}"
      AUTHZ_DEFAULT_ROLE: "viewer"
      AUTHZ_SUPER_ADMINS: "${AUTHZ_SUPER_ADMINS}"
EOF

# Wait for the operator to reconcile (phase -> Ready)
kubectl get namespaceconfig ${WORKSPACE_NAME}-nsc -n ${NAMESPACE} \
  -o jsonpath='{.status.phase}{"\n"}' --watch

storageBackend uses the wasbs://<container>@<account>.dfs.core.windows.net URI from Part 1 Step 7. The serviceAccounts block lists the core components; add laminar, copilot, and metriq keys when you enable those add-ons.

Step 5: Engine NodeClass and NodePool

Already created in Part 1 Step 6C.5 (Karpenter). If you chose a static pool or NAP instead, make sure its nodes carry the label and taint workspace-name=${WORKSPACE_NAME} so the NamespaceConfig's nodeSelector and tolerations match.

Step 6: NVMe init DaemonSet (only for NVMe VM families)

If your engine NodePool includes the L family (Lsv3/Lasv3, local NVMe), this DaemonSet formats and RAIDs the NVMe drives and mounts them at /app/tmp:

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvme-raid-disks-${WORKSPACE_NAME}
  namespace: ${NAMESPACE}
  labels:
    k8s-app: nvme-raid-disks
    workspace: ${WORKSPACE_NAME}
spec:
  selector:
    matchLabels:
      name: nvme-raid-disks
      workspace: ${WORKSPACE_NAME}
  template:
    metadata:
      labels:
        name: nvme-raid-disks
        workspace: ${WORKSPACE_NAME}
    spec:
      automountServiceAccountToken: false
      priorityClassName: system-node-critical
      nodeSelector:
        workspace-name: ${WORKSPACE_NAME}
      tolerations:
        - { key: kubernetes.azure.com/scalesetpriority, operator: Equal, value: spot, effect: NoSchedule }
        - { key: workspace-name, operator: Equal, value: ${WORKSPACE_NAME}, effect: NoSchedule }
      containers:
        - name: startup-script
          image: e6labs.azurecr.io/e6data/az-nvme-provisioner:3.0.5-e4fd230d
          imagePullPolicy: Always
          securityContext:
            privileged: true
          volumeMounts:
            - { mountPath: /app/tmp, name: tmp-volume, mountPropagation: Bidirectional }
      volumes:
        - name: tmp-volume
          hostPath:
            path: /app/tmp
EOF

Step 7: Deploy the QueryRouter

The QueryRouter deploys Envoy (data path, TLS termination, JWT enforcement) and the xDS control plane. The external load balancer is L4, so JWT auth in the QueryRouter is required - Envoy authenticates every request. The merged-single-port mode fronts gRPC, HTTP, and PostgreSQL on port 443 behind one load balancer.

cat << EOF | kubectl apply -f -
apiVersion: e6data.io/v1alpha2
kind: QueryRouter
metadata:
  name: ${WORKSPACE_NAME}-qr
  namespace: ${NAMESPACE}
  annotations:
    e6data.io/auth-mode: jwt
spec:
  envoy:
    hpa:
      enabled: true
      targetCPUUtilization: 70
      targetMemoryUtilization: 80
    image:
      name: envoy
      repository: ${IMAGE_REPOSITORY}
      tag: ${ENVOY_IMAGE_TAG}
      pullPolicy: IfNotPresent
    maxReplicas: 10
    replicas: 2
    resources: { cpu: 500m, memory: 512Mi }
    service:
      type: ClusterIP
  xds:
    discovery: {}
    image:
      name: xds-control-plane
      repository: ${IMAGE_REPOSITORY}/e6data
      tag: ${XDS_IMAGE_TAG}
      pullPolicy: IfNotPresent
    pollInterval: 5
    replicas: 1
    resources: { cpu: 200m, memory: 256Mi }
  trafficDefaults:
    blueWeight: 100
    greenWeight: 0
  httpQuery:
    enabled: true
    timeout: 5m
  pgQuery:
    enabled: true
    timeout: 1h
  auth:
    domain: ${CONSOLE_HOSTNAME}
    jwt:
      issuer: ${JWT_ISSUER}
      jwksUri: ${JWT_JWKS_URI}
      audience: ${JWT_AUDIENCE}
    tls:
      secretName: envoy-tls
    mode: merged-single-port
    mergedService:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol: tcp
EOF

# Wait for phase -> Ready
kubectl get queryrouter ${WORKSPACE_NAME}-qr -n ${NAMESPACE} -w

Public by default. With no azure-load-balancer-internal annotation, Azure gives the load balancer a public IP (JWT auth still protects every request). Point your DNS A-record at that public IP.

To instead restrict it to a private endpoint reachable only inside your VNet, add service.beta.kubernetes.io/azure-load-balancer-internal: "true" to mergedService.annotations, and reach it through the VNet (jumpbox, VPN, or peering). The DNS record then points at the private IP.

Step 8: Create an AccessToken (optional)

Static tokens for HTTP/JDBC query authentication:

cat << EOF | kubectl apply -f -
apiVersion: e6data.io/v1alpha1
kind: AccessToken
metadata:
  name: ${WORKSPACE_NAME}-tokens
  namespace: ${NAMESPACE}
spec:
  workspaceRef: ${WORKSPACE_NAME}-workspace
  tokens:
    - uuid: "tok-${WORKSPACE_NAME}-001"
      email: "[email protected]"
      token: "<token-value>"
      createdAt: "2026-01-01T00:00:00Z"
      comment: "Initial access token"
EOF

Step 9: Configure DNS

Point the workspace hostname at the load balancer's IP address:

LB_IP=$(kubectl get svc ${WORKSPACE_NAME}-qr-envoy-external -n ${NAMESPACE} \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "LB_IP=$LB_IP"

Use the EXTERNAL-IP (.status.loadBalancer.ingress[0].ip), not the ClusterIP. With the public default this is a real public address - not a 172.20.x (cluster-internal ClusterIP). A 10.x private VNet IP appears only if you kept the load balancer internal.

# --record-set-name is the hostname WITHOUT the zone suffix
# (e.g. CONSOLE_HOSTNAME=e6data.acme.com, zone=acme.com -> record-set-name "e6data")
az network dns record-set a add-record \
  --resource-group <dns-zone-rg> \
  --zone-name <your-dns-zone> \
  --record-set-name "<hostname-without-zone>" \
  --ipv4-address "${LB_IP}"

For a private/internal load balancer, put the A-record in a Private DNS zone (or an internal resolver), since a public record pointing at a 10.x address only resolves usefully from inside the VNet.

Verify the deployment

# Workspace custom resources and pods
kubectl get namespaceconfigs,queryrouters -n ${NAMESPACE}
kubectl get metadataservices,queryservices -n ${NAMESPACE}   # created by operator/console
kubectl get pods -n ${NAMESPACE}

# External endpoint
kubectl get svc -n ${NAMESPACE} | grep LoadBalancer

# TLS check (run from inside the VNet if the load balancer is internal)
curl -vk https://${CONSOLE_HOSTNAME} 2>&1 | grep -E "subject:|HTTP/"

Expected pods once active: console-*, ${WORKSPACE_NAME}-qr-envoy-*, ${WORKSPACE_NAME}-qr-xds-*, and mds-schema-* / mds-storage-* (after the first MetadataServices is created). Metadata pods may take 1–2 minutes to appear.

Next

• Register a catalog - connect your data.
• Create a cluster - provision compute.
• Run your first query - validate the install.

If something doesn't come up cleanly, see Troubleshooting.