Troubleshooting

Common issues when deploying or running e6data In Your VPC on Azure, with diagnostic commands and fixes.

Common issues during In-VPC deployment and operation on Azure, with the diagnostic commands and fixes for each.

Metadata service pods not appearing after deploy

Symptom: You applied the NamespaceConfig, but mds-schema-* and mds-storage-* pods don't appear after a few minutes.

The operator waits 60 seconds after the NamespaceConfig is created before provisioning metadata services. If pods haven't appeared after 3–4 minutes total:

kubectl logs deployment/e6operator -n e6operator -c manager | grep mdsAutoCreation

If you see unknown field "status.mdsAutoCreation", the CRDs need updating - re-apply them (Part 2 Step 2.3):

kubectl apply --server-side --force-conflicts -f ./e6-operator-crds/crds/

Metadata service pods stuck in Pending

kubectl describe pod <mds-pod> -n <WORKSPACE_NAME> | tail -20

Event

Fix

node(s) had untolerated taint

Add tolerations to your node pool, or remove the pod's nodeSelector

insufficient cpu/memory

Scale up your node pool or attach a larger SKU

volume node affinity conflict

A PVC is bound to a zone that doesn't match where the pod is scheduled

Load balancer not getting an external IP

Symptom: The LoadBalancer service stays in Pending and never gets an external IP.

kubectl describe svc <WORKSPACE_NAME>-qr-envoy-external -n <WORKSPACE_NAME>

The AKS cluster identity needs Network Contributor on the VNet to provision load balancers (this is granted in Part 1, Step 4). If the LB stays pending, confirm the assignment exists, then create it if missing:

CLUSTER_IDENTITY=$(az aks show --name <CLUSTER_NAME> --resource-group <RESOURCE_GROUP> \
  --query "identity.principalId" -o tsv)

az role assignment create \
  --assignee-object-id $CLUSTER_IDENTITY \
  --assignee-principal-type ServicePrincipal \
  --role "Network Contributor" \
  --scope "/subscriptions/<SUB_ID>/resourceGroups/<RG>/providers/Microsoft.Network/virtualNetworks/<VNET_NAME>"

Role assignment can take a few minutes to propagate. With the public default, the assigned EXTERNAL-IP is a public address; if you set azure-load-balancer-internal: "true", it is a private 10.x VNet address instead.

Engine service account missing Workload Identity annotation

Symptom: Engine pods can't authenticate to Azure storage; logs show AccessDenied or NoCredentialsError.

The operator may recreate the engine service account during reconciliation without the Workload Identity annotation. Re-annotate and restart:

kubectl annotate sa <WORKSPACE_NAME>-engine -n <WORKSPACE_NAME> \
  azure.workload.identity/client-id=<client-id> --overwrite
kubectl label sa <WORKSPACE_NAME>-engine -n <WORKSPACE_NAME> \
  azure.workload.identity/use=true --overwrite
kubectl rollout restart deployment -n <WORKSPACE_NAME> -l app=engine

Image pull errors

Symptom: Pods stuck in ImagePullBackOff or ErrImagePull.

kubectl describe pod <failing-pod> -n <WORKSPACE_NAME> | grep -A5 Events

Diagnose by which registry-access option you chose:

Option A (token): verify the pull secret exists and is the right type:
```
kubectl get secret e6data-acr-pull -n <WORKSPACE_NAME> -o jsonpath='{.type}'
# Should output: kubernetes.io/dockerconfigjson
```
If missing or wrong, recreate it - see Configure registry, Kubernetes, and networking.
Option B (Managed Identity): confirm with e6data that the AcrPull role assignment is applied and propagated (up to 5 minutes). Share your kubelet identity object ID with e6data support if pulls still fail.

Workspace namespace stuck in Terminating

kubectl get namespace <WORKSPACE_NAME> -o json | jq .status

Look for NamespaceFinalizersRemaining. List remaining resources, then clear the stuck finalizer (usually an e6data custom resource):

kubectl patch <resource-type> <name> -n <WORKSPACE_NAME> \
  --type='merge' -p '{"metadata":{"finalizers":[]}}'

The namespace finishes terminating in 30–60 seconds after finalizers clear.

Operator pod crash-looping

kubectl logs deployment/e6operator -n e6operator -c manager --tail=100

Cause

Fix

CRDs from an older version

Re-apply the CRDs: kubectl apply --server-side --force-conflicts -f ./e6-operator-crds/crds/

Operator can't reach the Kubernetes API

Check NetworkPolicy resources and egress rules

Operator can't pull its image

See "Image pull errors" above

Operator panics on a malformed CR

Capture the panic stack from logs and contact e6data support

When to contact e6data support

Provide your AKS cluster name, resource group, and region; workspace and tenant name; the output of kubectl get pods -n <WORKSPACE_NAME> and kubectl get pods -n e6operator; relevant log excerpts; and what you've already tried. For production incidents, use the support channel in your e6data contract.