Troubleshooting

Common issues during In-VPC deployment and operation on Azure, with the diagnostic commands and fixes for each.

Metadata service pods not appearing after deploy

Symptom: You applied the NamespaceConfig, but mds-schema-* and mds-storage-* pods don’t appear after a few minutes. The operator waits 60 seconds after the NamespaceConfig is created before provisioning metadata services. If pods haven’t appeared after 3–4 minutes total:

kubectl logs deployment/e6operator -n e6operator -c manager | grep mdsAutoCreation

If you see unknown field "status.mdsAutoCreation", the CRDs need updating - re-apply them (Part 2 Step 2.3):

kubectl apply --server-side --force-conflicts -f ./e6-operator-crds/crds/

Metadata service pods stuck in Pending

kubectl describe pod <mds-pod> -n <WORKSPACE_NAME> | tail -20

Event	Fix
`node(s) had untolerated taint`	Add tolerations to your node pool, or remove the pod’s nodeSelector
`insufficient cpu/memory`	Scale up your node pool or attach a larger SKU
`volume node affinity conflict`	A PVC is bound to a zone that doesn’t match where the pod is scheduled

Load balancer not getting an external IP

Symptom: The LoadBalancer service stays in Pending and never gets an external IP.

kubectl describe svc <WORKSPACE_NAME>-qr-envoy-external -n <WORKSPACE_NAME>

The AKS cluster identity needs Network Contributor on the VNet to provision load balancers (this is granted in Part 1, Step 4). If the LB stays pending, confirm the assignment exists, then create it if missing:

CLUSTER_IDENTITY=$(az aks show --name <CLUSTER_NAME> --resource-group <RESOURCE_GROUP> \
  --query "identity.principalId" -o tsv)

az role assignment create \
  --assignee-object-id $CLUSTER_IDENTITY \
  --assignee-principal-type ServicePrincipal \
  --role "Network Contributor" \
  --scope "/subscriptions/<SUB_ID>/resourceGroups/<RG>/providers/Microsoft.Network/virtualNetworks/<VNET_NAME>"

Role assignment can take a few minutes to propagate. With the public default, the assigned EXTERNAL-IP is a public address; if you set azure-load-balancer-internal: "true", it is a private 10.x VNet address instead.

Engine service account missing Workload Identity annotation

Symptom: Engine pods can’t authenticate to Azure storage; logs show AccessDenied or NoCredentialsError. The operator may recreate the engine service account during reconciliation without the Workload Identity annotation. Re-annotate and restart:

kubectl annotate sa <WORKSPACE_NAME>-engine -n <WORKSPACE_NAME> \
  azure.workload.identity/client-id=<client-id> --overwrite
kubectl label sa <WORKSPACE_NAME>-engine -n <WORKSPACE_NAME> \
  azure.workload.identity/use=true --overwrite
kubectl rollout restart deployment -n <WORKSPACE_NAME> -l app=engine

Image pull errors

Symptom: Pods stuck in ImagePullBackOff or ErrImagePull.

kubectl describe pod <failing-pod> -n <WORKSPACE_NAME> | grep -A5 Events

Diagnose by which registry-access option you chose:

Option A (token): verify the pull secret exists and is the right type:
kubectl get secret e6data-acr-pull -n <WORKSPACE_NAME> -o jsonpath='{.type}' # Should output: kubernetes.io/dockerconfigjson
If missing or wrong, recreate it - see Configure registry, Kubernetes, and networking.
Option B (Managed Identity): confirm with e6data that the AcrPull role assignment is applied and propagated (up to 5 minutes). Share your kubelet identity object ID with e6data support if pulls still fail.

Workspace namespace stuck in Terminating

kubectl get namespace <WORKSPACE_NAME> -o json | jq .status

Look for NamespaceFinalizersRemaining. List remaining resources, then clear the stuck finalizer (usually an e6data custom resource):

kubectl patch <resource-type> <name> -n <WORKSPACE_NAME> \
  --type='merge' -p '{"metadata":{"finalizers":[]}}'

The namespace finishes terminating in 30–60 seconds after finalizers clear.

Operator pod crash-looping

kubectl logs deployment/e6operator -n e6operator -c manager --tail=100

Cause	Fix
CRDs from an older version	Re-apply the CRDs: `kubectl apply --server-side --force-conflicts -f ./e6-operator-crds/crds/`
Operator can’t reach the Kubernetes API	Check NetworkPolicy resources and egress rules
Operator can’t pull its image	See “Image pull errors” above
Operator panics on a malformed CR	Capture the panic stack from logs and contact e6data support

When to contact e6data support

Provide your AKS cluster name, resource group, and region; workspace and tenant name; the output of kubectl get pods -n <WORKSPACE_NAME> and kubectl get pods -n e6operator; relevant log excerpts; and what you’ve already tried. For production incidents, use the support channel in your e6data contract.

Platform

Get started

Deploy a workspace

Manage workspaces

Identity and access

Tokens and support access

Operate

Automate

Security and trust

Reference

Metadata service pods not appearing after deploy

Metadata service pods stuck in Pending

Load balancer not getting an external IP

Engine service account missing Workload Identity annotation

Image pull errors

Workspace namespace stuck in Terminating

Operator pod crash-looping

When to contact e6data support

See also

​Metadata service pods not appearing after deploy

​Metadata service pods stuck in Pending

​Load balancer not getting an external IP

​Engine service account missing Workload Identity annotation

​Image pull errors

​Workspace namespace stuck in Terminating

​Operator pod crash-looping

​When to contact e6data support

​See also

Metadata service pods not appearing after deploy

Metadata service pods stuck in Pending

Load balancer not getting an external IP

Engine service account missing Workload Identity annotation

Image pull errors

Workspace namespace stuck in Terminating

Operator pod crash-looping

When to contact e6data support

See also