> ## Documentation Index
> Fetch the complete documentation index at: https://docs.e6data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Common issues when deploying or running e6data In Your VPC on Azure, with diagnostic commands and fixes.

Common issues during In-VPC deployment and operation on Azure, with the diagnostic commands and fixes for each.

## Metadata service pods not appearing after deploy

**Symptom:** You applied the `NamespaceConfig`, but `mds-schema-*` and `mds-storage-*` pods don't appear after a few minutes.

The operator waits 60 seconds after the NamespaceConfig is created before provisioning metadata services. If pods haven't appeared after 3–4 minutes total:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
kubectl logs deployment/e6operator -n e6operator -c manager | grep mdsAutoCreation
```

If you see `unknown field "status.mdsAutoCreation"`, the CRDs need updating - re-apply them (Part 2 Step 2.3):

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
kubectl apply --server-side --force-conflicts -f ./e6-operator-crds/crds/
```

## Metadata service pods stuck in Pending

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
kubectl describe pod <mds-pod> -n <WORKSPACE_NAME> | tail -20
```

| Event | Fix |
| - | - |
| `node(s) had untolerated taint` | Add tolerations to your node pool, or remove the pod's nodeSelector |
| `insufficient cpu/memory` | Scale up your node pool or attach a larger SKU |
| `volume node affinity conflict` | A PVC is bound to a zone that doesn't match where the pod is scheduled |

## Load balancer not getting an external IP

**Symptom:** The `LoadBalancer` service stays in Pending and never gets an external IP.

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
kubectl describe svc <WORKSPACE_NAME>-qr-envoy-external -n <WORKSPACE_NAME>
```

The AKS cluster identity needs **Network Contributor** on the VNet to provision load balancers (this is granted in Part 1, Step 4). If the LB stays pending, confirm the assignment exists, then create it if missing:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
CLUSTER_IDENTITY=$(az aks show --name <CLUSTER_NAME> --resource-group <RESOURCE_GROUP> \
  --query "identity.principalId" -o tsv)

az role assignment create \
  --assignee-object-id $CLUSTER_IDENTITY \
  --assignee-principal-type ServicePrincipal \
  --role "Network Contributor" \
  --scope "/subscriptions/<SUB_ID>/resourceGroups/<RG>/providers/Microsoft.Network/virtualNetworks/<VNET_NAME>"
```

Role assignment can take a few minutes to propagate. With the public default, the assigned `EXTERNAL-IP` is a public address; if you set `azure-load-balancer-internal: "true"`, it is a private `10.x` VNet address instead.

## Engine service account missing Workload Identity annotation

**Symptom:** Engine pods can't authenticate to Azure storage; logs show `AccessDenied` or `NoCredentialsError`.

The operator may recreate the engine service account during reconciliation without the Workload Identity annotation. Re-annotate and restart:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
kubectl annotate sa <WORKSPACE_NAME>-engine -n <WORKSPACE_NAME> \
  azure.workload.identity/client-id=<client-id> --overwrite
kubectl label sa <WORKSPACE_NAME>-engine -n <WORKSPACE_NAME> \
  azure.workload.identity/use=true --overwrite
kubectl rollout restart deployment -n <WORKSPACE_NAME> -l app=engine
```

## Image pull errors

**Symptom:** Pods stuck in `ImagePullBackOff` or `ErrImagePull`.

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
kubectl describe pod <failing-pod> -n <WORKSPACE_NAME> | grep -A5 Events
```

Diagnose by which registry-access option you chose:

* **Option A (token):** verify the pull secret exists and is the right type:

  ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
  kubectl get secret e6data-acr-pull -n <WORKSPACE_NAME> -o jsonpath='{.type}'
  # Should output: kubernetes.io/dockerconfigjson
  ```

  If missing or wrong, recreate it - see [Configure registry, Kubernetes, and networking](/platform/deploy/azure-cloudprem/configure-registry-kubernetes-networking).
* **Option B (Managed Identity):** confirm with e6data that the `AcrPull` role assignment is applied and propagated (up to 5 minutes). Share your kubelet identity object ID with e6data support if pulls still fail.

## Workspace namespace stuck in Terminating

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
kubectl get namespace <WORKSPACE_NAME> -o json | jq .status
```

Look for `NamespaceFinalizersRemaining`. List remaining resources, then clear the stuck finalizer (usually an e6data custom resource):

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
kubectl patch <resource-type> <name> -n <WORKSPACE_NAME> \
  --type='merge' -p '{"metadata":{"finalizers":[]}}'
```

The namespace finishes terminating in 30–60 seconds after finalizers clear.

## Operator pod crash-looping

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
kubectl logs deployment/e6operator -n e6operator -c manager --tail=100
```

| Cause | Fix |
| - | - |
| CRDs from an older version | Re-apply the CRDs: `kubectl apply --server-side --force-conflicts -f ./e6-operator-crds/crds/` |
| Operator can't reach the Kubernetes API | Check NetworkPolicy resources and egress rules |
| Operator can't pull its image | See "Image pull errors" above |
| Operator panics on a malformed CR | Capture the panic stack from logs and contact e6data support |

## When to contact e6data support

Provide your AKS cluster name, resource group, and region; workspace and tenant name; the output of `kubectl get pods -n <WORKSPACE_NAME>` and `kubectl get pods -n e6operator`; relevant log excerpts; and what you've already tried. For production incidents, use the support channel in your e6data contract.

## See also

* [Configure registry, Kubernetes, and networking](/platform/deploy/azure-cloudprem/configure-registry-kubernetes-networking)
* [Deploy workspace and e6data](/platform/deploy/azure-cloudprem/deploy-workspace)
* [Delete workspace](/platform/deploy/azure-cloudprem/delete-workspace)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.