> For the complete documentation index, see [llms.txt](https://docs.e6data.com/query-engine/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.e6data.com/query-engine/guides/deployment/azure-in-vpc/troubleshooting.md).

# Troubleshooting

Common issues when deploying or running e6data In Your VPC on Azure, with diagnostic commands and fixes.

Common issues during In-VPC deployment and operation on Azure, with the diagnostic commands and fixes for each.

## Metadata service pods not appearing after deploy

**Symptom:** You applied the `NamespaceConfig`, but `mds-schema-*` and `mds-storage-*` pods don't appear after a few minutes.

The operator waits 60 seconds after the NamespaceConfig is created before provisioning metadata services. If pods haven't appeared after 3–4 minutes total:

```bash
kubectl logs deployment/e6operator -n e6operator -c manager | grep mdsAutoCreation
```

If you see `unknown field "status.mdsAutoCreation"`, the CRDs need updating - re-apply them (Part 2 Step 2.3):

```bash
kubectl apply --server-side --force-conflicts -f ./e6-operator-crds/crds/
```

## Metadata service pods stuck in Pending

```bash
kubectl describe pod <mds-pod> -n <WORKSPACE_NAME> | tail -20
```

| Event                           | Fix                                                                    |
| ------------------------------- | ---------------------------------------------------------------------- |
| `node(s) had untolerated taint` | Add tolerations to your node pool, or remove the pod's nodeSelector    |
| `insufficient cpu/memory`       | Scale up your node pool or attach a larger SKU                         |
| `volume node affinity conflict` | A PVC is bound to a zone that doesn't match where the pod is scheduled |

## Load balancer not getting an external IP

**Symptom:** The `LoadBalancer` service stays in Pending and never gets an external IP.

```bash
kubectl describe svc <WORKSPACE_NAME>-qr-envoy-external -n <WORKSPACE_NAME>
```

The AKS cluster identity needs **Network Contributor** on the VNet to provision load balancers (this is granted in Part 1, Step 4). If the LB stays pending, confirm the assignment exists, then create it if missing:

```bash
CLUSTER_IDENTITY=$(az aks show --name <CLUSTER_NAME> --resource-group <RESOURCE_GROUP> \
  --query "identity.principalId" -o tsv)

az role assignment create \
  --assignee-object-id $CLUSTER_IDENTITY \
  --assignee-principal-type ServicePrincipal \
  --role "Network Contributor" \
  --scope "/subscriptions/<SUB_ID>/resourceGroups/<RG>/providers/Microsoft.Network/virtualNetworks/<VNET_NAME>"
```

Role assignment can take a few minutes to propagate. With the public default, the assigned `EXTERNAL-IP` is a public address; if you set `azure-load-balancer-internal: "true"`, it is a private `10.x` VNet address instead.

## Engine service account missing Workload Identity annotation

**Symptom:** Engine pods can't authenticate to Azure storage; logs show `AccessDenied` or `NoCredentialsError`.

The operator may recreate the engine service account during reconciliation without the Workload Identity annotation. Re-annotate and restart:

```bash
kubectl annotate sa <WORKSPACE_NAME>-engine -n <WORKSPACE_NAME> \
  azure.workload.identity/client-id=<client-id> --overwrite
kubectl label sa <WORKSPACE_NAME>-engine -n <WORKSPACE_NAME> \
  azure.workload.identity/use=true --overwrite
kubectl rollout restart deployment -n <WORKSPACE_NAME> -l app=engine
```

## Image pull errors

**Symptom:** Pods stuck in `ImagePullBackOff` or `ErrImagePull`.

```bash
kubectl describe pod <failing-pod> -n <WORKSPACE_NAME> | grep -A5 Events
```

Diagnose by which registry-access option you chose:

* **Option A (token):** verify the pull secret exists and is the right type:

  ```bash
  kubectl get secret e6data-acr-pull -n <WORKSPACE_NAME> -o jsonpath='{.type}'
  # Should output: kubernetes.io/dockerconfigjson
  ```

  If missing or wrong, recreate it - see [Configure registry, Kubernetes, and networking](/query-engine/guides/deployment/azure-in-vpc/configure-registry-kubernetes-networking.md).
* **Option B (Managed Identity):** confirm with e6data that the `AcrPull` role assignment is applied and propagated (up to 5 minutes). Share your kubelet identity object ID with e6data support if pulls still fail.

## Workspace namespace stuck in Terminating

```bash
kubectl get namespace <WORKSPACE_NAME> -o json | jq .status
```

Look for `NamespaceFinalizersRemaining`. List remaining resources, then clear the stuck finalizer (usually an e6data custom resource):

```bash
kubectl patch <resource-type> <name> -n <WORKSPACE_NAME> \
  --type='merge' -p '{"metadata":{"finalizers":[]}}'
```

The namespace finishes terminating in 30–60 seconds after finalizers clear.

## Operator pod crash-looping

```bash
kubectl logs deployment/e6operator -n e6operator -c manager --tail=100
```

| Cause                                   | Fix                                                                                            |
| --------------------------------------- | ---------------------------------------------------------------------------------------------- |
| CRDs from an older version              | Re-apply the CRDs: `kubectl apply --server-side --force-conflicts -f ./e6-operator-crds/crds/` |
| Operator can't reach the Kubernetes API | Check NetworkPolicy resources and egress rules                                                 |
| Operator can't pull its image           | See "Image pull errors" above                                                                  |
| Operator panics on a malformed CR       | Capture the panic stack from logs and contact e6data support                                   |

## When to contact e6data support

Provide your AKS cluster name, resource group, and region; workspace and tenant name; the output of `kubectl get pods -n <WORKSPACE_NAME>` and `kubectl get pods -n e6operator`; relevant log excerpts; and what you've already tried. For production incidents, use the support channel in your e6data contract.

## See also

* [Configure registry, Kubernetes, and networking](/query-engine/guides/deployment/azure-in-vpc/configure-registry-kubernetes-networking.md)
* [Deploy workspace and e6data](/query-engine/guides/deployment/azure-in-vpc/deploy-workspace-and-e6data.md)
* [Delete workspace](/query-engine/guides/deployment/azure-in-vpc/delete-workspace.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.e6data.com/query-engine/guides/deployment/azure-in-vpc/troubleshooting.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.