Troubleshooting

Diagnose common issues in an In-VPC e6data deployment - scheduling, image pulls, the operator, reconciliation, Pod Identity, load balancers, and monitoring.

Pods stuck in Pending

kubectl describe pod <pod-name> -n <namespace> | tail -30

Common causes: no matching nodes (check taints and tolerations), insufficient resources, or the NodePool not creating nodes. If nodes aren't being provisioned, check Karpenter:

kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter --tail=50

Image pull errors

kubectl get secret -n ${NAMESPACE}

# Verify ECR access from your machine
aws ecr get-login-password --region ${REGION} | \
  docker login --username AWS --password-stdin <ECR_REPO>

kubectl describe pod <pod-name> -n ${NAMESPACE} | grep -A5 "Events:"

e6-operator not starting

kubectl logs -n e6operator -l app.kubernetes.io/name=e6-operator --tail=100
kubectl get endpoints e6operator-webhook-service -n e6operator
kubectl get crds | grep e6data.io

NamespaceConfig or QueryRouter stuck in Error

kubectl get namespaceconfig ${WORKSPACE_NAME}-nsc -n ${NAMESPACE} -o yaml | grep -A10 status:
kubectl get queryrouter ${WORKSPACE_NAME}-qr -n ${NAMESPACE} -o yaml | grep -A10 status:

kubectl logs -n e6operator -l app.kubernetes.io/name=e6-operator --tail=100 | grep -i error

Pod Identity issues

aws eks list-pod-identity-associations --cluster-name ${CLUSTER_NAME}-eks --region ${REGION}

aws eks describe-pod-identity-association \
  --cluster-name ${CLUSTER_NAME}-eks --association-id <association-id> --region ${REGION}

# Verify the role trust policy allows pods.eks.amazonaws.com
aws iam get-role --role-name ${WORKSPACE_NAME}-engine-role \
  --query 'Role.AssumeRolePolicyDocument' --output json

LoadBalancer not getting an external IP

kubectl logs -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller --tail=50
kubectl describe svc ${WORKSPACE_NAME}-qr-envoy-external -n ${NAMESPACE}

# Verify subnet tags (required by the ALB Controller)
aws ec2 describe-subnets --subnet-ids ${PRIVATE_SUBNET_1} ${PRIVATE_SUBNET_2} \
  --query 'Subnets[].Tags[?Key==`kubernetes.io/role/internal-elb`]'

Monitoring not sending data

kubectl get monitoringservices -n ${NAMESPACE} -o yaml | grep -A20 status:

kubectl logs -n ${NAMESPACE} -l app=o11y-collector -c vector --tail=50
kubectl logs -n ${NAMESPACE} -l app=o11y-collector -c metrics-bridge --tail=50

# Verify GreptimeDB connectivity from a pod
kubectl run test-curl --rm -it --restart=Never --image=curlimages/curl -- \
  curl -v ${GREPTIME_ENDPOINT}/health

Support

For deployment assistance, contact e6data support at [email protected], or reach your e6data account team for your dedicated support channel.