> ## Documentation Index
> Fetch the complete documentation index at: https://docs.e6data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure registry, VPC, EKS, and networking

> Build the AWS infrastructure (VPC, IAM, EKS, networking, ECR access) and install the Kubernetes platform components for an In-VPC e6data deployment.

This page builds the AWS infrastructure for an In-VPC deployment and installs the Kubernetes platform components that the e6data workspace runs on. Run these steps with the AWS CLI, `kubectl`, and Helm.

## Infrastructure requirements

Review these requirements before you begin. They apply whether you create a new VPC (Part 1, Step 1) or reuse an existing one.

### VPC options

You can either create a new VPC or use an existing one:

| Option | When to use |
| - | - |
| **New VPC** | Recommended for an isolated e6data deployment. Follow Step 1 in Part 1 below. |
| **Existing VPC** | Use when e6data must share a network with other workloads. Skip Step 1 and confirm your VPC meets the requirements below. |

### VPC and subnet CIDR ranges

| Resource | Minimum | Recommended | Notes |
| - | - | - | - |
| **VPC CIDR** | /20 (4,096 IPs) | /16 (65,536 IPs) | Must accommodate EKS nodes, pods, and services. |
| **Private subnets** | /22 per subnet (1,024 IPs) | /20 per subnet (4,096 IPs) | e6data engine nodes run here. |
| **Public subnets** | /24 per subnet (256 IPs) | /24 per subnet | NAT Gateway and load balancers only. |

Each e6data engine pod consumes one IP address. For large workloads with 100+ concurrent pods, make sure private subnets have enough IP space.

### Availability zones

| Requirement | Minimum | Recommended |
| - | - | - |
| **Number of AZs** | 2 | 3 |
| **Private subnets** | 1 per AZ | 1 per AZ |
| **Public subnets** | 1 per AZ | 1 per AZ |

e6data spreads workloads across AZs for high availability. Tag subnets so EKS, the load balancer controller, and Karpenter can discover them:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Private subnets (internal load balancers)
kubernetes.io/role/internal-elb = 1
# Public subnets (internet-facing load balancers)
kubernetes.io/role/elb = 1
# All subnets (EKS auto-discovery)
kubernetes.io/cluster/<cluster-name> = shared
# Private subnets (Karpenter node provisioning)
karpenter.sh/discovery = <cluster-name>-eks
```

### Service quotas

Verify these quotas before deploying. Request increases early - they can take 24–48 hours.

| Service | Quota | Minimum required |
| - | - | - |
| EC2 | Running On-Demand instances | 100 vCPUs |
| EC2 | Running Spot instances | 500 vCPUs |
| EC2 | EC2-VPC Elastic IPs | 5 |
| EKS | Clusters per Region | 1+ available |
| VPC | NAT Gateways per AZ | 1+ available |
| VPC | VPCs per Region | 1+ available |

### Instance types

| Component | Instance families | Architecture | Notes |
| - | - | - | - |
| System nodes | t3, m5 | x86\_64 (amd64) | EKS managed node group for system pods. |
| e6-operator | t3, t4g | amd64 or arm64 | Operator controller pods. |
| Engine nodes | r6g, r6gd, r7g, r7gd, m7g, c7g | arm64 (Graviton) | Query engine pods, provisioned by Karpenter. |

The e6data engine is optimized for AWS Graviton (arm64) instances for better price-performance. Confirm your Region has capacity for the r6g/r7g families.

### VPC CNI configuration

The Amazon VPC CNI plugin manages pod networking:

| Setting | Value | Rationale |
| - | - | - |
| `WARM_IP_TARGET` | 5 | Pre-allocates IPs for faster pod startup. |
| `MINIMUM_IP_TARGET` | 10 | Keeps a minimum number of IPs available per node. |
| `ENABLE_PREFIX_DELEGATION` | true | Increases IP capacity per node (recommended for large clusters). |

For clusters with 100+ pods, enable prefix delegation to avoid IP exhaustion. It assigns /28 prefixes (16 IPs) instead of individual IPs, allowing up to 110 pods per node on most instance types:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1
```

### Existing VPC checklist

If you reuse an existing VPC, verify:

* [ ] DNS hostnames enabled (`enableDnsHostnames: true`).
* [ ] DNS resolution enabled (`enableDnsSupport: true`).
* [ ] Private subnets route to a NAT Gateway for internet access.
* [ ] Private subnets tagged `kubernetes.io/role/internal-elb = 1`.
* [ ] Public subnets tagged `kubernetes.io/role/elb = 1`.
* [ ] Private subnets tagged `karpenter.sh/discovery = <cluster-name>-eks`.
* [ ] Cluster security group tagged `karpenter.sh/discovery = <cluster-name>-eks` (after EKS creation).
* [ ] Sufficient IP space in private subnets (minimum /22 per subnet).

## Part 1 - AWS infrastructure

### Step 1: Create the VPC

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Set your variables
export CLUSTER_NAME="e6data-cluster"
export REGION="us-east-1"
export VPC_CIDR="10.0.0.0/16"

# Create VPC
VPC_ID=$(aws ec2 create-vpc \
  --cidr-block $VPC_CIDR \
  --tag-specifications "ResourceType=vpc,Tags=[{Key=Name,Value=${CLUSTER_NAME}-vpc}]" \
  --query 'Vpc.VpcId' --output text --region $REGION)

echo "VPC_ID=$VPC_ID"

# Enable DNS hostnames
aws ec2 modify-vpc-attribute --vpc-id $VPC_ID --enable-dns-hostnames --region $REGION

# Create Internet Gateway
IGW_ID=$(aws ec2 create-internet-gateway \
  --tag-specifications "ResourceType=internet-gateway,Tags=[{Key=Name,Value=${CLUSTER_NAME}-igw}]" \
  --query 'InternetGateway.InternetGatewayId' --output text --region $REGION)

aws ec2 attach-internet-gateway --vpc-id $VPC_ID --internet-gateway-id $IGW_ID --region $REGION

# Get availability zones
AZS=$(aws ec2 describe-availability-zones --region $REGION --query 'AvailabilityZones[0:3].ZoneName' --output text)
AZ1=$(echo $AZS | awk '{print $1}')
AZ2=$(echo $AZS | awk '{print $2}')
AZ3=$(echo $AZS | awk '{print $3}')

# Create public subnets
PUBLIC_SUBNET_1=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.1.0/24 \
  --availability-zone $AZ1 \
  --tag-specifications "ResourceType=subnet,Tags=[{Key=Name,Value=${CLUSTER_NAME}-public-1},{Key=kubernetes.io/role/elb,Value=1}]" \
  --query 'Subnet.SubnetId' --output text --region $REGION)

PUBLIC_SUBNET_2=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.2.0/24 \
  --availability-zone $AZ2 \
  --tag-specifications "ResourceType=subnet,Tags=[{Key=Name,Value=${CLUSTER_NAME}-public-2},{Key=kubernetes.io/role/elb,Value=1}]" \
  --query 'Subnet.SubnetId' --output text --region $REGION)

# Create private subnets
PRIVATE_SUBNET_1=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.10.0/24 \
  --availability-zone $AZ1 \
  --tag-specifications "ResourceType=subnet,Tags=[{Key=Name,Value=${CLUSTER_NAME}-private-1},{Key=kubernetes.io/role/internal-elb,Value=1}]" \
  --query 'Subnet.SubnetId' --output text --region $REGION)

PRIVATE_SUBNET_2=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.11.0/24 \
  --availability-zone $AZ2 \
  --tag-specifications "ResourceType=subnet,Tags=[{Key=Name,Value=${CLUSTER_NAME}-private-2},{Key=kubernetes.io/role/internal-elb,Value=1}]" \
  --query 'Subnet.SubnetId' --output text --region $REGION)

# Create Elastic IP for NAT Gateway
EIP_ALLOC=$(aws ec2 allocate-address --domain vpc \
  --tag-specifications "ResourceType=elastic-ip,Tags=[{Key=Name,Value=${CLUSTER_NAME}-nat-eip}]" \
  --query 'AllocationId' --output text --region $REGION)

# Create NAT Gateway
NAT_GW=$(aws ec2 create-nat-gateway \
  --subnet-id $PUBLIC_SUBNET_1 \
  --allocation-id $EIP_ALLOC \
  --tag-specifications "ResourceType=natgateway,Tags=[{Key=Name,Value=${CLUSTER_NAME}-nat}]" \
  --query 'NatGateway.NatGatewayId' --output text --region $REGION)

echo "Waiting for NAT Gateway to become available..."
aws ec2 wait nat-gateway-available --nat-gateway-ids $NAT_GW --region $REGION

# Create route tables
PUBLIC_RT=$(aws ec2 create-route-table --vpc-id $VPC_ID \
  --tag-specifications "ResourceType=route-table,Tags=[{Key=Name,Value=${CLUSTER_NAME}-public-rt}]" \
  --query 'RouteTable.RouteTableId' --output text --region $REGION)

aws ec2 create-route --route-table-id $PUBLIC_RT --destination-cidr-block 0.0.0.0/0 \
  --gateway-id $IGW_ID --region $REGION

aws ec2 associate-route-table --route-table-id $PUBLIC_RT --subnet-id $PUBLIC_SUBNET_1 --region $REGION
aws ec2 associate-route-table --route-table-id $PUBLIC_RT --subnet-id $PUBLIC_SUBNET_2 --region $REGION

PRIVATE_RT=$(aws ec2 create-route-table --vpc-id $VPC_ID \
  --tag-specifications "ResourceType=route-table,Tags=[{Key=Name,Value=${CLUSTER_NAME}-private-rt}]" \
  --query 'RouteTable.RouteTableId' --output text --region $REGION)

aws ec2 create-route --route-table-id $PRIVATE_RT --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id $NAT_GW --region $REGION

aws ec2 associate-route-table --route-table-id $PRIVATE_RT --subnet-id $PRIVATE_SUBNET_1 --region $REGION
aws ec2 associate-route-table --route-table-id $PRIVATE_RT --subnet-id $PRIVATE_SUBNET_2 --region $REGION

# Create S3 VPC Endpoint (saves NAT costs for S3 traffic)
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --service-name com.amazonaws.${REGION}.s3 \
  --route-table-ids $PRIVATE_RT \
  --tag-specifications "ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=${CLUSTER_NAME}-s3-endpoint}]" \
  --region $REGION

echo "VPC setup complete!"
echo "VPC_ID=$VPC_ID"
echo "PUBLIC_SUBNETS=$PUBLIC_SUBNET_1,$PUBLIC_SUBNET_2"
echo "PRIVATE_SUBNETS=$PRIVATE_SUBNET_1,$PRIVATE_SUBNET_2"

# Tag private subnets for Karpenter discovery
aws ec2 create-tags --resources $PRIVATE_SUBNET_1 $PRIVATE_SUBNET_2 \
  --tags Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}-eks \
  --region $REGION

# Note: the cluster security group is tagged for Karpenter after the EKS cluster is created (Step 3).
```

### Step 2: Create IAM roles

**EKS cluster role:**

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat > eks-cluster-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "eks.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role \
  --role-name ${CLUSTER_NAME}-eks-cluster-role \
  --assume-role-policy-document file://eks-cluster-trust-policy.json

aws iam attach-role-policy \
  --role-name ${CLUSTER_NAME}-eks-cluster-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
```

**EKS node role:**

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat > eks-node-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "ec2.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role \
  --role-name ${CLUSTER_NAME}-eks-node-role \
  --assume-role-policy-document file://eks-node-trust-policy.json

aws iam attach-role-policy --role-name ${CLUSTER_NAME}-eks-node-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
aws iam attach-role-policy --role-name ${CLUSTER_NAME}-eks-node-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
aws iam attach-role-policy --role-name ${CLUSTER_NAME}-eks-node-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
aws iam attach-role-policy --role-name ${CLUSTER_NAME}-eks-node-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

aws iam create-instance-profile --instance-profile-name ${CLUSTER_NAME}-eks-node-profile
aws iam add-role-to-instance-profile \
  --instance-profile-name ${CLUSTER_NAME}-eks-node-profile \
  --role-name ${CLUSTER_NAME}-eks-node-role
```

**Karpenter role:**

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

cat > karpenter-trust-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "pods.eks.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
  ]
}
EOF

aws iam create-role \
  --role-name ${CLUSTER_NAME}-karpenter-role \
  --assume-role-policy-document file://karpenter-trust-policy.json

cat > karpenter-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Karpenter",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateFleet", "ec2:CreateLaunchTemplate", "ec2:CreateTags",
        "ec2:DeleteLaunchTemplate", "ec2:DescribeAvailabilityZones",
        "ec2:DescribeImages", "ec2:DescribeInstances",
        "ec2:DescribeInstanceTypeOfferings", "ec2:DescribeInstanceTypes",
        "ec2:DescribeLaunchTemplates", "ec2:DescribeSecurityGroups",
        "ec2:DescribeSpotPriceHistory", "ec2:DescribeSubnets",
        "ec2:RunInstances", "ec2:TerminateInstances",
        "iam:PassRole", "pricing:GetProducts", "ssm:GetParameter"
      ],
      "Resource": "*"
    },
    {
      "Sid": "SQS",
      "Effect": "Allow",
      "Action": ["sqs:DeleteMessage", "sqs:GetQueueAttributes", "sqs:GetQueueUrl", "sqs:ReceiveMessage"],
      "Resource": "arn:aws:sqs:${REGION}:${AWS_ACCOUNT_ID}:${CLUSTER_NAME}-karpenter"
    }
  ]
}
EOF

aws iam put-role-policy \
  --role-name ${CLUSTER_NAME}-karpenter-role \
  --policy-name KarpenterPolicy \
  --policy-document file://karpenter-policy.json

aws iam create-instance-profile --instance-profile-name ${CLUSTER_NAME}-karpenter
aws iam add-role-to-instance-profile \
  --instance-profile-name ${CLUSTER_NAME}-karpenter \
  --role-name ${CLUSTER_NAME}-eks-node-role
```

**ALB Controller role:**

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws iam create-role \
  --role-name ${CLUSTER_NAME}-alb-controller-role \
  --assume-role-policy-document file://karpenter-trust-policy.json

curl -o alb-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json

aws iam put-role-policy \
  --role-name ${CLUSTER_NAME}-alb-controller-role \
  --policy-name ALBControllerPolicy \
  --policy-document file://alb-policy.json
```

**EBS CSI driver role:**

The EBS CSI driver addon uses EKS Pod Identity for its IAM permissions:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat > ebs-csi-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "pods.eks.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
  ]
}
EOF

aws iam create-role \
  --role-name ${CLUSTER_NAME}-ebs-csi-driver-role \
  --assume-role-policy-document file://ebs-csi-trust-policy.json

aws iam attach-role-policy \
  --role-name ${CLUSTER_NAME}-ebs-csi-driver-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy
```

### Step 3: Create the EKS cluster

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws eks create-cluster \
  --name ${CLUSTER_NAME}-eks \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-eks-cluster-role \
  --resources-vpc-config subnetIds=${PRIVATE_SUBNET_1},${PRIVATE_SUBNET_2},endpointPublicAccess=true,endpointPrivateAccess=true \
  --access-config authenticationMode=API_AND_CONFIG_MAP \
  --kubernetes-version 1.31 \
  --region $REGION

echo "Waiting for EKS cluster to become active (this takes ~10-15 minutes)..."
aws eks wait cluster-active --name ${CLUSTER_NAME}-eks --region $REGION

aws eks update-kubeconfig --name ${CLUSTER_NAME}-eks --region $REGION

CLUSTER_ENDPOINT=$(aws eks describe-cluster --name ${CLUSTER_NAME}-eks \
  --query 'cluster.endpoint' --output text --region $REGION)
OIDC_ISSUER=$(aws eks describe-cluster --name ${CLUSTER_NAME}-eks \
  --query 'cluster.identity.oidc.issuer' --output text --region $REGION)

# Tag the cluster security group for Karpenter discovery
CLUSTER_SG=$(aws eks describe-cluster --name ${CLUSTER_NAME}-eks \
  --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId' --output text --region $REGION)

aws ec2 create-tags --resources $CLUSTER_SG \
  --tags Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}-eks \
  --region $REGION
```

### Step 4: Create the OIDC provider and Pod Identity add-ons

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
eksctl utils associate-iam-oidc-provider \
  --cluster ${CLUSTER_NAME}-eks --region $REGION --approve

aws eks create-addon --cluster-name ${CLUSTER_NAME}-eks --addon-name eks-pod-identity-agent --region $REGION
aws eks create-addon --cluster-name ${CLUSTER_NAME}-eks --addon-name vpc-cni --region $REGION

# Install the EBS CSI driver with a Pod Identity association so it can manage EBS volumes
aws eks create-addon --cluster-name ${CLUSTER_NAME}-eks --addon-name aws-ebs-csi-driver \
  --pod-identity-associations "serviceAccount=ebs-csi-controller-sa,roleArn=arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-ebs-csi-driver-role" \
  --region $REGION
```

### Step 5: Create the system node group

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws eks create-nodegroup \
  --cluster-name ${CLUSTER_NAME}-eks \
  --nodegroup-name system-nodes \
  --scaling-config minSize=2,maxSize=4,desiredSize=2 \
  --instance-types t3.medium \
  --node-role arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-eks-node-role \
  --subnets $PRIVATE_SUBNET_1 $PRIVATE_SUBNET_2 \
  --labels role=system \
  --region $REGION

aws eks wait nodegroup-active \
  --cluster-name ${CLUSTER_NAME}-eks --nodegroup-name system-nodes --region $REGION
```

### Step 6: Create the S3 metadata bucket

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export WORKSPACE_NAME="my-workspace"  # Replace with your workspace name

# Note: us-east-1 is S3's default region and rejects LocationConstraint
if [ "$REGION" = "us-east-1" ]; then
  aws s3api create-bucket \
    --bucket e6-${WORKSPACE_NAME}-metadata \
    --region $REGION
else
  aws s3api create-bucket \
    --bucket e6-${WORKSPACE_NAME}-metadata \
    --region $REGION \
    --create-bucket-configuration LocationConstraint=$REGION
fi

aws s3api put-public-access-block \
  --bucket e6-${WORKSPACE_NAME}-metadata \
  --public-access-block-configuration \
  "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

aws s3api put-bucket-versioning \
  --bucket e6-${WORKSPACE_NAME}-metadata \
  --versioning-configuration Status=Enabled

aws s3api put-bucket-encryption \
  --bucket e6-${WORKSPACE_NAME}-metadata \
  --server-side-encryption-configuration '{
    "Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]
  }'

cat > bucket-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EnforceSSL",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata",
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata/*"
      ],
      "Condition": { "Bool": {"aws:SecureTransport": "false"} }
    }
  ]
}
EOF

aws s3api put-bucket-policy \
  --bucket e6-${WORKSPACE_NAME}-metadata \
  --policy file://bucket-policy.json
```

<Warning>
  Do not enable S3 Object Lock on this bucket - it can break metadata operations.
</Warning>

### Step 7: Create workspace IAM roles

Create the **engine**, **console**, and **app** roles. All three trust `pods.eks.amazonaws.com`:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat > engine-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "pods.eks.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
  ]
}
EOF

# Engine role (query engine and Laminar)
aws iam create-role --role-name ${WORKSPACE_NAME}-engine-role \
  --assume-role-policy-document file://engine-trust-policy.json

cat > engine-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3MetadataAccess",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:AbortMultipartUpload"],
      "Resource": [
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata",
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata/*"
      ]
    },
    {
      "Sid": "CrossAccountAccess",
      "Effect": "Allow",
      "Action": ["sts:AssumeRole", "sts:TagSession"],
      "Resource": "*"
    }
  ]
}
EOF

aws iam put-role-policy --role-name ${WORKSPACE_NAME}-engine-role \
  --policy-name EnginePolicy --policy-document file://engine-policy.json

# Console role (UI backend)
aws iam create-role --role-name ${WORKSPACE_NAME}-console-role \
  --assume-role-policy-document file://engine-trust-policy.json

cat > console-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3MetadataAccess",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:AbortMultipartUpload"],
      "Resource": [
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata",
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata/*"
      ]
    },
    {
      "Sid": "VPCEndpointDiscovery",
      "Effect": "Allow",
      "Action": ["ec2:DescribeVpcEndpoints", "ec2:DescribeInstances"],
      "Resource": "*"
    },
    {
      "Sid": "EKSPodIdentityLookup",
      "Effect": "Allow",
      "Action": ["eks:DescribeCluster", "eks:ListPodIdentityAssociations", "eks:DescribePodIdentityAssociation"],
      "Resource": [
        "arn:aws:eks:${REGION}:${AWS_ACCOUNT_ID}:cluster/${CLUSTER_NAME}-eks",
        "arn:aws:eks:${REGION}:${AWS_ACCOUNT_ID}:podidentityassociation/${CLUSTER_NAME}-eks/*"
      ]
    }
  ]
}
EOF

aws iam put-role-policy --role-name ${WORKSPACE_NAME}-console-role \
  --policy-name ConsolePolicy --policy-document file://console-policy.json

# App role (Copilot and Metriq)
aws iam create-role --role-name ${WORKSPACE_NAME}-app-role \
  --assume-role-policy-document file://engine-trust-policy.json

cat > app-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3MetadataAccess",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:AbortMultipartUpload"],
      "Resource": [
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata",
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata/*"
      ]
    },
    {
      "Sid": "SecretsManagerAccess",
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
      "Resource": "arn:aws:secretsmanager:${REGION}:${AWS_ACCOUNT_ID}:secret:copilot-metriq-secret-*"
    }
  ]
}
EOF

aws iam put-role-policy --role-name ${WORKSPACE_NAME}-app-role \
  --policy-name AppPolicy --policy-document file://app-policy.json
```

### Step 8: Create Pod Identity associations

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
kubectl create namespace $WORKSPACE_NAME

aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace $WORKSPACE_NAME --service-account ${WORKSPACE_NAME}-engine \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${WORKSPACE_NAME}-engine-role --region $REGION

aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace $WORKSPACE_NAME --service-account ${WORKSPACE_NAME}-console \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${WORKSPACE_NAME}-console-role --region $REGION

aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace $WORKSPACE_NAME --service-account ${WORKSPACE_NAME}-copilot \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${WORKSPACE_NAME}-app-role --region $REGION

aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace $WORKSPACE_NAME --service-account ${WORKSPACE_NAME}-metriq \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${WORKSPACE_NAME}-app-role --region $REGION

aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace $WORKSPACE_NAME --service-account ${WORKSPACE_NAME}-laminar \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${WORKSPACE_NAME}-engine-role --region $REGION
```

### Step 9: Create the Karpenter SQS queue

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws sqs create-queue --queue-name ${CLUSTER_NAME}-karpenter --region $REGION

QUEUE_ARN=$(aws sqs get-queue-attributes \
  --queue-url https://sqs.${REGION}.amazonaws.com/${AWS_ACCOUNT_ID}/${CLUSTER_NAME}-karpenter \
  --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)

cat > sqs-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "events.amazonaws.com"},
      "Action": "sqs:SendMessage",
      "Resource": "${QUEUE_ARN}"
    }
  ]
}
EOF

aws sqs set-queue-attributes \
  --queue-url https://sqs.${REGION}.amazonaws.com/${AWS_ACCOUNT_ID}/${CLUSTER_NAME}-karpenter \
  --attributes Policy="$(cat sqs-policy.json | jq -c .)"
```

## Part 2 - Kubernetes platform components

<Info>
  e6data uses the `helm template | kubectl apply --server-side` pattern instead of `helm install` to avoid Helm's 1 MB release Secret limit and ensure clean field-manager handling during upgrades.
</Info>

### Step 1: Install Karpenter CRDs

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export KARPENTER_VERSION="1.8.1"
tmpdir=$(mktemp -d)

helm template karpenter-crd \
  oci://public.ecr.aws/karpenter/karpenter-crd \
  --version "${KARPENTER_VERSION}" --include-crds \
  > "${tmpdir}/karpenter-crds.yaml"

kubectl apply --server-side --force-conflicts -f "${tmpdir}/karpenter-crds.yaml"
kubectl wait --for=condition=Established crd/nodepools.karpenter.sh crd/nodeclaims.karpenter.sh --timeout=60s
kubectl get crds | grep karpenter
```

### Step 2: Install the Karpenter controller

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
CLUSTER_ENDPOINT=$(aws eks describe-cluster --name ${CLUSTER_NAME}-eks \
  --query 'cluster.endpoint' --output text --region $REGION)

cat > karpenter-values.yaml << EOF
settings:
  clusterName: ${CLUSTER_NAME}-eks
  clusterEndpoint: ${CLUSTER_ENDPOINT}
  interruptionQueue: ${CLUSTER_NAME}-karpenter
replicas: 2
controller:
  resources:
    requests: { cpu: 200m, memory: 256Mi }
    limits: { cpu: 1, memory: 1Gi }
EOF

helm template karpenter \
  oci://public.ecr.aws/karpenter/karpenter \
  --version "${KARPENTER_VERSION}" --namespace kube-system --skip-crds \
  -f karpenter-values.yaml \
  | kubectl apply --server-side --force-conflicts -f -

# Link the karpenter service account to the IAM role created in Part 1, Step 2
aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace kube-system --service-account karpenter \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter-role --region $REGION

# Restart Karpenter so it picks up the new credentials
kubectl rollout restart deployment/karpenter -n kube-system

kubectl rollout status deployment/karpenter -n kube-system --timeout=120s
kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter
```

### Step 3: Install the AWS Load Balancer Controller

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export ALB_CONTROLLER_VERSION="1.8.1"

helm repo add eks https://aws.github.io/eks-charts
helm repo update eks

cat > alb-values.yaml << EOF
clusterName: ${CLUSTER_NAME}-eks
region: ${REGION}
vpcId: ${VPC_ID}
EOF

helm upgrade --install aws-load-balancer-controller \
  eks/aws-load-balancer-controller \
  --version "${ALB_CONTROLLER_VERSION}" --namespace kube-system \
  -f alb-values.yaml

# Link the aws-load-balancer-controller service account to the IAM role created in Part 1, Step 2
aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace kube-system --service-account aws-load-balancer-controller \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-alb-controller-role --region $REGION

# Restart the controller so it picks up the new credentials
kubectl rollout restart deployment/aws-load-balancer-controller -n kube-system

kubectl rollout status deployment/aws-load-balancer-controller -n kube-system --timeout=120s
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller
```

### Step 4: Create the e6-operator NodePool

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat << EOF | kubectl apply -f -
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: e6operator
spec:
  amiSelectorTerms:
    - alias: al2023@latest
  instanceProfile: ${CLUSTER_NAME}-karpenter
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: ${CLUSTER_NAME}-eks
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: ${CLUSTER_NAME}-eks
  tags:
    Name: ${CLUSTER_NAME}-e6operator
EOF

cat << EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: e6operator
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: e6operator
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: [t3.medium, t3.large, t4g.medium, t4g.large]
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["${AZ1}", "${AZ2}"]
      taints:
        - key: e6operator
          value: "true"
          effect: NoSchedule
  limits:
    cpu: 100
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30s
EOF

kubectl get nodepools.karpenter.sh e6operator
kubectl get ec2nodeclasses.karpenter.k8s.aws e6operator
```

### Step 5: Install cert-manager

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export CERT_MANAGER_VERSION="v1.19.2"

helm upgrade --install cert-manager \
  oci://quay.io/jetstack/charts/cert-manager \
  --version "${CERT_MANAGER_VERSION}" --namespace cert-manager --create-namespace \
  --set crds.enabled=true --set webhook.timeoutSeconds=4 --timeout 10m --wait

kubectl rollout status deployment/cert-manager -n cert-manager --timeout=120s
kubectl rollout status deployment/cert-manager-webhook -n cert-manager --timeout=120s
kubectl rollout status deployment/cert-manager-cainjector -n cert-manager --timeout=120s
kubectl get pods -n cert-manager
```

### Step 6: Install the e6-operator CRDs

e6data provides the CRD files. Apply them with server-side apply:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export E6_OPERATOR_CRDS_PATH="./e6-operator-crds"

kubectl apply --server-side --force-conflicts -f "${E6_OPERATOR_CRDS_PATH}/"
kubectl wait --for=condition=Established crd/namespaceconfigs.e6data.io --timeout=120s
kubectl get crds | grep e6data.io
```

All CRDs from the provided archive are installed - expect 20+ resources with the `e6data.io` suffix, including:

```
e6catalogs.e6data.io
metadataservices.e6data.io
monitoringservices.e6data.io
namespaceconfigs.e6data.io
queryrouters.e6data.io
queryservices.e6data.io
```

### Step 7: Create the gp3 StorageClass

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat << 'EOF' | kubectl apply --server-side --force-conflicts -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  fsType: ext4
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF

kubectl get storageclass gp3
```

### Step 8: Install the e6-operator controller

e6data provides the operator Helm chart. Install it with `helm template | kubectl apply`:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export E6_OPERATOR_IMAGE_REPO="<ECR_REPO_PROVIDED_BY_E6DATA>"
export E6_OPERATOR_IMAGE_TAG="<VERSION_PROVIDED_BY_E6DATA>"
export E6_OPERATOR_CHART="./e6-operator-chart"

kubectl create namespace e6operator --dry-run=client -o yaml | kubectl apply -f -

cat > operator-values.yaml << EOF
image:
  repository: ${E6_OPERATOR_IMAGE_REPO}
  tag: ${E6_OPERATOR_IMAGE_TAG}
  pullPolicy: IfNotPresent
replicaCount: 2
resources:
  requests: { cpu: 200m, memory: 256Mi }
  limits: { cpu: 1, memory: 1Gi }
logLevel: info
tolerations:
  - key: e6operator
    operator: Equal
    value: "true"
    effect: NoSchedule
EOF

helm template e6operator "${E6_OPERATOR_CHART}" \
  --namespace e6operator -f operator-values.yaml \
  | kubectl apply --server-side --force-conflicts -f -

kubectl rollout status deployment/e6operator -n e6operator --timeout=900s
kubectl get pods -n e6operator
```

## Next

Continue to [Deploy workspace and e6data](/platform/deploy/aws-cloudprem/deploy-workspace).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.