Configure registry, VPC, EKS, and networking

Build the AWS infrastructure (VPC, IAM, EKS, networking, ECR access) and install the Kubernetes platform components for an In-VPC e6data deployment.

This page builds the AWS infrastructure for an In-VPC deployment and installs the Kubernetes platform components that the e6data workspace runs on. Run these steps with the AWS CLI, kubectl, and Helm.

Infrastructure requirements

Review these requirements before you begin. They apply whether you create a new VPC (Part 1, Step 1) or reuse an existing one.

VPC options

You can either create a new VPC or use an existing one:

Option	When to use
New VPC	Recommended for an isolated e6data deployment. Follow Step 1 in Part 1 below.
Existing VPC	Use when e6data must share a network with other workloads. Skip Step 1 and confirm your VPC meets the requirements below.

VPC and subnet CIDR ranges

Resource	Minimum	Recommended	Notes
VPC CIDR	/20 (4,096 IPs)	/16 (65,536 IPs)	Must accommodate EKS nodes, pods, and services.
Private subnets	/22 per subnet (1,024 IPs)	/20 per subnet (4,096 IPs)	e6data engine nodes run here.
Public subnets	/24 per subnet (256 IPs)	/24 per subnet	NAT Gateway and load balancers only.

Each e6data engine pod consumes one IP address. For large workloads with 100+ concurrent pods, make sure private subnets have enough IP space.

Availability zones

Requirement	Minimum	Recommended
Number of AZs	2	3
Private subnets	1 per AZ	1 per AZ
Public subnets	1 per AZ	1 per AZ

e6data spreads workloads across AZs for high availability. Tag subnets so EKS, the load balancer controller, and Karpenter can discover them:

# Private subnets (internal load balancers)
kubernetes.io/role/internal-elb = 1
# Public subnets (internet-facing load balancers)
kubernetes.io/role/elb = 1
# All subnets (EKS auto-discovery)
kubernetes.io/cluster/<cluster-name> = shared
# Private subnets (Karpenter node provisioning)
karpenter.sh/discovery = <cluster-name>-eks

Service quotas

Verify these quotas before deploying. Request increases early - they can take 24–48 hours.

Service	Quota	Minimum required
EC2	Running On-Demand instances	100 vCPUs
EC2	Running Spot instances	500 vCPUs
EC2	EC2-VPC Elastic IPs	5
EKS	Clusters per Region	1+ available
VPC	NAT Gateways per AZ	1+ available
VPC	VPCs per Region	1+ available

Instance types

Component	Instance families	Architecture	Notes
System nodes	t3, m5	x86_64 (amd64)	EKS managed node group for system pods.
e6-operator	t3, t4g	amd64 or arm64	Operator controller pods.
Engine nodes	r6g, r6gd, r7g, r7gd, m7g, c7g	arm64 (Graviton)	Query engine pods, provisioned by Karpenter.

The e6data engine is optimized for AWS Graviton (arm64) instances for better price-performance. Confirm your Region has capacity for the r6g/r7g families.

VPC CNI configuration

The Amazon VPC CNI plugin manages pod networking:

Setting	Value	Rationale
WARM_IP_TARGET	5	Pre-allocates IPs for faster pod startup.
MINIMUM_IP_TARGET	10	Keeps a minimum number of IPs available per node.
ENABLE_PREFIX_DELEGATION	true	Increases IP capacity per node (recommended for large clusters).

For clusters with 100+ pods, enable prefix delegation to avoid IP exhaustion. It assigns /28 prefixes (16 IPs) instead of individual IPs, allowing up to 110 pods per node on most instance types:

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1

Existing VPC checklist

If you reuse an existing VPC, verify:

• DNS hostnames enabled (enableDnsHostnames: true).
• DNS resolution enabled (enableDnsSupport: true).
• Private subnets route to a NAT Gateway for internet access.
• Private subnets tagged kubernetes.io/role/internal-elb = 1.
• Public subnets tagged kubernetes.io/role/elb = 1.
• Private subnets tagged karpenter.sh/discovery = <cluster-name>-eks.
• Cluster security group tagged karpenter.sh/discovery = <cluster-name>-eks (after EKS creation).
• Sufficient IP space in private subnets (minimum /22 per subnet).

Part 1 - AWS infrastructure

Step 1: Create the VPC

# Set your variables
export CLUSTER_NAME="e6data-cluster"
export REGION="us-east-1"
export VPC_CIDR="10.0.0.0/16"

# Create VPC
VPC_ID=$(aws ec2 create-vpc \
  --cidr-block $VPC_CIDR \
  --tag-specifications "ResourceType=vpc,Tags=[{Key=Name,Value=${CLUSTER_NAME}-vpc}]" \
  --query 'Vpc.VpcId' --output text --region $REGION)

echo "VPC_ID=$VPC_ID"

# Enable DNS hostnames
aws ec2 modify-vpc-attribute --vpc-id $VPC_ID --enable-dns-hostnames --region $REGION

# Create Internet Gateway
IGW_ID=$(aws ec2 create-internet-gateway \
  --tag-specifications "ResourceType=internet-gateway,Tags=[{Key=Name,Value=${CLUSTER_NAME}-igw}]" \
  --query 'InternetGateway.InternetGatewayId' --output text --region $REGION)

aws ec2 attach-internet-gateway --vpc-id $VPC_ID --internet-gateway-id $IGW_ID --region $REGION

# Get availability zones
AZS=$(aws ec2 describe-availability-zones --region $REGION --query 'AvailabilityZones[0:3].ZoneName' --output text)
AZ1=$(echo $AZS | awk '{print $1}')
AZ2=$(echo $AZS | awk '{print $2}')
AZ3=$(echo $AZS | awk '{print $3}')

# Create public subnets
PUBLIC_SUBNET_1=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.1.0/24 \
  --availability-zone $AZ1 \
  --tag-specifications "ResourceType=subnet,Tags=[{Key=Name,Value=${CLUSTER_NAME}-public-1},{Key=kubernetes.io/role/elb,Value=1}]" \
  --query 'Subnet.SubnetId' --output text --region $REGION)

PUBLIC_SUBNET_2=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.2.0/24 \
  --availability-zone $AZ2 \
  --tag-specifications "ResourceType=subnet,Tags=[{Key=Name,Value=${CLUSTER_NAME}-public-2},{Key=kubernetes.io/role/elb,Value=1}]" \
  --query 'Subnet.SubnetId' --output text --region $REGION)

# Create private subnets
PRIVATE_SUBNET_1=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.10.0/24 \
  --availability-zone $AZ1 \
  --tag-specifications "ResourceType=subnet,Tags=[{Key=Name,Value=${CLUSTER_NAME}-private-1},{Key=kubernetes.io/role/internal-elb,Value=1}]" \
  --query 'Subnet.SubnetId' --output text --region $REGION)

PRIVATE_SUBNET_2=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.11.0/24 \
  --availability-zone $AZ2 \
  --tag-specifications "ResourceType=subnet,Tags=[{Key=Name,Value=${CLUSTER_NAME}-private-2},{Key=kubernetes.io/role/internal-elb,Value=1}]" \
  --query 'Subnet.SubnetId' --output text --region $REGION)

# Create Elastic IP for NAT Gateway
EIP_ALLOC=$(aws ec2 allocate-address --domain vpc \
  --tag-specifications "ResourceType=elastic-ip,Tags=[{Key=Name,Value=${CLUSTER_NAME}-nat-eip}]" \
  --query 'AllocationId' --output text --region $REGION)

# Create NAT Gateway
NAT_GW=$(aws ec2 create-nat-gateway \
  --subnet-id $PUBLIC_SUBNET_1 \
  --allocation-id $EIP_ALLOC \
  --tag-specifications "ResourceType=natgateway,Tags=[{Key=Name,Value=${CLUSTER_NAME}-nat}]" \
  --query 'NatGateway.NatGatewayId' --output text --region $REGION)

echo "Waiting for NAT Gateway to become available..."
aws ec2 wait nat-gateway-available --nat-gateway-ids $NAT_GW --region $REGION

# Create route tables
PUBLIC_RT=$(aws ec2 create-route-table --vpc-id $VPC_ID \
  --tag-specifications "ResourceType=route-table,Tags=[{Key=Name,Value=${CLUSTER_NAME}-public-rt}]" \
  --query 'RouteTable.RouteTableId' --output text --region $REGION)

aws ec2 create-route --route-table-id $PUBLIC_RT --destination-cidr-block 0.0.0.0/0 \
  --gateway-id $IGW_ID --region $REGION

aws ec2 associate-route-table --route-table-id $PUBLIC_RT --subnet-id $PUBLIC_SUBNET_1 --region $REGION
aws ec2 associate-route-table --route-table-id $PUBLIC_RT --subnet-id $PUBLIC_SUBNET_2 --region $REGION

PRIVATE_RT=$(aws ec2 create-route-table --vpc-id $VPC_ID \
  --tag-specifications "ResourceType=route-table,Tags=[{Key=Name,Value=${CLUSTER_NAME}-private-rt}]" \
  --query 'RouteTable.RouteTableId' --output text --region $REGION)

aws ec2 create-route --route-table-id $PRIVATE_RT --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id $NAT_GW --region $REGION

aws ec2 associate-route-table --route-table-id $PRIVATE_RT --subnet-id $PRIVATE_SUBNET_1 --region $REGION
aws ec2 associate-route-table --route-table-id $PRIVATE_RT --subnet-id $PRIVATE_SUBNET_2 --region $REGION

# Create S3 VPC Endpoint (saves NAT costs for S3 traffic)
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --service-name com.amazonaws.${REGION}.s3 \
  --route-table-ids $PRIVATE_RT \
  --tag-specifications "ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=${CLUSTER_NAME}-s3-endpoint}]" \
  --region $REGION

echo "VPC setup complete!"
echo "VPC_ID=$VPC_ID"
echo "PUBLIC_SUBNETS=$PUBLIC_SUBNET_1,$PUBLIC_SUBNET_2"
echo "PRIVATE_SUBNETS=$PRIVATE_SUBNET_1,$PRIVATE_SUBNET_2"

# Tag private subnets for Karpenter discovery
aws ec2 create-tags --resources $PRIVATE_SUBNET_1 $PRIVATE_SUBNET_2 \
  --tags Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}-eks \
  --region $REGION

# Note: the cluster security group is tagged for Karpenter after the EKS cluster is created (Step 3).

Step 2: Create IAM roles

EKS cluster role:

cat > eks-cluster-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "eks.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role \
  --role-name ${CLUSTER_NAME}-eks-cluster-role \
  --assume-role-policy-document file://eks-cluster-trust-policy.json

aws iam attach-role-policy \
  --role-name ${CLUSTER_NAME}-eks-cluster-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy

EKS node role:

cat > eks-node-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "ec2.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role \
  --role-name ${CLUSTER_NAME}-eks-node-role \
  --assume-role-policy-document file://eks-node-trust-policy.json

aws iam attach-role-policy --role-name ${CLUSTER_NAME}-eks-node-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
aws iam attach-role-policy --role-name ${CLUSTER_NAME}-eks-node-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
aws iam attach-role-policy --role-name ${CLUSTER_NAME}-eks-node-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
aws iam attach-role-policy --role-name ${CLUSTER_NAME}-eks-node-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

aws iam create-instance-profile --instance-profile-name ${CLUSTER_NAME}-eks-node-profile
aws iam add-role-to-instance-profile \
  --instance-profile-name ${CLUSTER_NAME}-eks-node-profile \
  --role-name ${CLUSTER_NAME}-eks-node-role

Karpenter role:

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

cat > karpenter-trust-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "pods.eks.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
  ]
}
EOF

aws iam create-role \
  --role-name ${CLUSTER_NAME}-karpenter-role \
  --assume-role-policy-document file://karpenter-trust-policy.json

cat > karpenter-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Karpenter",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateFleet", "ec2:CreateLaunchTemplate", "ec2:CreateTags",
        "ec2:DeleteLaunchTemplate", "ec2:DescribeAvailabilityZones",
        "ec2:DescribeImages", "ec2:DescribeInstances",
        "ec2:DescribeInstanceTypeOfferings", "ec2:DescribeInstanceTypes",
        "ec2:DescribeLaunchTemplates", "ec2:DescribeSecurityGroups",
        "ec2:DescribeSpotPriceHistory", "ec2:DescribeSubnets",
        "ec2:RunInstances", "ec2:TerminateInstances",
        "iam:PassRole", "pricing:GetProducts", "ssm:GetParameter"
      ],
      "Resource": "*"
    },
    {
      "Sid": "SQS",
      "Effect": "Allow",
      "Action": ["sqs:DeleteMessage", "sqs:GetQueueAttributes", "sqs:GetQueueUrl", "sqs:ReceiveMessage"],
      "Resource": "arn:aws:sqs:${REGION}:${AWS_ACCOUNT_ID}:${CLUSTER_NAME}-karpenter"
    }
  ]
}
EOF

aws iam put-role-policy \
  --role-name ${CLUSTER_NAME}-karpenter-role \
  --policy-name KarpenterPolicy \
  --policy-document file://karpenter-policy.json

aws iam create-instance-profile --instance-profile-name ${CLUSTER_NAME}-karpenter
aws iam add-role-to-instance-profile \
  --instance-profile-name ${CLUSTER_NAME}-karpenter \
  --role-name ${CLUSTER_NAME}-eks-node-role

ALB Controller role:

aws iam create-role \
  --role-name ${CLUSTER_NAME}-alb-controller-role \
  --assume-role-policy-document file://karpenter-trust-policy.json

curl -o alb-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json

aws iam put-role-policy \
  --role-name ${CLUSTER_NAME}-alb-controller-role \
  --policy-name ALBControllerPolicy \
  --policy-document file://alb-policy.json

EBS CSI driver role:

The EBS CSI driver addon uses EKS Pod Identity for its IAM permissions:

cat > ebs-csi-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "pods.eks.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
  ]
}
EOF

aws iam create-role \
  --role-name ${CLUSTER_NAME}-ebs-csi-driver-role \
  --assume-role-policy-document file://ebs-csi-trust-policy.json

aws iam attach-role-policy \
  --role-name ${CLUSTER_NAME}-ebs-csi-driver-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy

Step 3: Create the EKS cluster

aws eks create-cluster \
  --name ${CLUSTER_NAME}-eks \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-eks-cluster-role \
  --resources-vpc-config subnetIds=${PRIVATE_SUBNET_1},${PRIVATE_SUBNET_2},endpointPublicAccess=true,endpointPrivateAccess=true \
  --access-config authenticationMode=API_AND_CONFIG_MAP \
  --kubernetes-version 1.31 \
  --region $REGION

echo "Waiting for EKS cluster to become active (this takes ~10-15 minutes)..."
aws eks wait cluster-active --name ${CLUSTER_NAME}-eks --region $REGION

aws eks update-kubeconfig --name ${CLUSTER_NAME}-eks --region $REGION

CLUSTER_ENDPOINT=$(aws eks describe-cluster --name ${CLUSTER_NAME}-eks \
  --query 'cluster.endpoint' --output text --region $REGION)
OIDC_ISSUER=$(aws eks describe-cluster --name ${CLUSTER_NAME}-eks \
  --query 'cluster.identity.oidc.issuer' --output text --region $REGION)

# Tag the cluster security group for Karpenter discovery
CLUSTER_SG=$(aws eks describe-cluster --name ${CLUSTER_NAME}-eks \
  --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId' --output text --region $REGION)

aws ec2 create-tags --resources $CLUSTER_SG \
  --tags Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}-eks \
  --region $REGION

Step 4: Create the OIDC provider and Pod Identity add-ons

eksctl utils associate-iam-oidc-provider \
  --cluster ${CLUSTER_NAME}-eks --region $REGION --approve

aws eks create-addon --cluster-name ${CLUSTER_NAME}-eks --addon-name eks-pod-identity-agent --region $REGION
aws eks create-addon --cluster-name ${CLUSTER_NAME}-eks --addon-name vpc-cni --region $REGION

# Install the EBS CSI driver with a Pod Identity association so it can manage EBS volumes
aws eks create-addon --cluster-name ${CLUSTER_NAME}-eks --addon-name aws-ebs-csi-driver \
  --pod-identity-associations "serviceAccount=ebs-csi-controller-sa,roleArn=arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-ebs-csi-driver-role" \
  --region $REGION

Step 5: Create the system node group

aws eks create-nodegroup \
  --cluster-name ${CLUSTER_NAME}-eks \
  --nodegroup-name system-nodes \
  --scaling-config minSize=2,maxSize=4,desiredSize=2 \
  --instance-types t3.medium \
  --node-role arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-eks-node-role \
  --subnets $PRIVATE_SUBNET_1 $PRIVATE_SUBNET_2 \
  --labels role=system \
  --region $REGION

aws eks wait nodegroup-active \
  --cluster-name ${CLUSTER_NAME}-eks --nodegroup-name system-nodes --region $REGION

Step 6: Create the S3 metadata bucket

export WORKSPACE_NAME="my-workspace"  # Replace with your workspace name

# Note: us-east-1 is S3's default region and rejects LocationConstraint
if [ "$REGION" = "us-east-1" ]; then
  aws s3api create-bucket \
    --bucket e6-${WORKSPACE_NAME}-metadata \
    --region $REGION
else
  aws s3api create-bucket \
    --bucket e6-${WORKSPACE_NAME}-metadata \
    --region $REGION \
    --create-bucket-configuration LocationConstraint=$REGION
fi

aws s3api put-public-access-block \
  --bucket e6-${WORKSPACE_NAME}-metadata \
  --public-access-block-configuration \
  "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

aws s3api put-bucket-versioning \
  --bucket e6-${WORKSPACE_NAME}-metadata \
  --versioning-configuration Status=Enabled

aws s3api put-bucket-encryption \
  --bucket e6-${WORKSPACE_NAME}-metadata \
  --server-side-encryption-configuration '{
    "Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]
  }'

cat > bucket-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EnforceSSL",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata",
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata/*"
      ],
      "Condition": { "Bool": {"aws:SecureTransport": "false"} }
    }
  ]
}
EOF

aws s3api put-bucket-policy \
  --bucket e6-${WORKSPACE_NAME}-metadata \
  --policy file://bucket-policy.json

Do not enable S3 Object Lock on this bucket - it can break metadata operations.

Step 7: Create workspace IAM roles

Create the engine, console, and app roles. All three trust pods.eks.amazonaws.com:

cat > engine-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "pods.eks.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
  ]
}
EOF

# Engine role (query engine and Laminar)
aws iam create-role --role-name ${WORKSPACE_NAME}-engine-role \
  --assume-role-policy-document file://engine-trust-policy.json

cat > engine-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3MetadataAccess",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:AbortMultipartUpload"],
      "Resource": [
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata",
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata/*"
      ]
    },
    {
      "Sid": "CrossAccountAccess",
      "Effect": "Allow",
      "Action": ["sts:AssumeRole", "sts:TagSession"],
      "Resource": "*"
    }
  ]
}
EOF

aws iam put-role-policy --role-name ${WORKSPACE_NAME}-engine-role \
  --policy-name EnginePolicy --policy-document file://engine-policy.json

# Console role (UI backend)
aws iam create-role --role-name ${WORKSPACE_NAME}-console-role \
  --assume-role-policy-document file://engine-trust-policy.json

cat > console-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3MetadataAccess",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:AbortMultipartUpload"],
      "Resource": [
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata",
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata/*"
      ]
    },
    {
      "Sid": "VPCEndpointDiscovery",
      "Effect": "Allow",
      "Action": ["ec2:DescribeVpcEndpoints", "ec2:DescribeInstances"],
      "Resource": "*"
    },
    {
      "Sid": "EKSPodIdentityLookup",
      "Effect": "Allow",
      "Action": ["eks:DescribeCluster", "eks:ListPodIdentityAssociations", "eks:DescribePodIdentityAssociation"],
      "Resource": [
        "arn:aws:eks:${REGION}:${AWS_ACCOUNT_ID}:cluster/${CLUSTER_NAME}-eks",
        "arn:aws:eks:${REGION}:${AWS_ACCOUNT_ID}:podidentityassociation/${CLUSTER_NAME}-eks/*"
      ]
    }
  ]
}
EOF

aws iam put-role-policy --role-name ${WORKSPACE_NAME}-console-role \
  --policy-name ConsolePolicy --policy-document file://console-policy.json

# App role (Copilot and Metriq)
aws iam create-role --role-name ${WORKSPACE_NAME}-app-role \
  --assume-role-policy-document file://engine-trust-policy.json

cat > app-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3MetadataAccess",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:AbortMultipartUpload"],
      "Resource": [
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata",
        "arn:aws:s3:::e6-${WORKSPACE_NAME}-metadata/*"
      ]
    },
    {
      "Sid": "SecretsManagerAccess",
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
      "Resource": "arn:aws:secretsmanager:${REGION}:${AWS_ACCOUNT_ID}:secret:copilot-metriq-secret-*"
    }
  ]
}
EOF

aws iam put-role-policy --role-name ${WORKSPACE_NAME}-app-role \
  --policy-name AppPolicy --policy-document file://app-policy.json

Step 8: Create Pod Identity associations

kubectl create namespace $WORKSPACE_NAME

aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace $WORKSPACE_NAME --service-account ${WORKSPACE_NAME}-engine \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${WORKSPACE_NAME}-engine-role --region $REGION

aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace $WORKSPACE_NAME --service-account ${WORKSPACE_NAME}-console \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${WORKSPACE_NAME}-console-role --region $REGION

aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace $WORKSPACE_NAME --service-account ${WORKSPACE_NAME}-copilot \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${WORKSPACE_NAME}-app-role --region $REGION

aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace $WORKSPACE_NAME --service-account ${WORKSPACE_NAME}-metriq \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${WORKSPACE_NAME}-app-role --region $REGION

aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace $WORKSPACE_NAME --service-account ${WORKSPACE_NAME}-laminar \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${WORKSPACE_NAME}-engine-role --region $REGION

Step 9: Create the Karpenter SQS queue

aws sqs create-queue --queue-name ${CLUSTER_NAME}-karpenter --region $REGION

QUEUE_ARN=$(aws sqs get-queue-attributes \
  --queue-url https://sqs.${REGION}.amazonaws.com/${AWS_ACCOUNT_ID}/${CLUSTER_NAME}-karpenter \
  --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)

cat > sqs-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "events.amazonaws.com"},
      "Action": "sqs:SendMessage",
      "Resource": "${QUEUE_ARN}"
    }
  ]
}
EOF

aws sqs set-queue-attributes \
  --queue-url https://sqs.${REGION}.amazonaws.com/${AWS_ACCOUNT_ID}/${CLUSTER_NAME}-karpenter \
  --attributes Policy="$(cat sqs-policy.json | jq -c .)"

Part 2 - Kubernetes platform components

e6data uses the helm template | kubectl apply --server-side pattern instead of helm install to avoid Helm's 1 MB release Secret limit and ensure clean field-manager handling during upgrades.

Step 1: Install Karpenter CRDs

export KARPENTER_VERSION="1.8.1"
tmpdir=$(mktemp -d)

helm template karpenter-crd \
  oci://public.ecr.aws/karpenter/karpenter-crd \
  --version "${KARPENTER_VERSION}" --include-crds \
  > "${tmpdir}/karpenter-crds.yaml"

kubectl apply --server-side --force-conflicts -f "${tmpdir}/karpenter-crds.yaml"
kubectl wait --for=condition=Established crd/nodepools.karpenter.sh crd/nodeclaims.karpenter.sh --timeout=60s
kubectl get crds | grep karpenter

Step 2: Install the Karpenter controller

CLUSTER_ENDPOINT=$(aws eks describe-cluster --name ${CLUSTER_NAME}-eks \
  --query 'cluster.endpoint' --output text --region $REGION)

cat > karpenter-values.yaml << EOF
settings:
  clusterName: ${CLUSTER_NAME}-eks
  clusterEndpoint: ${CLUSTER_ENDPOINT}
  interruptionQueue: ${CLUSTER_NAME}-karpenter
replicas: 2
controller:
  resources:
    requests: { cpu: 200m, memory: 256Mi }
    limits: { cpu: 1, memory: 1Gi }
EOF

helm template karpenter \
  oci://public.ecr.aws/karpenter/karpenter \
  --version "${KARPENTER_VERSION}" --namespace kube-system --skip-crds \
  -f karpenter-values.yaml \
  | kubectl apply --server-side --force-conflicts -f -

# Link the karpenter service account to the IAM role created in Part 1, Step 2
aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace kube-system --service-account karpenter \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter-role --region $REGION

# Restart Karpenter so it picks up the new credentials
kubectl rollout restart deployment/karpenter -n kube-system

kubectl rollout status deployment/karpenter -n kube-system --timeout=120s
kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter

Step 3: Install the AWS Load Balancer Controller

export ALB_CONTROLLER_VERSION="1.8.1"

helm repo add eks https://aws.github.io/eks-charts
helm repo update eks

cat > alb-values.yaml << EOF
clusterName: ${CLUSTER_NAME}-eks
region: ${REGION}
vpcId: ${VPC_ID}
EOF

helm upgrade --install aws-load-balancer-controller \
  eks/aws-load-balancer-controller \
  --version "${ALB_CONTROLLER_VERSION}" --namespace kube-system \
  -f alb-values.yaml

# Link the aws-load-balancer-controller service account to the IAM role created in Part 1, Step 2
aws eks create-pod-identity-association --cluster-name ${CLUSTER_NAME}-eks \
  --namespace kube-system --service-account aws-load-balancer-controller \
  --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-alb-controller-role --region $REGION

# Restart the controller so it picks up the new credentials
kubectl rollout restart deployment/aws-load-balancer-controller -n kube-system

kubectl rollout status deployment/aws-load-balancer-controller -n kube-system --timeout=120s
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller

Step 4: Create the e6-operator NodePool

cat << EOF | kubectl apply -f -
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: e6operator
spec:
  amiSelectorTerms:
    - alias: al2023@latest
  instanceProfile: ${CLUSTER_NAME}-karpenter
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: ${CLUSTER_NAME}-eks
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: ${CLUSTER_NAME}-eks
  tags:
    Name: ${CLUSTER_NAME}-e6operator
EOF

cat << EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: e6operator
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: e6operator
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: [t3.medium, t3.large, t4g.medium, t4g.large]
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["${AZ1}", "${AZ2}"]
      taints:
        - key: e6operator
          value: "true"
          effect: NoSchedule
  limits:
    cpu: 100
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30s
EOF

kubectl get nodepools.karpenter.sh e6operator
kubectl get ec2nodeclasses.karpenter.k8s.aws e6operator

Step 5: Install cert-manager

export CERT_MANAGER_VERSION="v1.19.2"

helm upgrade --install cert-manager \
  oci://quay.io/jetstack/charts/cert-manager \
  --version "${CERT_MANAGER_VERSION}" --namespace cert-manager --create-namespace \
  --set crds.enabled=true --set webhook.timeoutSeconds=4 --timeout 10m --wait

kubectl rollout status deployment/cert-manager -n cert-manager --timeout=120s
kubectl rollout status deployment/cert-manager-webhook -n cert-manager --timeout=120s
kubectl rollout status deployment/cert-manager-cainjector -n cert-manager --timeout=120s
kubectl get pods -n cert-manager

Step 6: Install the e6-operator CRDs

e6data provides the CRD files. Apply them with server-side apply:

export E6_OPERATOR_CRDS_PATH="./e6-operator-crds"

kubectl apply --server-side --force-conflicts -f "${E6_OPERATOR_CRDS_PATH}/"
kubectl wait --for=condition=Established crd/namespaceconfigs.e6data.io --timeout=120s
kubectl get crds | grep e6data.io

All CRDs from the provided archive are installed - expect 20+ resources with the e6data.io suffix, including:

e6catalogs.e6data.io
metadataservices.e6data.io
monitoringservices.e6data.io
namespaceconfigs.e6data.io
queryrouters.e6data.io
queryservices.e6data.io

Step 7: Create the gp3 StorageClass

cat << 'EOF' | kubectl apply --server-side --force-conflicts -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  fsType: ext4
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF

kubectl get storageclass gp3

Step 8: Install the e6-operator controller

e6data provides the operator Helm chart. Install it with helm template | kubectl apply:

export E6_OPERATOR_IMAGE_REPO="<ECR_REPO_PROVIDED_BY_E6DATA>"
export E6_OPERATOR_IMAGE_TAG="<VERSION_PROVIDED_BY_E6DATA>"
export E6_OPERATOR_CHART="./e6-operator-chart"

kubectl create namespace e6operator --dry-run=client -o yaml | kubectl apply -f -

cat > operator-values.yaml << EOF
image:
  repository: ${E6_OPERATOR_IMAGE_REPO}
  tag: ${E6_OPERATOR_IMAGE_TAG}
  pullPolicy: IfNotPresent
replicaCount: 2
resources:
  requests: { cpu: 200m, memory: 256Mi }
  limits: { cpu: 1, memory: 1Gi }
logLevel: info
tolerations:
  - key: e6operator
    operator: Equal
    value: "true"
    effect: NoSchedule
EOF

helm template e6operator "${E6_OPERATOR_CHART}" \
  --namespace e6operator -f operator-values.yaml \
  | kubectl apply --server-side --force-conflicts -f -

kubectl rollout status deployment/e6operator -n e6operator --timeout=900s
kubectl get pods -n e6operator

Next

Continue to Deploy workspace and e6data.