Prerequisite Infrastructure

The following components are required before setting up the infrastructure needed by e6data. These are commonly present in most cloud environments, but if any are not present, please follow the linked guides below to create them.

  1. Create VNET, SUBNETS AND NAT Gateway

  2. AKS Cluster

1. Prerequisites

Ensure you have the Azure CLI installed and you are logged in.

az login

2. Create a Virtual Network

Create a virtual network in the resource group.

az network vnet create \
  --name <prefix>-network \
  --resource-group <resource-group-name> \
  --address-prefix <cidr-block> \
  --location <region>

Replace <prefix> with your prefix, <resource-group-name> with your resource group name, <cidr-block> with your desired CIDR block (e.g., 10.0.0.0/16), and <region> with the Azure region.

3. Create Subnets

Create AKS Subnet

az network vnet subnet create \
  --name <prefix>-subnet-aks \
  --resource-group <resource-group-name> \
  --vnet-name <prefix>-network \
  --address-prefixes <aks-subnet-cidr>

Replace <prefix> with your prefix, <resource-group-name> with your resource group name, <aks-subnet-cidr> with the CIDR block for the AKS subnet (e.g., 10.0.1.0/24).

Create ACI Subnet

az network vnet subnet create \
  --name <prefix>-subnet-aci \
  --resource-group <resource-group-name> \
  --vnet-name <prefix>-network \
  --address-prefixes <aci-subnet-cidr>

Replace <prefix> with your prefix, <resource-group-name> with your resource group name, <aci-subnet-cidr> with the CIDR block for the ACI subnet (e.g., 10.0.2.0/24).

4. Delegate ACI Subnet

az network vnet subnet update \
  --name <prefix>-subnet-aci \
  --resource-group <resource-group-name> \
  --vnet-name <prefix>-network \
  --delegations Microsoft.ContainerInstance/containerGroups

5. Create a Public IP Address

Create a static public IP address to be used by the NAT gateway.

az network public-ip create \
    --resource-group <resource-group-name> \
    --name <prefix>-PIP \
    --sku Standard \
    --location <region> \
    --allocation-method Static

Replace <prefix> with your prefix, <resource-group-name> with your resource group name, and <region> with the Azure region.

6. Create a NAT Gateway

Create a NAT gateway and associate it with the public IP address.

az network nat gateway create \
    --resource-group <resource-group-name> \
    --name <prefix>-nat \
    --public-ip-addresses <prefix>-PIP \
    --idle-timeout 30 \
    --location <region>

7. Associate the NAT Gateway with the AKS Subnet

az network vnet subnet update \
    --resource-group <resource-group-name> \
    --vnet-name <prefix>-network \
    --name <prefix>-subnet-aks \
    --nat-gateway <prefix>-nat

Replace <prefix> with your prefix, <resource-group-name> with your resource group name, <region> with the Azure region, and <aks-subnet-cidr> with the AKS subnet CIDR block.

Creating a New Azure AKS Cluster (Skip if You Already Have an AKS Cluster)

  1. Ensure you have the Azure CLI installed and configured on your local machine. If you haven't installed it yet, please follow the instructions on How to install the Azure CLI to set it up.

  2. Open a terminal or command prompt.

  3. Run the following command to create a new AKS cluster:

az aks create \
  --resource-group <your-resource-group-name> \
  --name <your-cluster-name> \
  --location <your-region> \
  --kubernetes-version <kube-version> \
  --node-count <default-node-pool-node-count> \
  --node-vm-size <default-node-pool-vm-size> \
  --nodepool-name <default-node-pool-name> \
  --vnet-subnet-id <aks-subnet-id> \
  --network-plugin azure \
  --network-policy cilium \
  --network-plugin-mode overlay \
  --network-dataplane cilium \
  --enable-aad \
  --aad-admin-group-object-ids <admin-group-object-ids> \
  --enable-managed-identity \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --ssh-key-value <path-to-public-ssh-key> \
  --aci-subnet-name <aci-subnet-name> \
  --enable-private-cluster \
  --tags <your-tags>

Creating an AKS Cluster with Azure CLI

To create an Azure Kubernetes Service (AKS) cluster with our desired configuration, we use the az aks create command. This command allows us to specify various parameters to customize our AKS cluster.

Command Breakdown

  • --resource-group: Specifies the resource group where the AKS cluster will be created.

  • --name: Sets the name of the AKS cluster.

  • --location: Defines the Azure region where the cluster will be deployed.

  • --kubernetes-version: Specifies the Kubernetes version to use.

  • --node-count: Sets the number of nodes in the default node pool.

  • --node-vm-size: Defines the VM size for the nodes.

  • --vnet-subnet-id: Specifies the subnet ID where the cluster will be deployed.

  • --network-plugin azure: Uses Azure CNI for networking.

  • --network-policy cilium: Enables Cilium for network policy.

  • --network-plugin-mode overlay: Sets the network plugin mode to overlay.

  • --network-dataplane cilium: Uses Cilium as the network data plane.

  • --enable-aad: Enables Azure Active Directory integration.

  • --aad-admin-group-object-ids: Specifies the AAD group object IDs for cluster admins.

  • --enable-managed-identity: Uses managed identity for the cluster.

  • --enable-oidc-issuer: Enables OIDC issuer for the cluster.

  • --enable-workload-identity: Enables workload identity.

  • --ssh-key-value: Specifies the SSH public key for the Linux nodes.

  • --aci-subnet-name: Specifies the subnet for Azure Container Instances.

  • --enable-private-cluster: Creates a private cluster.

  • --nodepool-name: Sets the name of the default node pool.

  • --tags: Adds tags to the AKS cluster.

  • For detailed instructions and more advanced configurations, you can refer to the official Azure documentation on https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-cli

Note: If you haven't already configured Azure AD groups for AKS RBAC, you can refer to the following link for instructions: Configuring groups for Azure AKS with Azure AD RBAC. This will guide you in setting up and managing Azure AD groups for role-based access control within your AKS cluster.

  1. Wait for the cluster creation process to complete. This may take some time.

  2. Once the AKS cluster is created, you can retrieve the connection information by running the following command:

az aks get-credentials --resource-group [RESOURCE_GROUP] --name [CLUSTER_NAME]

Replace [RESOURCE_GROUP] and [CLUSTER_NAME] with the appropriate values. This command will configure the kubectl command-line tool to connect to the AKS cluster.

  1. Verify the connection to the AKS cluster by running the following command:

kubectl get nodes

This should display the list of nodes in your AKS cluster.

Setup Kubernetes Components

Add e6data Helm Charts Repository

The Helm chart creates a service account that will be used by the e6data cluster to leverage OIDC (OpenID Connect) authentication and obtain read permissions to access the data buckets. It also creates roles and role bindings in the AKS cluster for the e6data control plane.

These roles and role bindings define the permissions and access levels for the control plane user within the cluster, allowing it to perform specific actions and interact with resources as required by the e6data workspace.

helm repo add e6data https://e6x-labs.github.io/helm-charts/
helm repo update

GitHub - e6x-labs/helm-chartsGitHub

Edit values.yaml

  1. Navigate to ./helm-charts/charts/workspace/

  2. Open the values.yaml file

  3. Replace <E6DATA_USER_ASSIGNED_IDENTITY_CLIENT_ID> with the ClientID of the managed identity created previously.

  4. Replace <OBJECT_ID_OF_THE_E6DATA_APP_REGISTRATION> with the ObjectID of the app registration created previously.

  5. Replace <WORKSPACE_NAME> with the name of the e6data workspace you will be creating.

  6. Replace the <NODE_POOL_NAME>&<NODE_CLASS_NAME> with the name provided in the previous node pool and node class creation steps.

  7. Save the edited values.yaml file

cloud:
  type: "AZURE"
  oidc_value: <E6DATA_USER_ASSIGNED_IDENTITY_CLIENT_ID>
  control_plane_user:
    - <OBJECT_ID_OF_THE_E6DATA_APP_REGISTRATION>
karpenter:
  nodepool:
    - "<NODE_POOL_NAME>"
  nodeclass:
    - "<NODE_CLASS_NAME>"

Please make note of the Workspace Name, it will be required when creating the Workspace in the e6data Console:

Run Helm Charts3

helm upgrade -i -f /path/to/your/helm/values.yaml -n <KUBERNETES_NAMESPACE> <WORKSPACE_NAME> --version "2.0.8" e6x-labs/workspace

When the Helm chart finishes running, proceed to the e6data Console to create a Workspace.

PreviousAzure SetupNextInfrastructure & Permissions for e6data

Last updated

Change request #930: Cross account hive GCP