Prerequisite Infrastructure

The following components are required before setting up the infrastructure needed by e6data. These are commonly present in most cloud environments, but if any are not present, please follow the linked guides below to create them.

1. Create VNET, SUBNETS AND NAT Gateway

2. AKS Cluster

1. Prerequisites

Ensure that you have the Azure CLI installed on your system. You can install it from here.

Once installed, log in to your Azure account using the following command:

az login

2. Create Resource Group

In Azure, a resource group acts as a logical container that holds related resources for your solution. It allows you to manage, deploy, and organize resources conveniently. To create a resource group, use the following Azure CLI command:

az group create \
  --name <resource-group-name> \
  --location <region>

Command Breakdown

--name <resource-group-name>: The name of the resource group you want to create. This name should be relevant to your project or environment. 

--location <region>: The Azure region where your resource group will be created. The region determines the physical location of the resources in the group. Example regions include eastus, westeurope, or southeastasia.

For example

az group create \
  --name e6data-app-rg \
  --location "EastUS"

3. Create a Virtual Network

After creating a resource group, the next step is to create a Virtual Network (VNet) within that group. A VNet is an essential part of Azure networking and allows you to manage your network resources efficiently.

az network vnet create \
  --name <prefix>-network \
  --resource-group <resource-group-name> \
  --address-prefix <cidr-block> \
  --location <region>

Command Breakdown

--name <prefix>-network: Replace <prefix> with a meaningful identifier to name your VNet. 

--resource-group <resource-group-name>: Specify the resource group where the VNet will be created. This should be the name of an existing resource group.
 
--address-prefix <cidr-block>: Specify the address range for the VNet in CIDR notation (e.g., 10.0.0.0/16).

--location <region>: Specify the Azure region for the VNet. Replace <region> with a region like eastus, westeurope, or centralindia.

For example

az network vnet create \
  --name e6data-app-network \
  --resource-group e6data-app-rg \
  --address-prefix 10.0.0.0/16 \
  --location "EastUS"

4. Create Subnets

Create AKS Subnet

To create a subnet specifically for Azure Kubernetes Service (AKS), use the following command:

az network vnet subnet create \
  --name <prefix>-subnet-aks \
  --resource-group <resource-group-name> \
  --vnet-name <prefix>-network \
  --address-prefixes <aks-subnet-cidr>

Command Breakdown

--name <prefix>-subnet-aks: Replace <prefix> with a meaningful identifier that you’re using throughout your setup. 

--resource-group <resource-group-name>: Specify the name of the resource group you created earlier.

--vnet-name <prefix>-network: This should match the name of the VNet you created earlier.

--address-prefixes <aks-subnet-cidr>: Replace this with the CIDR block for your AKS subnet (e.g., 10.0.1.0/24).

For example

az network vnet subnet create \
  --name e6data-subnet-aks \
  --resource-group e6data-app-rg \
  --vnet-name e6data-app-network \
  --address-prefixes 10.0.1.0/24

Create ACI Subnet

To create a subnet specifically for Azure Container Instances (ACI), use the following command:

az network vnet subnet create \
  --name <prefix>-subnet-aci \
  --resource-group <resource-group-name> \
  --vnet-name <prefix>-network \
  --address-prefixes <aci-subnet-cidr>

Command Breakdown

--name <prefix>-subnet-aci: Replace <prefix> with your chosen identifier. 

--resource-group <resource-group-name>: Specify the resource group name created earlier.

--vnet-name <prefix>-network: This should match the virtual network name you set earlier.

--address-prefixes <aci-subnet-cidr>: Replace this with the CIDR block for your ACI subnet (e.g., 10.0.2.0/24).

For example

az network vnet subnet create \
  --name e6data-subnet-aci \
  --resource-group e6data-app-rg \
  --vnet-name e6data-app-network \
  --address-prefixes 10.0.2.0/24

5. Delegate ACI Subnet

Update ACI Subnet Delegation

To update an existing subnet and delegate it for use by Azure Container Instances (ACI), use the following command:

az network vnet subnet update \
  --name <prefix>-subnet-aci \
  --resource-group <resource-group-name> \
  --vnet-name <prefix>-network \
  --delegations Microsoft.ContainerInstance/containerGroups

Command Breakdown

--name <prefix>-subnet-aci: Replace <prefix> with your identifier.

--resource-group <resource-group-name>: Specify the name of the resource group containing the virtual network.

--vnet-name <prefix>-network: This should be the name of the virtual network that contains the subnet.

--delegations Microsoft.ContainerInstance/containerGroups: This flag updates the subnet to delegate it specifically for Azure Container Instances.

For example

az network vnet subnet update \
  --name e6data-subnet-aci \
  --resource-group e6data-app-rg \
  --vnet-name e6data-app-network \
  --delegations Microsoft.ContainerInstance/containerGroups

Note:

• Delegation is required to allow Azure Container Instances to use the subnet.

• Ensure the subnet is properly configured and does not conflict with other network configurations.

6. Create a Public IP Address

To configure a NAT gateway, you need to create a static public IP address. Follow these steps to create the public IP address required for the NAT gateway:

az network public-ip create \
    --resource-group <resource-group-name> \
    --name <prefix>-PIP \
    --sku Standard \
    --location <region> \
    --allocation-method Static

Command Breakdown

--resource-group <resource-group-name>: Replace <resource-group-name> with the name of your resource group.

--name <prefix>-PIP: Replace <prefix> with a meaningful identifier. --sku Standard: Choose the Standard SKU for the public IP address. This SKU is necessary for NAT gateway integration and offers enhanced features compared to the Basic SKU.

--location <region>: Specify the Azure region where you want to create the public IP address (e.g., eastus, westeurope).

--allocation-method Static: Set the IP address allocation to Static to ensure that the IP address remains constant and does not change.

For example

az network public-ip create \
    --resource-group e6data-app-rg \
    --name e6data-app-pip \
    --sku Standard \
    --location "EastUS" \
    --allocation-method Static

7. Create a NAT Gateway

To set up network address translation (NAT) for outbound traffic, you need to create a NAT gateway and associate it with a public IP address. Follow these steps:

az network nat gateway create \
    --resource-group <resource-group-name> \
    --name <prefix>-nat \
    --public-ip-addresses <prefix>-PIP \
    --idle-timeout 30 \
    --location <region>

Command Breakdown

--resource-group <resource-group-name>: Replace <resource-group-name> with the name of your resource group.

--name <prefix>-nat: Replace <prefix> with your chosen identifier. 
--public-ip-addresses <prefix>-PIP: Specify the name of the public IP address created earlier.

--idle-timeout 30: Set the idle timeout to 30 minutes. This is the amount of time a connection will remain open when idle before being closed. Adjust as needed based on your requirements.

--location <region>: Specify the Azure region where you want to create the NAT gateway (e.g., eastus, westeurope).

For example

az network nat gateway create \
    --resource-group e6data-app-rg \
    --name e6data-app-nat \
    --public-ip-addresses e6data-app-pip \
    --idle-timeout 30 \
    --location "EastUS"

8. Associate the NAT Gateway with the AKS Subnet

To enable outbound connectivity through the NAT gateway for your AKS subnet, follow these steps:

az network vnet subnet update \
    --resource-group <resource-group-name> \
    --vnet-name <prefix>-network \
    --name <prefix>-subnet-aks \
    --nat-gateway <prefix>-nat

Command Breakdown

--resource-group <resource-group-name>: Replace <resource-group-name> with the name of your resource group.

--vnet-name <prefix>-network: Replace <prefix> with your chosen identifier.
 
--name <prefix>-subnet-aks: Replace <prefix> with your identifier.

--nat-gateway <prefix>-nat: Specify the NAT gateway name created earlier.

For example

az network vnet subnet update \
    --resource-group e6data-app-rg \
    --vnet-name e6data-app-network \
    --name e6data-subnet-aks \
    --nat-gateway e6data-app-nat

Note:

• Associating the NAT gateway with the AKS subnet ensures that all outbound traffic from the AKS cluster is routed through the NAT gateway, providing a single, stable IP address for outbound traffic.

• Verify that the NAT gateway and subnet configurations are correctly set up to avoid connectivity issues.

9. Create a Key Vault

Create an Azure Key Vault to securely store certificates used for TLS connectivity. This vault will provide centralized, secure management of certificates, ensuring encrypted communication within your services or applications, such as in an AKS cluster.

az keyvault create \
  --name <vault-name> \
  --resource-group <aks-resource-group-name> \
  --location <region> \
  --sku standard \
  --enable-rbac-authorization true

Command Breakdown

--name <vault-name>: Specifies the name of the Key Vault to be created. Replace <vault-name> with the desired name for your Key Vault.

--resource-group <aks-resource-group-name>: Defines the resource group where the Key Vault will be created. Replace <aks-resource-group-name> with the name of the resource group hosting your AKS cluster or other Azure resources.

--location <region>: Sets the Azure region (data center location) where the Key Vault will be deployed. Replace <region> with your preferred Azure region (e.g., eastus, westeurope).

--sku standard: Defines the pricing tier for the Key Vault. The standard SKU is typically used for most cases. For advanced scenarios, you could use premium.

--enable-rbac-authorization true: Enables role-based access control (RBAC) for managing access to the Key Vault, which provides more granular permissions than the default Key Vault policies. Set to true to enable RBAC.

For example

az keyvault create \
  --name e6data-app-vault \
  --resource-group e6data-app-rg \
  --location "EastUS" \
  --sku standard \
  --enable-rbac-authorization true

10. Creating a New Azure AKS Cluster

Follow these instructions to set up a new Azure Kubernetes Service (AKS) cluster. Ensure that the Azure CLI is installed and configured on your local machine. If you haven’t installed the Azure CLI yet, please refer to the How to install the Azure CLI guide for setup.

1. Open a Terminal or Command Prompt

2. Run the Following Command to Create a New AKS Cluster:

az aks create \
  --resource-group <your-resource-group-name> \
  --name <your-cluster-name> \
  --location <your-region> \
  --kubernetes-version <kube-version> \
  --node-count <default-node-pool-node-count> \
  --node-vm-size <default-node-pool-vm-size> \
  --nodepool-name <default-node-pool-name> \
  --node-os-upgrade-channel none \
  --vnet-subnet-id <aks-subnet-id> \
  --network-plugin azure \
  --network-policy cilium \
  --network-plugin-mode overlay \
  --network-dataplane cilium \
  --enable-aad \
  --aad-admin-group-object-ids <admin-group-object-ids> \
  --enable-managed-identity \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --generate-ssh-keys \
  --aci-subnet-name <aci-subnet-name> \
  --tags <your-tags>

Command Breakdown

--resource-group: Specifies the resource group where the AKS cluster will be created.

--name: Sets the name of the AKS cluster.

--location: Defines the Azure region where the cluster will be deployed.

--kubernetes-version: Specifies the Kubernetes version to use.

--node-count: Sets the number of nodes in the default node pool.

--node-vm-size: Defines the VM size for the nodes.

--nodepool-name: Sets the name of the default node pool.

--node-os-upgrade-channel: Manner in which the OS on your nodes is updated.

--vnet-subnet-id: Specifies the subnet ID where the cluster will be deployed.

--network-plugin azure: Uses Azure CNI for networking.

--network-policy cilium: Enables Cilium for network policy.

--network-plugin-mode overlay: Sets the network plugin mode to overlay.

--network-dataplane cilium: Uses Cilium as the network data plane.

--enable-aad: Enables Azure Active Directory integration.

--aad-admin-group-object-ids: Specifies the AAD group object IDs for cluster admins.

--enable-managed-identity: Uses managed identity for the cluster.

--enable-oidc-issuer: Enables OIDC issuer for the cluster.

--enable-workload-identity: Enables workload identity.

--generate-ssh-keys: Specifies the SSH public key for the Linux nodes.

--aci-subnet-name: Specifies the subnet for Azure Container Instances.

--tags: Adds tags to the AKS cluster.

For detailed instructions and more advanced configurations, refer to the official Azure documentation onQuickstart: Deploy an Azure Kubernetes Service (AKS) cluster using Azure CLI - Azure Kubernetes Service .

Example Command:

az aks create \
  --resource-group e6data-app-rg \
  --name e6data-app-cluster \
  --location "EastUS" \
  --kubernetes-version "1.30" \
  --node-count 3 \
  --node-vm-size Standard_DS2_v2 \
  --nodepool-name e6datapool \
  --node-os-upgrade-channel none \
  --vnet-subnet-id $(az network vnet subnet show \
    --resource-group e6data-app-rg \
    --vnet-name e6data-app-network \
    --name e6data-subnet-aks \
    --query id -o tsv) \
  --network-plugin azure \
  --network-policy cilium \
  --network-plugin-mode overlay \
  --network-dataplane cilium \
  --enable-aad \
  --aad-admin-group-object-ids "abcdedftg-18b7-1234-acc4-ascgrgvvv" \
  --enable-managed-identity \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --generate-ssh-keys \
  --aci-subnet-name e6data-subnet-aci \
  --tags "env=dev" "project=app"

If you haven't already configured Azure AD groups for AKS RBAC, you can refer to the following link for instructions: Configuring groups for Azure AKS with Azure AD RBAC. This will guide you in setting up and managing Azure AD groups for role-based access control within your AKS cluster.

Azure CNI Overlay networking is a prerequisite for using Karpenter in AKS. This networking mode is essential because:

1. It assigns pod IPs from a separate private CIDR, distinct from the VNet.

2. It prevents VNet IP exhaustion, which is crucial for Karpenter's dynamic node scaling.

• Network Configuration: The cluster is configured with the Azure CNI and Cilium for network policy enforcement and data plane management.

• Service and DNS IPs: The service CIDR and DNS service IP should be configured to avoid overlaps with your existing network.

• This configuration requires you to have a Microsoft Entra group for your cluster. This group is registered as an admin group on the cluster to grant admin permissions. If you don't have an existing Microsoft Entra group, you can create one using the az ad group create command.

Important Note:

Here, we are disabling the node OS upgrade channel by setting it to none. This prevents automatic OS upgrades that would restart the nodes in the default node pool, which could result in the bootstrap token rotation. The bootstrap token is used in the environment variables for Karpenter.

If a manual upgrade is initiated, which causes the nodes to restart, it is critical to update the bootstrap token in the Karpenter environment variables to ensure smooth operation and prevent any potential disruptions in scaling.

1. Wait for the cluster creation process to complete. This may take some time.

2. Once the AKS cluster is created, you can retrieve the connection information by running the following command:

az aks get-credentials --resource-group [RESOURCE_GROUP] --name [CLUSTER_NAME]

For Example:

az aks get-credentials \
  --resource-group e6data-app-rg \
  --name e6data-app-cluster

1. Verify the connection to the AKS cluster by running the following command:

kubectl get nodes

This should display the list of nodes in your AKS cluster.

Set up Karpenter

To set up the Karpenter for your AKS cluster, refer to the official Karpenter documentation available on GitHub - Azure/karpenter-provider-azure: AKS Karpenter Provider .

Karpenter has two main components:

• AKSNODECLASS

• NODEPOOL

EC2 NodeClass

NodeClasses in Karpenter act as specialized templates for worker nodes, customized for specific cloud platforms like AKSNodeClasses for Azure. These templates specify essential node configurations, including the operating system image, network security settings, subnet placement, and access permissions.

A. Create an e6data EC2 Node Class

apiVersion: karpenter.azure.com/v1alpha2
kind: AKSNodeClass
metadata:
  name: <NODECLASS_NAME>
  labels:
    app: e6data
    e6data-workspace-name: <WORKSPACE_NAME>
spec:
  imageFamily: AzureLinux
  tags: <TAGS>

NodePool

A single Karpenter NodePool in Azure AKS manages diverse pods, streamlining node management by eliminating the need for multiple node groups. The consolidation policy set to WhenEmpty optimizes costs by removing nodes when they become empty.

B. Create and e6data nodpeool

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: ${nodepool_name}
  labels:
    app: e6data
    e6data-workspace-name: ${workspace_name}  
spec:
  template:
    metadata:
      labels:
        app: e6data
        e6data-workspace-name: ${workspace_name}  
    spec:
      requirements:
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.azure.com/sku-family
          operator: In
          values: ${sku_family}
      nodeClassRef:
        name: ${nodeclass_name}
      taints:
        - key: "e6data-workspace-name"
          value: ${workspace_name}
          effect: NoSchedule  
  limits:
    cpu: ${nodepool_cpu_limits}
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30s

Set up Nginx Ingress Controller

An ingress controller is required in the AKS cluster to manage external access to services, particularly for connectivity between the e6data Console and e6data Cluster, as well as for providing connectivity between querying/BI tools and the e6data Query Engine.

To install the NGINX Ingress Controller in your Azure Kubernetes Service (AKS) cluster, follow these steps:

1. Add the NGINX Ingress Controller Helm repository:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

1. Install the NGINX Ingress Controller using Helm:

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace kube-system \
  --create-namespace \
  --set controller.service.annotations."service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"=/healthz \
  --set controller.service.externalTrafficPolicy=Local

Replace <nginx-ingress-namespace> with your desired namespace.

1. Wait for the NGINX Ingress Controller to be fully deployed:

kubectl wait --namespace <nginx-ingress-namespace> \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=120s

1. Create a dummy Ingress resource to ensure the controller is working:

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dummy-ingress
  namespace: <your-namespace>
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  ingressClassName: nginx
  rules:
  - host: dummy.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dummy-service
            port: 
              number: 80
EOF

Replace <your-namespace> with the namespace where you want to create the dummy Ingress.

1. Verify the Ingress resource was created:

kubectl get ingress -n <your-namespace>

Deploying Azure Key Vault to Kubernetes (akv2k8s) using Helm

The akv2k8s tool is essential for e6data's secure operation in AKS. It provides a seamless and secure method to access Azure Key Vault resources within the Kubernetes environment. Specifically for e6data:

• TLS Connectivity: akv2k8s allows e6data to retrieve TLS certificates stored in Azure Key Vault, ensuring secure communications.

• Gateway Connectivity: It facilitates the acquisition of domain certificates from Azure Key Vault, necessary for establishing gateway connections to the e6data cluster.

The following section provides a step-by-step guide to deploying the akv2k8s (Azure Key Vault to Kubernetes) Helm chart into your Azure Kubernetes Service (AKS) cluster. This deployment allows seamless integration between Azure Key Vault and Kubernetes, enabling your workloads to securely fetch secrets directly from Azure Key Vault.

Prerequisites

Before starting the deployment, ensure the following prerequisites are met:

• Helm Installed: Helm should be installed on your local machine. You can verify this by running helm version.

• Kubeconfig Access: Ensure you have access to your Kubernetes cluster via your kubeconfig file, typically located at ~/.kube/config.

• Kubernetes Access: You need kubectl installed and configured to interact with your AKS cluster.Install Tools

Step-by-Step Deployment Instructions

Step 1: Add the Helm Repository

Start by adding the Helm repository containing the akv2k8s chart:

helm repo add spv-charts http://charts.spvapi.no

This command adds the spv-charts repository to Helm, where the akv2k8s chart is hosted.

Step 2: Update Helm Repositories

Next, update your Helm repositories to ensure you have access to the latest charts:

helm repo update

Step 3: Install the akv2k8s Chart

Install the akv2k8s chart into the kube-system namespace of your AKS cluster:

helm install akv2k8s spv-charts/akv2k8s --namespace kube-system

• akv2k8s: The release name for this deployment.

• spv-charts/akv2k8s: Specifies the repository and chart name.

• --namespace kube-system: Deploys the release into the kube-system namespace.

Step 4: Verify the Installation

To confirm the successful installation of the chart, list the Helm releases in the kube-system namespace:

helm list --namespace kube-system

You should see the akv2k8s release listed among your installed Helm charts.

Step 5: Monitor the Pods

Check the status of the pods created by the akv2k8s deployment:

kubectl get pods -n kube-system

This command will show the running status of the pods related to akv2k8s in your cluster.

Summary of Commands

helm repo add spv-charts <http://charts.spvapi.no>
helm repo update
helm install akv2k8s spv-charts/akv2k8s --namespace kube-system
helm list --namespace kube-system
kubectl get pods -n kube-system

If a Key Vault is not already present, you can follow the official Microsoft documentation to create one via the Azure portal: Quickstart: Create a Key Vault using the Azure portal.