The following components are required before setting up the infrastructure needed by e6data. These are commonly present in most cloud environments, but if any are not present, please follow the linked guides below to create them.
Create VNET, SUBNETS AND NAT Gateway
AKS Cluster
1. Prerequisites
Ensure that you have the Azure CLI installed on your system. You can install it from here.
Once installed, log in to your Azure account using the following command:
az login
2. Create Resource Group
In Azure, a resource group acts as a logical container that holds related resources for your solution. It allows you to manage, deploy, and organize resources conveniently. To create a resource group, use the following Azure CLI command:
az group create \
--name <resource-group-name> \
--location <region>
Command Breakdown
--name <resource-group-name>: The name of the resource group you want to create. This name should be relevant to your project or environment.
--location <region>: The Azure region where your resource group will be created. The region determines the physical location of the resources in the group. Example regions include eastus, westeurope, or southeastasia.
For example
az group create \
--name e6data-app-rg \
--location "EastUS"
3. Create a Virtual Network
After creating a resource group, the next step is to create a Virtual Network (VNet) within that group. A VNet is an essential part of Azure networking and allows you to manage your network resources efficiently.
--name <prefix>-network: Replace <prefix> with a meaningful identifier to name your VNet.
--resource-group <resource-group-name>: Specify the resource group where the VNet will be created. This should be the name of an existing resource group.
--address-prefix <cidr-block>: Specify the address range for the VNet in CIDR notation (e.g., 10.0.0.0/16).
--location <region>: Specify the Azure region for the VNet. Replace <region> with a region like eastus, westeurope, or centralindia.
--name <prefix>-subnet-aks: Replace <prefix> with a meaningful identifier that you’re using throughout your setup.
--resource-group <resource-group-name>: Specify the name of the resource group you created earlier.
--vnet-name <prefix>-network: This should match the name of the VNet you created earlier.
--address-prefixes <aks-subnet-cidr>: Replace this with the CIDR block for your AKS subnet (e.g., 10.0.1.0/24).
--name <prefix>-subnet-aci: Replace <prefix> with your chosen identifier.
--resource-group <resource-group-name>: Specify the resource group name created earlier.
--vnet-name <prefix>-network: This should match the virtual network name you set earlier.
--address-prefixes <aci-subnet-cidr>: Replace this with the CIDR block for your ACI subnet (e.g., 10.0.2.0/24).
--name <prefix>-subnet-aci: Replace <prefix> with your identifier.
--resource-group <resource-group-name>: Specify the name of the resource group containing the virtual network.
--vnet-name <prefix>-network: This should be the name of the virtual network that contains the subnet.
--delegations Microsoft.ContainerInstance/containerGroups: This flag updates the subnet to delegate it specifically for Azure Container Instances.
Delegation is required to allow Azure Container Instances to use the subnet.
Ensure the subnet is properly configured and does not conflict with other network configurations.
6. Create a Public IP Address
To configure a NAT gateway, you need to create a static public IP address. Follow these steps to create the public IP address required for the NAT gateway:
az network public-ip create \
--resource-group <resource-group-name> \
--name <prefix>-PIP \
--sku Standard \
--location <region> \
--allocation-method Static
Command Breakdown
--resource-group <resource-group-name>: Replace <resource-group-name> with the name of your resource group.
--name <prefix>-PIP: Replace <prefix> with a meaningful identifier. --sku Standard: Choose the Standard SKU for the public IP address. This SKU is necessary for NAT gateway integration and offers enhanced features compared to the Basic SKU.
--location <region>: Specify the Azure region where you want to create the public IP address (e.g., eastus, westeurope).
--allocation-method Static: Set the IP address allocation to Static to ensure that the IP address remains constant and does not change.
For example
az network public-ip create \
--resource-group e6data-app-rg \
--name e6data-app-pip \
--sku Standard \
--location "EastUS" \
--allocation-method Static
7. Create a NAT Gateway
To set up network address translation (NAT) for outbound traffic, you need to create a NAT gateway and associate it with a public IP address. Follow these steps:
--resource-group <resource-group-name>: Replace <resource-group-name> with the name of your resource group.
--name <prefix>-nat: Replace <prefix> with your chosen identifier.
--public-ip-addresses <prefix>-PIP: Specify the name of the public IP address created earlier.
--idle-timeout 30: Set the idle timeout to 30 minutes. This is the amount of time a connection will remain open when idle before being closed. Adjust as needed based on your requirements.
--location <region>: Specify the Azure region where you want to create the NAT gateway (e.g., eastus, westeurope).
--resource-group <resource-group-name>: Replace <resource-group-name> with the name of your resource group.
--vnet-name <prefix>-network: Replace <prefix> with your chosen identifier.
--name <prefix>-subnet-aks: Replace <prefix> with your identifier.
--nat-gateway <prefix>-nat: Specify the NAT gateway name created earlier.
Associating the NAT gateway with the AKS subnet ensures that all outbound traffic from the AKS cluster is routed through the NAT gateway, providing a single, stable IP address for outbound traffic.
Verify that the NAT gateway and subnet configurations are correctly set up to avoid connectivity issues.
9. Create a Key Vault
Create an Azure Key Vault to securely store certificates used for TLS connectivity. This vault will provide centralized, secure management of certificates, ensuring encrypted communication within your services or applications, such as in an AKS cluster.
az keyvault create \
--name <vault-name> \
--resource-group <aks-resource-group-name> \
--location <region> \
--sku standard \
--enable-rbac-authorization true
Command Breakdown
--name <vault-name>: Specifies the name of the Key Vault to be created. Replace <vault-name> with the desired name for your Key Vault.
--resource-group <aks-resource-group-name>: Defines the resource group where the Key Vault will be created. Replace <aks-resource-group-name> with the name of the resource group hosting your AKS cluster or other Azure resources.
--location <region>: Sets the Azure region (data center location) where the Key Vault will be deployed. Replace <region> with your preferred Azure region (e.g., eastus, westeurope).
--sku standard: Defines the pricing tier for the Key Vault. The standard SKU is typically used for most cases. For advanced scenarios, you could use premium.
--enable-rbac-authorization true: Enables role-based access control (RBAC) for managing access to the Key Vault, which provides more granular permissions than the default Key Vault policies. Set to true to enable RBAC.
For example
az keyvault create \
--name e6data-app-vault \
--resource-group e6data-app-rg \
--location "EastUS" \
--sku standard \
--enable-rbac-authorization true
10. Creating a New Azure AKS Cluster
Follow these instructions to set up a new Azure Kubernetes Service (AKS) cluster. Ensure that the Azure CLI is installed and configured on your local machine. If you haven’t installed the Azure CLI yet, please refer to the How to install the Azure CLI guide for setup.
Open a Terminal or Command Prompt
Run the Following Command to Create a New AKS Cluster:
--resource-group: Specifies the resource group where the AKS cluster will be created.
--name: Sets the name of the AKS cluster.
--location: Defines the Azure region where the cluster will be deployed.
--kubernetes-version: Specifies the Kubernetes version to use.
--node-count: Sets the number of nodes in the default node pool.
--node-vm-size: Defines the VM size for the nodes.
--nodepool-name: Sets the name of the default node pool.
--node-os-upgrade-channel: Manner in which the OS on your nodes is updated.
--vnet-subnet-id: Specifies the subnet ID where the cluster will be deployed.
--network-plugin azure: Uses Azure CNI for networking.
--network-policy cilium: Enables Cilium for network policy.
--network-plugin-mode overlay: Sets the network plugin mode to overlay.
--network-dataplane cilium: Uses Cilium as the network data plane.
--enable-aad: Enables Azure Active Directory integration.
--aad-admin-group-object-ids: Specifies the AAD group object IDs for cluster admins.
--enable-managed-identity: Uses managed identity for the cluster.
--enable-oidc-issuer: Enables OIDC issuer for the cluster.
--enable-workload-identity: Enables workload identity.
--generate-ssh-keys: Specifies the SSH public key for the Linux nodes.
--aci-subnet-name: Specifies the subnet for Azure Container Instances.
--tags: Adds tags to the AKS cluster.
If you haven't already configured Azure AD groups for AKS RBAC, you can refer to the following link for instructions: Configuring groups for Azure AKS with Azure AD RBAC. This will guide you in setting up and managing Azure AD groups for role-based access control within your AKS cluster.
Azure CNI Overlay networking is a prerequisite for using Karpenter in AKS. This networking mode is essential because:
It assigns pod IPs from a separate private CIDR, distinct from the VNet.
It prevents VNet IP exhaustion, which is crucial for Karpenter's dynamic node scaling.
Network Configuration: The cluster is configured with the Azure CNI and Cilium for network policy enforcement and data plane management.
Service and DNS IPs: The service CIDR and DNS service IP should be configured to avoid overlaps with your existing network.
This configuration requires you to have a Microsoft Entra group for your cluster. This group is registered as an admin group on the cluster to grant admin permissions. If you don't have an existing Microsoft Entra group, you can create one using the az ad group create command.
Important Note:
Here, we are disabling the node OS upgrade channel by setting it to none. This prevents automatic OS upgrades that would restart the nodes in the default node pool, which could result in the bootstrap token rotation. The bootstrap token is used in the environment variables for Karpenter.
If a manual upgrade is initiated, which causes the nodes to restart, it is critical to update the bootstrap token in the Karpenter environment variables to ensure smooth operation and prevent any potential disruptions in scaling.
Wait for the cluster creation process to complete. This may take some time.
Once the AKS cluster is created, you can retrieve the connection information by running the following command:
az aks get-credentials --resource-group [RESOURCE_GROUP] --name [CLUSTER_NAME]
For Example:
az aks get-credentials \
--resource-group e6data-app-rg \
--name e6data-app-cluster
Verify the connection to the AKS cluster by running the following command:
kubectl get nodes
This should display the list of nodes in your AKS cluster.
Set up Karpenter
Karpenter has two main components:
AKSNODECLASS
NODEPOOL
EC2 NodeClass
NodeClasses in Karpenter act as specialized templates for worker nodes, customized for specific cloud platforms like AKSNodeClasses for Azure. These templates specify essential node configurations, including the operating system image, network security settings, subnet placement, and access permissions.
A single Karpenter NodePool in Azure AKS manages diverse pods, streamlining node management by eliminating the need for multiple node groups. The consolidation policy set to WhenEmpty optimizes costs by removing nodes when they become empty.
An ingress controller is required in the AKS cluster to manage external access to services, particularly for connectivity between the e6data Console and e6data Cluster, as well as for providing connectivity between querying/BI tools and the e6data Query Engine.
To install the NGINX Ingress Controller in your Azure Kubernetes Service (AKS) cluster, follow these steps:
Replace <your-namespace> with the namespace where you want to create the dummy Ingress.
Verify the Ingress resource was created:
kubectl get ingress -n <your-namespace>
Deploying Azure Key Vault to Kubernetes (akv2k8s) using Helm
The akv2k8s tool is essential for e6data's secure operation in AKS. It provides a seamless and secure method to access Azure Key Vault resources within the Kubernetes environment. Specifically for e6data:
TLS Connectivity: akv2k8s allows e6data to retrieve TLS certificates stored in Azure Key Vault, ensuring secure communications.
Gateway Connectivity: It facilitates the acquisition of domain certificates from Azure Key Vault, necessary for establishing gateway connections to the e6data cluster.
The following section provides a step-by-step guide to deploying the akv2k8s (Azure Key Vault to Kubernetes) Helm chart into your Azure Kubernetes Service (AKS) cluster. This deployment allows seamless integration between Azure Key Vault and Kubernetes, enabling your workloads to securely fetch secrets directly from Azure Key Vault.
Prerequisites
Before starting the deployment, ensure the following prerequisites are met:
Helm Installed: Helm should be installed on your local machine. You can verify this by running helm version.
Kubeconfig Access: Ensure you have access to your Kubernetes cluster via your kubeconfig file, typically located at ~/.kube/config.
Step-by-Step Deployment Instructions
Step 1: Add the Helm Repository
Start by adding the Helm repository containing the akv2k8s chart:
helm repo add spv-charts http://charts.spvapi.no
This command adds the spv-charts repository to Helm, where the akv2k8s chart is hosted.
Step 2: Update Helm Repositories
Next, update your Helm repositories to ensure you have access to the latest charts:
helm repo update
Step 3: Install the akv2k8s Chart
Install the akv2k8s chart into the kube-system namespace of your AKS cluster: