Prerequisite Infrastructure

The following components are required before setting up the infrastructure needed by e6data. These are commonly present in most cloud environments, but if any are not present, please follow the linked guides below to create them.

  1. Create VNET, SUBNETS AND NAT Gateway

  2. AKS Cluster

1. Prerequisites

Ensure you have the Azure CLI installed and you are logged in.

az login

2. Create Resource Group

In Azure, a resource group acts as a logical container that holds related resources for your solution. It allows you to manage, deploy, and organize resources conveniently. To create a resource group, use the following Azure CLI command:

az group create \
  --name <resource-group-name> \
  --location <region>
Command Breakdown
--name <resource-group-name>: The name of the resource group you want to create. This name should be relevant to your project or environment. 

--location <region>: The Azure region where your resource group will be created. The region determines the physical location of the resources in the group. Example regions include eastus, westeurope, or southeastasia.

3. Create a Virtual Network

After creating a resource group, the next step is to create a Virtual Network (VNet) within that group. A VNet is an essential part of Azure networking and allows you to manage your network resources efficiently.

az network vnet create \
  --name <prefix>-network \
  --resource-group <resource-group-name> \
  --address-prefix <cidr-block> \
  --location <region>
Command Breakdown
--name <prefix>-network: Replace <prefix> with a meaningful identifier to name your VNet. 

--resource-group <resource-group-name>: Specify the resource group where the VNet will be created. This should be the name of an existing resource group.
 
--address-prefix <cidr-block>: Specify the address range for the VNet in CIDR notation (e.g., 10.0.0.0/16).

--location <region>: Specify the Azure region for the VNet. Replace <region> with a region like eastus, westeurope, or centralindia.

4. Create Subnets

Create AKS Subnet

To create a subnet specifically for Azure Kubernetes Service (AKS), use the following command: 

az network vnet subnet create \
  --name <prefix>-subnet-aks \
  --resource-group <resource-group-name> \
  --vnet-name <prefix>-network \
  --address-prefixes <aks-subnet-cidr>
Command Breakdown
--name <prefix>-subnet-aks: Replace <prefix> with a meaningful identifier that you’re using throughout your setup. 

--resource-group <resource-group-name>: Specify the name of the resource group you created earlier.

--vnet-name <prefix>-network: This should match the name of the VNet you created earlier.

--address-prefixes <aks-subnet-cidr>: Replace this with the CIDR block for your AKS subnet (e.g., 10.0.1.0/24).

Create ACI Subnet

To create a subnet specifically for Azure Container Instances (ACI), use the following command:

az network vnet subnet create \
  --name <prefix>-subnet-aci \
  --resource-group <resource-group-name> \
  --vnet-name <prefix>-network \
  --address-prefixes <aci-subnet-cidr>
Command Breakdown
--name <prefix>-subnet-aci: Replace <prefix> with your chosen identifier. 

--resource-group <resource-group-name>: Specify the resource group name created earlier.

--vnet-name <prefix>-network: This should match the virtual network name you set earlier.

--address-prefixes <aci-subnet-cidr>: Replace this with the CIDR block for your ACI subnet (e.g., 10.0.2.0/24).

5. Delegate ACI Subnet

Update ACI Subnet Delegation

To update an existing subnet and delegate it for use by Azure Container Instances (ACI), use the following command:

az network vnet subnet update \
  --name <prefix>-subnet-aci \
  --resource-group <resource-group-name> \
  --vnet-name <prefix>-network \
  --delegations Microsoft.ContainerInstance/containerGroups
Command Breakdown
--name <prefix>-subnet-aci: Replace <prefix> with your identifier.

--resource-group <resource-group-name>: Specify the name of the resource group containing the virtual network.

--vnet-name <prefix>-network: This should be the name of the virtual network that contains the subnet.

--delegations Microsoft.ContainerInstance/containerGroups: This flag updates the subnet to delegate it specifically for Azure Container Instances.

Note:

  • Delegation is required to allow Azure Container Instances to use the subnet.

  • Ensure that the subnet is properly configured and does not conflict with other network configurations.

6. Create a Public IP Address

To configure a NAT gateway, you need to create a static public IP address. Follow these steps to create the public IP address required for the NAT gateway:

az network public-ip create \
    --resource-group <resource-group-name> \
    --name <prefix>-PIP \
    --sku Standard \
    --location <region> \
    --allocation-method Static
Command Breakdown
--resource-group <resource-group-name>: Replace <resource-group-name> with the name of your resource group.

--name <prefix>-PIP: Replace <prefix> with a meaningful identifier. --sku Standard: Choose the Standard SKU for the public IP address. This SKU is necessary for NAT gateway integration and offers enhanced features compared to the Basic SKU.

--location <region>: Specify the Azure region where you want to create the public IP address (e.g., eastus, westeurope).

--allocation-method Static: Set the IP address allocation to Static to ensure that the IP address remains constant and does not change.

7. Create a NAT Gateway

To set up network address translation (NAT) for outbound traffic, you need to create a NAT gateway and associate it with a public IP address. Follow these steps:

az network nat gateway create \
    --resource-group <resource-group-name> \
    --name <prefix>-nat \
    --public-ip-addresses <prefix>-PIP \
    --idle-timeout 30 \
    --location <region>
Command Breakdown
--resource-group <resource-group-name>: Replace <resource-group-name> with the name of your resource group.

--name <prefix>-nat: Replace <prefix> with your chosen identifier. 
--public-ip-addresses <prefix>-PIP: Specify the name of the public IP address created earlier.

--idle-timeout 30: Set the idle timeout to 30 minutes. This is the amount of time a connection will remain open when idle before being closed. Adjust as needed based on your requirements.

--location <region>: Specify the Azure region where you want to create the NAT gateway (e.g., eastus, westeurope).

8. Associate the NAT Gateway with the AKS Subnet

To enable outbound connectivity through the NAT gateway for your AKS subnet, follow these steps:

az network vnet subnet update \
    --resource-group <resource-group-name> \
    --vnet-name <prefix>-network \
    --name <prefix>-subnet-aks \
    --nat-gateway <prefix>-nat
Command Breakdown
--resource-group <resource-group-name>: Replace <resource-group-name> with the name of your resource group.

--vnet-name <prefix>-network: Replace <prefix> with your chosen identifier.
 
--name <prefix>-subnet-aks: Replace <prefix> with your identifier.

--nat-gateway <prefix>-nat: Specify the NAT gateway name created earlier.

Note:

  • Associating the NAT gateway with the AKS subnet ensures that all outbound traffic from the AKS cluster is routed through the NAT gateway, providing a single, stable IP address for outbound traffic.

  • Verify that the NAT gateway and subnet configurations are correctly set up to avoid connectivity issues.

9. Creating a New Azure AKS Cluster

To set up a new Azure Kubernetes Service (AKS) cluster, follow these instructions. Ensure that the Azure CLI is installed and configured on your local machine. If you haven’t installed the Azure CLI yet, please refer to the How to install the Azure CLI guide for setup.

  1. Open a Terminal or Command Prompt

  2. Run the Following Command to Create a New AKS Cluster:

az aks create \
  --resource-group <your-resource-group-name> \
  --name <your-cluster-name> \
  --location <your-region> \
  --kubernetes-version <kube-version> \
  --node-count <node-count> \
  --node-vm-size <node-vm-size> \
  --nodepool-name <nodepool-name> \
  --vnet-subnet-id <vnet-subnet-id> \
  --network-plugin azure \
  --network-policy cilium \
  --network-plugin-mode overlay \
  --network-dataplane cilium \
  --enable-aad \
  --aad-admin-group-object-ids <aad-admin-group-object-ids> \
  --enable-managed-identity \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --generate-ssh-keys \
  --aci-subnet-name <aci-subnet-name> \
  --enable-private-cluster \
  --tags <your-tags> \
  --service-cidr <service-cidr> \
  --dns-service-ip <dns-service-ip>
Command Breakdown
--resource-group <your-resource-group-name>: Replace <your-resource-group-name> with the name of your resource group.

--name <your-cluster-name>: Replace <your-cluster-name> with the name you want to assign to your AKS cluster.

--location <your-region>: Specify the Azure region where you want to create the AKS cluster (e.g., eastus, westeurope).

--kubernetes-version <kube-version>: Replace <kube-version> with the Kubernetes version you wish to use (e.g., 1.24.6).

--node-count <default-node-pool-node-count>: Set the number of nodes for the default node pool (e.g., 3).

--node-vm-size <default-node-pool-vm-size>: Specify the VM size for the nodes in the default pool (e.g., Standard_DS2_v2).

--nodepool-name <default-node-pool-name>: Provide a name for the default node pool (e.g., defaultpool).

--vnet-subnet-id <aks-subnet-id>: Replace <aks-subnet-id> with the ID of the subnet in your virtual network where the AKS cluster will be deployed.

--network-plugin azure: Specifies Azure's network plugin for integration with Azure networking features.

--network-policy cilium: Sets the network policy to cilium, enabling advanced networking capabilities.

--network-plugin-mode overlay: Configures the network plugin to use overlay mode for network isolation.

--network-dataplane cilium: Specifies cilium as the data plane for network policies and security.

--enable-aad: Enables Azure Active Directory integration for AKS.

--aad-admin-group-object-ids <admin-group-object-ids>: Replace <admin-group-object-ids> with the object IDs of the AAD groups that will have admin access to the AKS cluster.

--enable-managed-identity: Enables managed identity for the AKS cluster, allowing it to access Azure resources securely.

--enable-oidc-issuer: Enables OpenID Connect (OIDC) issuer for authentication with external services.

--enable-workload-identity: Allows workloads in the AKS cluster to use Azure AD identities.

--ssh-key-value <path-to-public-ssh-key>: Specify the path to your SSH public key for secure access to the cluster nodes.

--aci-subnet-name <aci-subnet-name>: Replace <aci-subnet-name> with the name of the subnet used for Azure Container Instances (ACI).

--enable-private-cluster: Creates a private AKS cluster with no public IP addresses.

--tags <your-tags>: Add any tags to organize and manage your resources (e.g., environment=dev).

--dns-service-ip: An IP address assigned to the Kubernetes DNS service. This address must be within the Kubernetes service address range specified by "--service-cidr". For example, 10.0.0.10

--service-cidr: A CIDR notation IP range from which to assign service cluster IPs.This range must not overlap with any Subnet IP ranges. For example, 10.0.0.0/16.

Notes:

  • Private Cluster: This setup creates a private AKS cluster, enhancing security by restricting public access.

  • Network Configuration: The cluster is configured with the Azure CNI and Cilium for network policy enforcement and data plane management.

  • Service and DNS IPs: The service CIDR and DNS service IP should be configured to avoid overlaps with your existing network.

  • This configuration requires you to have a Microsoft Entra group for your cluster. This group is registered as an admin group on the cluster to grant admin permissions. If you don't have an existing Microsoft Entra group, you can create one using the az ad group create command.

Set up Karpenter

Karpenter has two main components:

  • AKSNODECLASS

  • NODEPOOL

EC2 NodeClass

NodeClasses in Karpenter act as specialized templates for worker nodes, customized for specific cloud platforms like AKSNodeClasses for Azure. These templates specify essential node configurations, including the operating system image, network security settings, subnet placement, and access permissions.

A. Create an e6data EC2 Node Class

apiVersion: karpenter.azure.com/v1alpha2
kind: AKSNodeClass
metadata:
  name: <NODECLASS_NAME>
  labels:
    app: e6data
    e6data-workspace-name: <WORKSPACE_NAME>
spec:
  imageFamily: AzureLinux
  tags: <TAGS>

NodePool

A single Karpenter NodePool in Azure AKS manages diverse pods, streamlining node management by eliminating the need for multiple node groups. The consolidation policy set to WhenEmpty optimizes costs by removing nodes when they become empty.

B. Create and e6data nodpeool

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: ${nodepool_name}
  labels:
    app: e6data
    e6data-workspace-name: ${workspace_name}  
spec:
  template:
    metadata:
      labels:
        app: e6data
        e6data-workspace-name: ${workspace_name}  
    spec:
      requirements:
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.azure.com/sku-family
          operator: In
          values: ${sku_family}
      nodeClassRef:
        name: ${nodeclass_name}
      taints:
        - key: "e6data-workspace-name"
          value: ${workspace_name}
          effect: NoSchedule  
  limits:
    cpu: ${nodepool_cpu_limits}
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30s

Set up Nginx Ingress Controller

An ingress controller is required in the AKS cluster to manage external access to services, particularly for connectivity between the e6data Console and e6data Cluster, as well as for providing connectivity between querying/BI tools and the e6data Query Engine.

To install the NGINX Ingress Controller in your Azure Kubernetes Service (AKS) cluster, follow these steps:

  1. Add the NGINX Ingress Controller Helm repository:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

  1. Install the NGINX Ingress Controller using Helm:

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace kube-system \
  --create-namespace \
  --set controller.service.annotations."service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"=/healthz \
  --set controller.service.externalTrafficPolicy=Local

Replace <nginx-ingress-namespace> with your desired namespace.

  1. Wait for the NGINX Ingress Controller to be fully deployed:

kubectl wait --namespace <nginx-ingress-namespace> \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=120s

  1. Create a dummy Ingress resource to ensure the controller is working:

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dummy-ingress
  namespace: <your-namespace>
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  ingressClassName: nginx
  rules:
  - host: dummy.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dummy-service
            port: 
              number: 80
EOF

Replace <your-namespace> with the namespace where you want to create the dummy Ingress.

  1. Verify the Ingress resource was created:

kubectl get ingress -n <your-namespace>

Azure Key Vault to Kubernetes

The akv2k8s tool is essential for e6data's secure operation in AKS. It provides a seamless and secure method to access Azure Key Vault resources within the Kubernetes environment. Specifically for e6data:

  1. TLS Connectivity(attach links): akv2k8s allows e6data to retrieve TLS certificates stored in Azure Key Vault, ensuring secure communications.

  2. Gateway Connectivity(attach links): It facilitates the acquisition of domain certificates from Azure Key Vault, necessary for establishing gateway connections to the e6data cluster.

Here are the steps to install the Azure Key Vault to Kubernetes (akv2k8s) using Helm:

  1. Add the akv2k8s Helm repository:

helm repo add spv-charts <http://charts.spvapi.no>
helm repo update

  1. Install the akv2k8s Helm chart:

helm install akv2k8s spv-charts/akv2k8s \
  --namespace kube-system \
  --create-namespace

  1. Verify the installation:

kubectl get pods -n kube-system | grep akv2k8s

If a Key Vault is not already present, you can follow the official Microsoft documentation to create one via the Azure portal: Quickstart: Create a Key Vault using the Azure portal.

PreviousAzure SetupNextInfrastructure & Permissions for e6data

Last updated