Technical post incoming!

On the last project I needed to deploy a multi-node kubernetes cluster as a proof of concept in Azure. Now, we use k3s, a lightweight kubernetes distribution over ubuntu vms deployed in Azure. This is to simulate an edge environment, so this is not something you would do in production, this is for dev/test purposes or just learning/upskilling.

You can find deep explanations about what is a multi-node kubernetes cluster, and how to deploy one for k3s. So I will focus in the recipe, but feel free to go away from this blog to learn more. Internet is a vast and open space, and this blog is not a wallen garden.

We are going to use az cli here, which should work in Linux/Windows/Mac. I will use bash and WSL.

1. Az login

Loging into your azure account and select the subscription.

az login

2. Set environment variables

We are going to use a set of environment variables to make easier to execute the commands, change them at will:

export PROJECT_NAME="k3smultinode"
export RESOURCE_GROUP="rg-$PROJECT_NAME"
export LOCATION=westus2
export VNET="vnet-$PROJECT_NAME"
export SUBNET="subnet-$PROJECT_NAME"
export PUBLIC_IP_NODE1="subnet-$PROJECT_NAME-01"
export PUBLIC_IP_NODE2="subnet-$PROJECT_NAME-02"
export VM_NAME_NODE1="vm-$PROJECT_NAME-01"
export VM_NAME_NODE2="vm-$PROJECT_NAME-02"
export HOST_IP=$(curl -s https://checkip.amazonaws.com)
export K3S_TOKEN=<mysecret-replaceme>
export USER_ID=$(az ad signed-in-user show --query id -o tsv)

3. Create a Resource Group

az group create --name $RESOURCE_GROUP --location $LOCATION

4. Create an ACR (optional)

Optional if you want to push images into your cluster

Create a VNET and Subnet

In this virtual network, all the nodes will communicate internally.

az network vnet create --resource-group $RESOURCE_GROUP \
    --name $VNET \
    --address-prefix 10.0.0.0/16 \
    --subnet-name $SUBNET \
    --subnet-prefix 10.0.0.0/24

Create public IPs

If you want to connect from your computer to the nodes, this is necessary, you can also connect from inside Azure portal if you can’t create a public ip:

az network public-ip create --resource-group $RESOURCE_GROUP --name $PUBLIC_IP_NODE1 --sku Standard
az network public-ip create --resource-group $RESOURCE_GROUP --name $PUBLIC_IP_NODE2 --sku Standard

Create VMs

Creating the debian VMs

az vm create --resource-group $RESOURCE_GROUP \
    --name $VM_NAME_NODE1 \
    --image debian:debian-11:11:latest \
    --admin-username azureuser --generate-ssh-keys \
    --vnet-name $VNET --subnet $SUBNET \
    --public-ip-address $PUBLIC_IP_NODE1 \
    --nsg-rule NONE --size Standard_D8s_v3

az vm create --resource-group $RESOURCE_GROUP \
    --name $VM_NAME_NODE2 \
    --image debian:debian-11:11:latest \
    --admin-username azureuser --generate-ssh-keys \
    --vnet-name $VNET --subnet $SUBNET \
    --public-ip-address $PUBLIC_IP_NODE1 \
    --nsg-rule NONE --size Standard_D8s_v3

Set auto shutdown (optional)

Optional but recommended, you do not want Azure to send you a huge bill because you forgot to turn-off your vms.

az vm auto-shutdown --resource-group $RESOURCE_GROUP --name $VM_NAME_NODE1 --time 0800
az vm auto-shutdown --resource-group $RESOURCE_GROUP --name $VM_NAME_NODE2 --time 0800

Allow SSH from your host computer

Your security department may not be very happy with this:

az network nsg rule create --resource-group $RESOURCE_GROUP \
    --nsg-name "${VM_NAME_NODE1}NSG" \
    --name AllowSSHFromHost --protocol Tcp \
    --direction Inbound --priority 100 --source-address-prefix $HOST_IP \
    --source-port-range '*' --destination-address-prefix '*' --destination-port-range 22 --access Allow

az network nsg rule create --resource-group $RESOURCE_GROUP \
    --nsg-name "${VM_NAME_NODE2}NSG" \
    --name AllowSSHFromHost --protocol Tcp \
    --direction Inbound --priority 100 --source-address-prefix $HOST_IP \
    --source-port-range '*' --destination-address-prefix '*' --destination-port-range 22 --access Allow

Configure SSH Config

Optional, but nice to connect to the vms by name:

vmPublicIp=$(az network public-ip show --resource-group $RESOURCE_GROUP --name $PUBLIC_IP_NODE1 --query ipAddress --output tsv)

# Configure SSH to connect to the VM
echo "Host $VM_NAME_NODE1
HostName $vmPublicIp
User azureuser
IdentityFile ~/.ssh/id_rsa
StrictHostKeyChecking no

" >> ~/.ssh/config

vmPublicIp=$(az network public-ip show --resource-group $RESOURCE_GROUP --name $PUBLIC_IP_NODE2 --query ipAddress --output tsv)

# Configure SSH to connect to the VM
echo "Host $VM_NAME_NODE2
HostName $vmPublicIp
User azureuser
IdentityFile ~/.ssh/id_rsa
StrictHostKeyChecking no

" >> ~/.ssh/config

Prepare the machine for k3s

Prepare the machine for k3s, make sure to check if I am not injecting any malware to your vm before executing the script.

ssh $VM_NAME_NODE1 'curl -sL <postCreateCommand> | bash' 
ssh $VM_NAME_NODE2 'curl -sL <install-docker> | bash'

Restart the VMS

Some changes require restart, but it is pretty fast:

az vm restart --resource-group $RESOURCE_GROUP --name $VM_NAME_NODE1
az vm restart --resource-group $RESOURCE_GROUP --name $VM_NAME_NODE2

Install k3s

Install the main node:

ssh $VM_NAME_NODE1 "curl -sfL https://get.k3s.io | K3S_TOKEN=$K3S_TOKEN sh -s - server --cluster-init"

Install the second node:

ssh $VM_NAME_NODE1 "curl -sfL https://get.k3s.io | K3S_TOKEN=$K3S_TOKEN sh -s - server --cluster-init --server https://$VM_NODE2_INTERNALIP:6443"

Configure kubectl inside machines:

ssh $VM_NAME_NODE1 "curl -sfL <replaceme> | bash"
ssh $VM_NAME_NODE2 "curl -sfL <replaceme> | bash"

Done, if you ssh in either of the machines you can see a multi-node k3s cluster for dev/test:

ssh $VM_NAME_NODE1 'kubectl get nodes -o wide'
ssh $VM_NAME_NODE1 'kubectl get nodes -o wide'

Next steps