Creating a Kubernetes Cluster on DigitalOcean with Python and Fabric

Posted by Michael Herman Michael Herman Last updated October 18th, 2021

In this tutorial, we'll spin up a three-node Kubernetes cluster using Ubuntu 20.04 DigitalOcean droplets. We'll also look at how to automate the set up of a Kubernetes cluster with Python and Fabric.

Feel free to swap out DigitalOcean for a different cloud hosting provider or your own on-premise environment.

Dependencies:

  1. Docker v20.10
  2. Kubernetes v1.21

Contents

What is Fabric?

Fabric is a Python library used for automating routine shell commands over SSH, which we'll be using to automate the setup of a Kubernetes cluster.

Install:

$ pip install fabric==2.6.0

Verify the version:

$ fab --version

Fabric 2.6.0
Paramiko 2.8.0
Invoke 1.6.0

Test it out by adding the following code to a new file called fabfile.py:

from fabric import task


@task
def ping(ctx, output):
    """Sanity check"""
    print("pong!")
    print(f"hello {output}!")

Try it out:

$ fab ping --output="world"

pong!
hello world!

For more, review the official Fabric docs.

Droplets Setup

First, sign up for an account on DigitalOcean (if you don’t already have one), add a public SSH key to your account, and then generate an access token so you can access the DigitalOcean API.

Add the token to your environment:

$ export DIGITAL_OCEAN_ACCESS_TOKEN=<YOUR_DIGITAL_OCEAN_ACCESS_TOKEN>

Next, to interact with the API programmatically, install the python-digitalocean module:

$ pip install python-digitalocean==1.17.0

Now, let's create another task to spin up three droplets: One for the Kubernetes master and two for the workers. Update fabfile.py like so:

import os

from digitalocean import Droplet, Manager
from fabric import task

DIGITAL_OCEAN_ACCESS_TOKEN = os.getenv("DIGITAL_OCEAN_ACCESS_TOKEN")


# tasks


@task
def ping(ctx, output):
    """Sanity check"""
    print("pong!")
    print(f"hello {output}!")


@task
def create_droplets(ctx):
    """
    Create three new DigitalOcean droplets -
    node-1, node-2, node-3
    """
    manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
    keys = manager.get_all_sshkeys()
    for num in range(3):
        node = f"node-{num + 1}"
        droplet = Droplet(
            token=DIGITAL_OCEAN_ACCESS_TOKEN,
            name=node,
            region="nyc3",
            image="ubuntu-20-04-x64",
            size="s-2vcpu-4gb",
            tags=[node],
            ssh_keys=keys,
        )
        droplet.create()
        print(f"{node} has been created.")

Take note of the arguments passed to the Droplet class. Essentially, this will create three Ubuntu 20.04 droplets in the NYC3 region with 4 GB of memory each. It will also add all SSH keys to each droplet. You may want to update this to only include the SSH key that you created specifically for this project:

@task
def create_droplets(ctx):
    """
    Create three new DigitalOcean droplets -
    node-1, node-2, node-3
    """
    manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
    # Get ALL SSH keys
    all_keys = manager.get_all_sshkeys()
    keys = []
    for key in all_keys:
        if key.name == "<ADD_YOUR_KEY_NAME_HERE>":
            keys.append(key)
    for num in range(3):
        node = f"node-{num + 1}"
        droplet = Droplet(
            token=DIGITAL_OCEAN_ACCESS_TOKEN,
            name=node,
            region="nyc3",
            image="ubuntu-20-04-x64",
            size="s-2vcpu-4gb",
            tags=[node],
            ssh_keys=keys,
        )
        droplet.create()
        print(f"{node} has been created.")

Create the droplets:

$ fab create-droplets

node-1 has been created.
node-2 has been created.
node-3 has been created.

Moving along, let's add a task that checks the status of each droplet, to ensure that each are up and ready to go before we start installing Docker and Kubernetes:

@task
def wait_for_droplets(ctx):
    """Wait for each droplet to be ready and active"""
    for num in range(3):
        node = f"node-{num + 1}"
        while True:
            status = get_droplet_status(node)
            if status == "active":
                print(f"{node} is ready.")
                break
            else:
                print(f"{node} is not ready.")
                time.sleep(1)

Add the get_droplet_status helper function:

def get_droplet_status(node):
    """Given a droplet's tag name, return the status of the droplet"""
    manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
    droplet = manager.get_all_droplets(tag_name=node)
    return droplet[0].status

Don't forget the import:

import time

Before we test, add another task to destroy the droplets:

@task
def destroy_droplets(ctx):
    """Destroy the droplets - node-1, node-2, node-3"""
    manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
    for num in range(3):
        node = f"node-{num + 1}"
        droplets = manager.get_all_droplets(tag_name=node)
        for droplet in droplets:
            droplet.destroy()
        print(f"{node} has been destroyed.")

Destroy the three droplets we just created:

$ fab destroy-droplets

node-1 has been destroyed.
node-2 has been destroyed.
node-3 has been destroyed.

Then, bring up three new droplets and verify that they are good to go:

$ fab create-droplets

node-1 has been created.
node-2 has been created.
node-3 has been created.

$ fab wait-for-droplets

node-1 is not ready.
node-1 is not ready.
node-1 is not ready.
node-1 is not ready.
node-1 is not ready.
node-1 is not ready.
node-1 is ready.
node-2 is not ready.
node-2 is not ready.
node-2 is ready.
node-3 is ready.

Provision the Machines

The following tasks need to be run on each droplet...

Set Addresses

Start by adding a task to set the host addresses in the hosts environment variable:

@@task
def get_addresses(ctx, type):
    """Get IP address"""
    manager = Manager(token=DIGITAL_OCEAN_ACCESS_TOKEN)
    if type == "master":
        droplet = manager.get_all_droplets(tag_name="node-1")
        print(droplet[0].ip_address)
        hosts.append(droplet[0].ip_address)
    elif type == "workers":
        for num in range(2, 4):
            node = f"node-{num}"
            droplet = manager.get_all_droplets(tag_name=node)
            print(droplet[0].ip_address)
            hosts.append(droplet[0].ip_address)
    elif type == "all":
        for num in range(3):
            node = f"node-{num + 1}"
            droplet = manager.get_all_droplets(tag_name=node)
            print(droplet[0].ip_address)
            hosts.append(droplet[0].ip_address)
    else:
        print('The "type" should be either "master", "workers", or "all".')
    print(f"Host addresses - {hosts}")

Define the following variables at the top, just below DIGITAL_OCEAN_ACCESS_TOKEN = os.getenv('DIGITAL_OCEAN_ACCESS_TOKEN'):

user = "root"
hosts = []

Run:

$ fab get-addresses --type=all

165.227.96.238
134.122.8.106
134.122.8.204
Host addresses - ['165.227.96.238', '134.122.8.106', '134.122.8.204']

With that, we can start installing the Docker and Kubernetes dependencies.

Install Dependencies

Install Docker along with-

  1. kubeadm - bootstraps a Kubernetes cluster
  2. kubelet - configures containers to run on a host
  3. kubectl - command line tool used managing a cluster

Add a task to install Docker to the fabfile:

@task
def install_docker(ctx):
    """Install Docker"""
    print(f"Installing Docker on {ctx.host}")
    ctx.sudo("apt-get update && apt-get install -qy docker.io")
    ctx.run("docker --version")
    ctx.sudo("systemctl enable docker.service")

Let's disable the swap file:

@task
def disable_selinux_swap(ctx):
    """
    Disable SELinux so kubernetes can communicate with other hosts
    Disable Swap https://github.com/kubernetes/kubernetes/issues/53533
    """
    ctx.sudo('sed -i "/ swap / s/^/#/" /etc/fstab')
    ctx.sudo("swapoff -a")

Install Kubernetes:

@task
def install_kubernetes(ctx):
    """Install Kubernetes"""
    print(f"Installing Kubernetes on {ctx.host}")
    ctx.sudo("apt-get update && apt-get install -y apt-transport-https")
    ctx.sudo(
        "curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -"
    )
    ctx.sudo(
        'echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | \
          tee -a /etc/apt/sources.list.d/kubernetes.list && apt-get update'
    )
    ctx.sudo(
        "apt-get update && apt-get install -y kubelet=1.21.1-00 kubeadm=1.21.1-00 kubectl=1.21.1-00"
    )
    ctx.sudo("apt-mark hold kubelet kubeadm kubectl")

Instead of running each of these separately, create a main provision_machines task:

@task
def provision_machines(ctx):
    for conn in get_connections(hosts):
        install_docker(conn)
        disable_selinux_swap(conn)
        install_kubernetes(conn)

Add the get_connections helper function:

def get_connections(hosts):
    for host in hosts:
        yield Connection(
            f"{user}@{host}",
        )

Update the import:

from fabric import Connection, task

Run:

$ fab get-addresses --type=all provision-machines

This will take a few minutes to install the required packages.

Configure the Master Node

Init the Kubernetes cluster and deploy the flannel network:

@task
def configure_master(ctx):
    """
    Init Kubernetes
    Set up the Kubernetes Config
    Deploy flannel network to the cluster
    """
    ctx.sudo("kubeadm init")
    ctx.sudo("mkdir -p $HOME/.kube")
    ctx.sudo("cp -i /etc/kubernetes/admin.conf $HOME/.kube/config")
    ctx.sudo("chown $(id -u):$(id -g) $HOME/.kube/config")
    ctx.sudo(
        "kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml"
    )

Save the join token:

@task
def get_join_key(ctx):
    sudo_command_res = ctx.sudo("kubeadm token create --print-join-command")
    token = re.findall("^kubeadm.*$", str(sudo_command_res), re.MULTILINE)[0]

    with open("join.txt", "w") as f:
        with stdout_redirected(f):
            print(token)

Add the following imports:

import re
import sys
from contextlib import contextmanager

Create the stdout_redirected context manager:

@contextmanager
def stdout_redirected(new_stdout):
    save_stdout = sys.stdout
    sys.stdout = new_stdout
    try:
        yield None
    finally:
        sys.stdout = save_stdout

Again, add a parent task to run these:

@task
def create_cluster(ctx):
    for conn in get_connections(hosts):
        configure_master(conn)
        get_join_key(conn)

Run it:

$ fab get-addresses --type=master create-cluster

This will take a minute or two to run. Once done, the join token command should be outputted to the screen and saved to a join.txt file:

kubeadm join 165.227.96.238:6443 --token mvk32y.7z7i5x3viga4f4kn --discovery-token-ca-cert-hash sha256:f358dfc00ae7160fff3cb8fa3e3a3c8865f3c5b83c1f242fc9e51efe94108960

Configure the Worker Nodes

Using the saved join command from above, add a task to "join" the workers to the master:

@task
def configure_worker_node(ctx):
    """Join a worker to the cluster"""
    with open("join.txt") as f:
        join_command = f.readline()
        for conn in get_connections(hosts):
            conn.sudo(f"{join_command}")

Run this on the two worker nodes:

$ fab get-addresses --type=workers configure-worker-node

Sanity Check

Finally, to ensure the cluster is up and running, add a task to view the nodes:

@task
def get_nodes(ctx):
    for conn in get_connections(hosts):
        conn.sudo("kubectl get nodes")

Run:

$ fab get-addresses --type=master get-nodes

You should see something similar to:

NAME     STATUS   ROLES                  AGE    VERSION
node-1   Ready    control-plane,master   3m6s   v1.21.1
node-2   Ready    <none>                 84s    v1.21.1
node-3   Ready    <none>                 77s    v1.21.1

Remove the droplets once done:

$ fab destroy-droplets

node-1 has been destroyed.
node-2 has been destroyed.
node-3 has been destroyed.

Automation Script

One last thing: Add a create.sh script to automate this full process:

#!/bin/bash


echo "Creating droplets..."
fab create-droplets
fab wait-for-droplets
sleep 20

echo "Provision the droplets..."
fab get-addresses --type=all provision-machines


echo "Configure the master..."
fab get-addresses --type=master create-cluster


echo "Configure the workers..."
fab get-addresses --type=workers configure-worker-node
sleep 20

echo "Running a sanity check..."
fab get-addresses --type=master get-nodes

Try it out:

$ sh create.sh

That's it!

You can find the scripts in the kubernetes-fabric repo on GitHub.

Featured Course

Test-Driven Development with Django, Django REST Framework, and Docker

In this course, you'll learn how to set up a development environment with Docker in order to build and deploy a RESTful API powered by Python, Django, and Django REST Framework.

Buy Now $30 View Course
Featured Course

Test-Driven Development with Django, Django REST Framework, and Docker

In this course, you'll learn how to set up a development environment with Docker in order to build and deploy a RESTful API powered by Python, Django, and Django REST Framework.

Buy Now $30 View Course