Building a distributed Kubernetes platform from scratch

Author: Ananda Kammampati

 

While I was building VMware Photon-1.2 platform on my Intel NUC, I ran into some challenges running the Worker node VMs successfully (hence the reason, I couldn’t continue on Part-2). While trying to debug the problem, soon I realize that I need to understand the Kubernetes platform a lot better.

So in this article, I am venturing into build a scalable distributed Kubernetes platform for scratch. My approach is always the same - to understand what it takes to build any platform in a distributed manner so that, we can have a better understanding on all the moving parts and their interactions among them.

The scope of this article is to help you build a fully distributed kubernetes platform all by yourself, to have it at your disposal on your own machine so that you can start playing with it and thereby, learning the platform better. This article doesn’t get into the basics of kubernetes platform per se. There are many articles out there online that do a fantastic job, which I am pointing them to, under References.

For now, I am focusing only on building the platform from scratch (you can stop right here if this is not of your interest :-)) Once I am done with that part, I am pointing to some good links/resources who have done a very good job in teaching how to run workloads with their YAML files and in the process, explaining all the features the platform as a whole, provides.

kubernetes-s

My physical setup is follows:

  • I use Intel NUC running ESXi6 Hypervisor. This is where all the virtual machines that make up the Kubernetes platform run
  • I physically connect my Intel NUC with my Macbook-Pro laptop with a crossover cable back-to-back
  • I define a dedicated network between the two.
  • I then built the entire Kubernetes platform on my NUC, there by accessing it from my Laptop
k8s-host
k8s-vms

Once the Kubernetes platform is completely built, you will have multiple networks, each for a different purpose:

  1. 10.1.1.0/24 network – This is the network defined between my Macbook pro and my Intel NUC
  2. 172.16.10.0/24 network – This is the private network on which all the VMs that make up the kubernetes platform reside
  3. 10.20.0.0/16 network – This is the network created by flanneld which is an overlay network on top of (2)
  4. 10.20.xx.yy/24 network – This is the network that gets created among each docker running on each NODE so that the network address space is spread across multiple NODES. The most important thing to understand is that on each NODE where the docker runs, it gets a subnet assigned for itself from flanneld, which then uses that subnet range to assign the IP addresses for the Containers on its node (This will be clear in Section 4)
  5. 10.240.0.0/24 network – This is the Cluster lever network whose scope resides within the Cluster.

 

To have a distributed Kubernetes platform, I built a dozen virtual machines in total. Apart from EDGE VM, rest of them are running CentOS7:

1 x EDGE VM running EFW distribution, configured as network gateway

1 x DNS VM – CentOS7 VM running FreeIPA for DNS service

1 x JUMPBOX VM – CentOS7 VM where you log in that has visibility to all the VMs

3 x ETCD VMs – Three CentOS7 VMs that run ETCD service, which is a distributed Key / Value store. This is where all the information about the Kubernetes platform is stored and shared among all the components

1 x MASTER VM – CentOS7 VM where kube-apiserver, kube-controller-manager, kube-scheduler, flannel (overlay networking) and docker will be running. It is worth noting that it is possible to architect the platform without running docker on the Master node. In a production system, it is recommended to have a secondary Master as well, and having a Load Balancer in the frontend. I will skip the secondary Master and Load Balancer for now and will try visiting at a later time

5 x NODE VMs – Five CentOS7 VMs where kubelet, kube-proxy, flanneld, docker will be installed. There are the Nodes where Kubernetes PODs will be running. Each POD holds one or more docker containers that run the workloads

On my Intel NUC running ESXi6 hypervisor, I created 2 standard switches – vSwitch0 (WAN) and vSwitch1 (LAN). Except for the EDGE VM which acts as a Gateway that saddles between vSwitch0 and vSwitch1, rest of the VMs are connected to vSwitch1, there by residing on a separate network (172.16.10.0/24)

Configuration of virtual machines are as follows:

 

k8s-configtable

I have broken down the steps in building the distributed Kubernetes platform into multiple sections:

  1. Configuration of ETCD VMs
  2. Configuration of JUMPBOX VM (specific to my setup)
  3. Configuration of the MASTER VM
  4. Configuration of NODES VMs
  5. Installation of Kubernetes Dashboard
  6. Running a simple NGINX server as a POD
  7. The real learning of Kubernetes platform starts now

 

Section 1. Configuration of ETCD VMs

 

I have seen articles that explain how to build ETCD distributed store as part of the Master VM. But here I am building the ETD as a separate cluster by itself, made up of 3 VMs (you can go up to 5 VMs). This way, the key /value store will be kept truly independent as how it should be in a production grade setup.

I started with the first VM of the ETCD cluster namely ETCD-01 VM. Except for its IP address, the steps are exactly the same for the other two ETCD VMs as well (ETCD-02 and ETCD-03).

To begin with, disable SELINUX by editing the file /etc/selinux/config followed by stopping/disabling firewall service, installing the packages open-vm-tools, net-tools and finally updating the entire system with yum update command.

ETCD-01# vi /etc/selinux/config

disable SELINUX

ETCD-01# setenforce 0

ETCD-01# systemctl stop firewalld.service

ETCD-01# systemctl disable firewalld.service

ETCD-01# yum -y install open-vm-tools net-tools

ETCD-01# yum -y update ; sync ; reboot

Create a user and group account named etcd under which the etcd service will be running

ETCD-01# mkdir /var/lib/etcd /etc/etcd

ETCD-01# groupadd -r etcd

ETCD-01# useradd –r -g etcd -d /var/lib/etcd -s /bin/nologin -c “etcd user” etcd

ETCD-01# chown -R etcd:etcd /var/lib/etcd

Create a service file for etcd so that it can be controlled by systemctl

ETCD-01# vi /usr/lib/systemd/system/etcd.service

-----

[Unit]

Description=etcd service

After=network.agent

[Service]

Type=notify

WorkingDirectory=/var/lib/etcd

EnvironmentFile=/etc/etcd/etcd.conf

User=etcd

ExecStart=/usr/bin/etcd

LimitNOFILE=65536

[Install]

WantedBy=multi.user.target

-----

Download and install ETCD software from github repository provided by CoreOS, by following the commands shown below:

 

ETCD-01# ETCD_VERSION=`curl -s -L https://github.com/coreos/etcd/releases/latest | grep linux-amd64\.tar\.gz | grep href | cut -f 6 -d '/' | sort -u`

ETCD-01# echo $ETCD_VERSION

ETCD-01# ETCD_DIR=/opt/etcd-$ETCD_VERSION

ETCD-01# echo $ETCD_DIR

ETCD-01# mkdir $ETCD_DIR

ETCD-01# curl -L https://github.com/coreos/etcd/releases/download/$ETCD_VERSION/etcd-$ETCD_VERSION-linux-amd64.tar.gz | tar xz --strip-components=1 -C $ETCD_DIR

ETCD-01# ln -sf $ETCD_DIR/etcd /usr/bin/etcd && ln -sf $ETCD_DIR/etcdctl /usr/bin/etcdctl

ETCD-01# etcdctl –version

etcdctl version: 3.2.1

API version: 2

k8s-etcd-files

Edit the file /etc/etcd/etcd.conf and define the required environment variables according to its IP address. Leave the default port numbers. The IP address of the VM ETCD-01 is 172.16.10.21

Also note that the IP addresses of VMs namely ETCD-02 (172.16.10.22) and ETCD-03 (172.16.10.23) are defined for the environment variable ETCD_INITIAL_CLUSTER

 

ETCD-01# vi /etc/etcd/etcd.conf

-----

# [member]

ETCD_NAME=etcd-01

ETCD_DATA_DIR="/var/lib/etcd/etcd-01.etcd"

ETCD_LISTEN_PEER_URLS="http://172.16.10.21:2380”

ETCD_LISTEN_CLIENT_URLS="http://172.16.10.21:2379”

ETCD_ADVERTISE_CLIENT_URLS="http://172.16.10.21:2379”

# [cluster]

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.10.21:2380”

ETCD_INITIAL_CLUSTER="default="etcd-01=http://172.16.10.21:2380,etcd-02=http://172.16.10.22:2380,etcd-03=http://172.16.10.23:2380”

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01" 

-----

Now that the configuration is done on ETCD-01 VM, enable and start the ETCD distributed key/value store service with the following commands:

ETCD-01# systemctl enable etcd.service

ETCD-01# systemctl start etcd.service

ETCD-01# systemctl status -l etcd.service

Finally, repeat the above commands in the other two VMs as well that are meant for ETCD cluster namely ETCD-02 (172.16.10.22) and ETCD-03 (172.16.10.23) with their appropriate IP addresses in their /etc/etcd/etcd.conf file

 

Section 2: Configuration of JUMPBOX VM

 

This section is specific to the way I design this lab. Since JUMPBOX VM is my primary VM where I will be working for the most part, I have created an user account named student and I am setting that user account’s environment in a way, I can manage both ETCD distributed store and Kubernetes platform remotely, from this JUMPBOX VM. Following are the list of things I am doing on JUMPBOX VM:

  • As root, copy the command etcdctl from any of the ETCD VMs into /usr/local/bin directory
  • As root, download kubectl command from kubernetes repository and place it in /usr/local/bin directory
  • Make changes to the file .bashrc in home directory (of user account – student)
  • Make a specific entry in the ETCD distributed store which is needed later for flanneld in creating the overlay network

 

Do the following as root:

JUMPBOX# cd /usr/local/bin

JUMPBOX# scp root@172.16.10.21:/opt/etcd-v3.2.1/etcdctl .

JUMPBOX# chmod +x ./etcdctl

JUMPBOX# curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl

JUMPBOX# chmod +x ./kubectl

Do the following as user - student on JUMPBOX. It is this account I use when I log into JUMPBOX (than using root account)

JUMPBOX$ hostname

jumpbox.kuber.local

JUMPBOX$ whoami

student

JUMPBOX$ pwd

/home/student

JUMPBOX$ vi .bashrc

-----

export ETCD_CTLAPI=3

export ENDPOINTS="http://172.16.10.21:2379,http://172.16.10.22:2379,http://172.16.10.23:2379"

alias etcdctl="etcdctl --endpoints=$ENDPOINTS "

alias kubectl=”kubectl --server=172.16.10.1:8080”

-----

Note that I am making a reference to the MASTER-VM (172.16.10.1) though I haven’t configured it yet. I will be doing that at a later stage. For now continuing as the user student

JUMPBOX$ . .bashrc

JUMPBOX$ etcdctl –version

etcdctl version: 3.2.1

API version: 2

Checking the health of the ETCD distributed store

JUMPBOX$ etcdctl cluster-health

 

 

k8s-cluster-health

The above output confirm that the ETCD is configured correctly and is up and running.

Next, the command below makes an entry in the ETCD distributed store. The IP address, subnet mask, port number all makes sense at a later stage when configuring flanneld

JUMPBOX$ etcdctl mk /overlay/network/config "{\"Network\": \"10.20.0.0/16\", \"SubnetLen\": 24, \"Backend\": { \"Type\": \"udp\", \"Port\": 7890 } }"

Verify that the entry was done right by retrieving it:

JUMPBOX$ etcdctl get overlay/network/config

 

k8s-get-overlay

Verify the same with curl command as well:

JUMPBOX$ curl -L http://172.16.10.21:2379/v2/keys/overlay/network/config

 

k8s-curl

Section 3: Configuration of MASTER-01 VM

The following components needs to be installed and configured on MASTER-01 VM:

  • kube-apiserver
  • kube-controller-manager
  • kube-scheduler
  • flanneld
  • docker

I have recreated the diagram showing the components running in the MASTER-01 VM, and their interactions with other components with other systems (Author: Edward Viaene)

 

k8s-master-01

To begin with, define the required repository first MASTER-01# vi /etc/yum.repos.d/virt7-docker-common-release.repo

-----

[virt7-docker-common-release]

name=virt7-docker-common-release

baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/

gpgcheck=0

-----

Installing Kubernetes and flannel packages will install docker and their dependencies as well.

MASTER-01# yum -y install --enablerepo=virt7-docker-common-release kubernetes flannel

Modify the file /etc/kubernetes/config to define the environment variables regarding ETCD Cluster and MASTER-VM

MASTER-01# vi /etc/kubernetes/config

-----

KUBE_ETCD_SERVERS="--etcd-servers=http://172.16.10.21:2379,http://172.16.10.22:2379,http://172.16.10.23:2379"

KUBE_LOGTOSTDERR="--logtostderr=true"

KUBE_LOG_LEVEL="--v=0"

KUBE_ALLOW_PRIV="--allow-privileged=false"

KUBE_MASTER="--master=http://172.16.10.1:8080"

-----

Down the following shell script from the net. Make some modifications and then run it, which will generate the required certificates and place them in the required directory – /srv/kubernetes

Copy the contents of file to /root in MASTER-01 VM.

https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/generate-cert/make-ca-cert.sh

EDIT line# 30 of make-ca-cert.sh

Change the value of the variable cert_group=${CERT_GROUP:-kube-cert}

Remove the “-cert” at the end.

MASTER-01# vi /root/make-ca-cert.sh

-----

cert_group=${CERT_GROUP:-kube}

-----

Now run that shell script

MASTER-01# bash /root/make-ca-cert.sh

The directory /srv/kubernetes will have the generated certificates

MASTER-01#ls /srv/kubernetes

 

k8s-certs

Define kube-apiserver requirements by modifying the file /etc/kubernetes/apiserver

MASTER-01# vi /etc/kubernetes/apiserver

-----

KUBE_API_ADDRESS="--address=172.16.10.1"

KUBE_API_PORT="--port=8080"

KUBELET_PORT="--kubelet-port=10250"

KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"

KUBE_API_ARGS” --client-ca-file=/srv/kubernetes/ca.crt --tls-cert-file=/srv/kubernetes/server.cert --tls-private-key-file=/srv/kubernetes/server.key"

-----

Define kube-controller-manager requirements by modifying the file /etc/kubernetes/controller-manager

MASTER-01# vi /etc/kubernetes/controller-manager

-----

# Add your own!

KUBE_CONTROLLER_MANAGER_ARGS="--root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key"

-----

Define flannel requirements by modifying the file /etc/sysconfig/flannel

MASTER-01# vi /etc/sysconfig/flannel

-----

FLANNEL_ETCD="http://172.16.10.21:2379,http://172.16.10.22:2379,http://172.16.10.23:2379"

FLANNEL_ETCD_PREFIX="/overlay/network/config"

-----

If you recall, /overlay/network/config is exactly what we added into the ETCD distributed store in Section 2. And it is that network address space (10.20.0.0/16) that will be used by flannel, which then creates subnets and assigns to each docker running on different NODE VM. Again, I would strongly recommend Peng Xiao’s website for a thorough explanation and better understanding.

Now that all the requirements are met, enable and start each server in the following order:

 

MASTER-01# systemctl enable kube-apiserver

MASTER-01# systemctl start kube-apiserver

MASTER-01# systemctl status -l kube-apiserver

 

MASTER-01# systemctl enable kube-controller-manager

MASTER-01# systemctl start kube-controller-manager

MASTER-01# systemctl status -l kube-controller-manager

 

MASTER-01# systemctl start kube-scheduler

MASTER-01# systemctl start kube-scheduler

MASTER-01# systemctl status -l kube-scheduler

 

MASTER-01# systemctl enable flanneld

MASTER-01# systemctl start flanneld

MASTER-01# systemctl status -l flanneld

 

MASTER-01: systemctl restart docker

MASTER-01# systemctl status -l docker

 

Section 4: Configuration of NODE-01 VM

 

There are total of 5 NODE VMs in my setup. These are the VMs that will eventually run the PODS/docker containers/application workloads.  

The following components will be installed and configured on all five NODE VMs:

  • kubelet
  • kube-proxy
  • flanneld
  • docker

Again, I have recreated the diagram that shows the components running in NODE VMs and their interactions with other components

k8s-node-01

The setup shown below is to the context of NODE-01 VM. Except for the IP address, everything else is pretty much the same that needs to be repeated on the other 4 NODE VMs. Define the repository first 

NODE-01# vi /etc/yum.repos.d/virt7-docker-common-release.repo

-----

[virt7-docker-common-release]

name=virt7-docker-common-release

baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/

gpgcheck=0

-----

Install the required packages and their dependencies

NODE-01# yum -y install --enablerepo=virt7-docker-common-release kubernetes flannel

[Note: Docker will also be installed as part of the dependency and it will also be started by default]

Start with the configuration about the MASTER-01 VM by editing the file /etc/kubernetes/config

NODE-01# vi /etc/kubernetes/config

-----

KUBE_LOGTOSTDERR="--logtostderr=true"

KUBE_LOG_LEVEL="--v=0"

KUBE_ALLOW_PRIV="--allow-privileged=false"

KUBE_MASTER="--master=http://172.16.10.1:8080"

-----

Define kubelet requirements in the file /etc/kubernetes/kubelet

NODE-01# vi /etc/kubernetes/kubelet

 -----

KUBELET_ADDRESS="--address=172.16.10.51"

KUBELET_PORT="--port=10250"

KUBELET_HOSTNAME="--hostname-override=node-01.kuber.local"

KUBELET_API_SERVER="--api-servers=http://172.16.10.1:8080"

KUBELET_ARGS=”--cluster-dns=172.16.10.250 --cluster-domain=kuber.local”

-----

Define flannel requirements in the file /etc/sysconfig/flanneld

NODE-01# vi /etc/sysconfig/flanneld

----- 

FLANNEL_ETCD="http://172.16.10.21:2379,http://172.16.10.22:2379,http://172.16.10.23:2379"

FLANNEL_ETCD_PREFIX="/overlay/network"

-----

Now that all the configurations are done, enable and start the services

 

NODE-01# systemctl enable kube-proxy

NODE-01# systemctl start kube-proxy

NODE-01# systemctl status –l kube-proxy

 

NODE-01# systemctl enable kubelet

NODE-01# systemctl start kubelet

NODE-01# systemctl status –l kubelet

 

NODE-01# systemctl enable flannel

NODE-01# systemctl start flannel

NODE-01# systemctl status –l flannel

 

The following files will created once flanneld gets started

/run/flannel/subnet.env

/run/flannel/docker

NODE-01# cat /run/flannel/subnet.env

-----

FLANNEL_NETWORK=10.20.0.0/16

FLANNEL_SUBNET=10.20.18.1/24

FLANNEL_MTU=1472

FLANNEL_IPMASQ=false

-----

NODE-01# cat /run/flannel/docker

-----

DOCKER_OPT_BIP=”--bip=10.20.18.1/24”

DOCKER_OPT_IPMASQ=”--ip-masq=true”

DOCKER_OPT_MTU=”--mtu=1472”

DOCKER_NETWORK_OPTIONS=”--bip=10.20.18.1/24 --ip-masq=true --mtu=1472”

-----

Note that the above IP address for docker (10.20.18.1/24) is obtained from the subnet range provided by flanneld service, which in turn gets it from the ETCD service where 10.20.0.0/16 network address space was already defined by us before (in Section 2). Flanneld refers ETCD service in taking care of assigning different subnet ranges to different docker(s) running on different NODES.

Initially when you run ‘ifconfig’ command, you will notice that docker0 (linux bridge) will be assigned a different IP address, when the docker package was installed and docker daemon was started the first time.

NODE-01# ifconfig

 

k8s-ifconfig-1

But once you restart the docker service after starting flanneld, you will now notice that it will consume the very first IP address from the flanneld assigned subnet address range (10.20.X), and no longer uses 172.17.X network.

NODE-01# systemctl restart docker

NODE-01# systemctl status –l docker

NODE-01# ifconfig

 

k8s-ifconfig-2

Docker will use the very first IP address from the flanneld’s subnet (docker0 = 10.20.18.1/24). From here onwards, all the Containers will be auto assigned from this network address space where Docker (10.20.18.1 in my case) will be their Gateway.

This is probably a good place make a pause, take a step back, try to assimilate all the details in understanding the mechanics of how overlay networking is being done by flanneld.

I would strongly recommend Peng Xiao’s website (References section) who has done an excellent job in explaining the networking fundamentals needed for Kubernetes platform.

To verify NODE-01 VM is talking to MASTER-01 and all the communications are happening as expected, go to JUMPBOX and issue the following command:

JUMPBOX$ kubectl get nodes –o wide

 

k8s-get-nodes-all

Section 5: Installation of Kubernetes dashboard

 

I have to admit that by the time I reached this far, I desperately wanted to see how Kubernetes dashboard UI is going to show up. So without debugging the errors messages that I faced, I just passed the parameter “--validate=false” to suppress them so that I can get on with the dashboard UI (I know I am cheating here)

JUMPBOX$ kubectl create -f https://git.io/kube-dashboard --validate=false

But I believe the right way to do is:

JUMPBOX$ kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml

If a password is asked, I believe you can retrieve the password with:

JUMPBOX$ kubectl config view

Verify the deployment of the dashboard POD as well

JUMPBOX$ kubectl create get pods –all-namespaces

 

k8s-dash-pod

And finally, from JUMPBOX VM you should be able to see Kubernetes dashboard UI at :

http://master-01.kuber.local:8080/ui

 

k8s-dashboard

Section 6: Installing NGINX POD

 

Follow the link below to run your very first pod. From now on, you can run all the commands from JUMPBOX VM.

https://github.com/kubernetes/kubernetes/blob/master/examples/simple-nginx.md

 

Section 7: The learning curve start here

 

If you have come this far, you’ll realize that what we have done so far is just the beginning J Next, we need to learn how to use, consume and administer the platform, for which I have a couple of suggestions:

Download the lab exercises provided by Edward Viaene and Peng Xiao and follow their documentation:

https://github.com/wardviaene/kubernetes-course

https://github.com/xiaopeng163/docker-k8s-lab

And if you really want to accelerate the learning process, I would recommend you to take these two courses. They go hand-on-hand with the above links.

https://www.udemy.com/learn-devops-the-complete-kubernetes-course/

https://www.udemy.com/learning-docker-and-k8s-by-practice/

Now that you have your own Kubernetes platform at you disposal running on your own server, having the lab exercises and video lessons will certainly save you a lot of time and keep you stay focused with you learning.

Needless you say, wishing you the very best!

 

Section 8: References

 

http://docker-k8s-lab.readthedocs.io/en/latest/

http://www.tothenew.com/blog/how-to-install-kubernetes-on-centos/

https://severalnines.com/blog/installing-kubernetes-cluster-minions-centos7-manage-pods-services

https://www.vultr.com/docs/getting-started-with-kubernetes-on-centos-7

https://github.com/kubernetes/dashboard