Creating a Raspberry Pi Kubernetes cluster is straight forward, however it becomes more complicated when you want to mix and match kubernetes nodes of different CPU architectures.
I recently deployed a new Kubernetes cluster in my home lab which initially consisted of two Ubuntu nodes running kubeadm and are VMs on VMware ESXi. Last night I tried to add a Raspberry Pi and ran into a couple of issues which I resolved and describe further down to save you some time.
kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-raspberry-01 Ready 4h v1.11.2 192.168.1.216 Raspbian GNU/Linux 9 (stretch) 4.14.62-v7+ docker://18.6.1 k8s-ubuntu-01 Ready master 5d v1.11.2 192.168.1.127 Ubuntu 18.04.1 LTS 4.15.0-33-generic docker://18.6.1 k8s-ubuntu-02 Ready 5d v1.11.2 192.168.1.156 Ubuntu 18.04.1 LTS 4.15.0-33-generic docker://18.6.1
My Kubernetes Setup:
Kubernetes: v1.11.2
Docker: ce-18.6.1
CNI: Flannel v0.10.0
CPU Architectures: AMD64 & ARM
OS: Ubuntu 18.04.1 LTS & Raspbian June 2018
NAS: NFS via FreeNAS
Note: If you’re looking for good instructions on setting up kubeadm on a Raspberry Pi, please take a look at Setup a Kubernetes 1.9.0 Raspberry Pi cluster on Raspbian using Kubeadm
Issue #1 – Kube-Proxy
If you start out with running Ubuntu on your master node, kube proxy will only be available for amd64 CPU architecture. In order to run kube proxy on a Raspberry Pi, you’ll need to deploy a separate daemonset for the ARM architecture. I created a yaml for this purpose and published it on my Github profile. Download kube-proxy-arm.yaml and deploy it with:
kubectl create -f kube-proxy-arm.yaml
If you were successful, you should be able to see kube-proxy-arm running as a container.
kubectl get pods -n kube-system | grep proxy kube-proxy-2kmp8 1/1 Running 0 5d kube-proxy-arm-9qt76 1/1 Running 2 4h kube-proxy-rt9hf 1/1 Running 0 5d
Issue #2 – CNI Flannel
Flannel v0.10.0 provides two
kube-flannel.yaml files and both are referenced by the official Kubernetes documentation. When I followed the instructions, I didn’t read all the instructions and deployed the official flannel v0.10.0 yaml. However this version only support the amd64 CPU architecture. If you want to mix CPU architectures across your kubernetes clsuter, you want to deploy the following kube-flannel.yaml as it support ARM, ARM64, AMD64, PPC64LE and S390X.
Once you fix the two issues above, you should have all nodes operational and ready to run some containers. Be sure to deploy images which are ARM compatible as not all images are compatible.
kubectl get ds -n kube-system NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE kube-flannel-ds-amd64 2 2 2 2 2 beta.kubernetes.io/arch=amd64 18h kube-flannel-ds-arm 1 1 1 1 1 beta.kubernetes.io/arch=arm 18h kube-flannel-ds-arm64 0 0 0 0 0 beta.kubernetes.io/arch=arm64 18h kube-flannel-ds-ppc64le 0 0 0 0 0 beta.kubernetes.io/arch=ppc64le 18h kube-flannel-ds-s390x 0 0 0 0 0 beta.kubernetes.io/arch=s390x 18h kube-proxy 2 2 2 2 2 beta.kubernetes.io/arch=amd64 5d kube-proxy-arm 1 1 1 1 1 beta.kubernetes.io/arch=arm 5h
I’m thinking about doing something similar. Have you managed to create a deployment where the pods can run on any of the nodes?
I’m seeing a problem in my mind where you’ll need a different image for each architecture so deployments will have to be targeted to nodes with labels or something. We would need something that picks an image depending on where it’s running.
I haven’t been able to do this yet. I wonder if someone already has done it. Currently I run ARM and non-ARM nodes and manage them via tags to deploy either an ARM or a non-ARM image.
Actually I ran into multi-arch image feature by creating “meta-images” kind of. Where you have a manifest that actually points to other images depending on architecture.
https://container-solutions.com/multi-arch-docker-images/
I believe this would solve that problem. The manifest command is experimental in docker now.
You can run `docker manifest inspect coredns/coredns:1.2.6` or a more readable output from `docker run mplatform/mquery coredns/coredns:1.2.6` (coredns as an example, a bunch of images already offer this, I believe all official images).
That’s a great find Arnar!