Deply Helm and Tiller on PI Cluster

The main purpose of this exercise is to be able to use Helm on the Rapsberry PI Cluster.

Key Aspects

The goal is to setup helm and tiller on the Raspberry PI cluster
Having the golang, glide…and related libraries setup in a PI for compilation is kind of complicated. I started but encounter too many issues (even small), had to install too many compilation related packages on my PI system, hence decided to use an Ubuntu VM to compile and prepare the binaries for image for helm and tiller. This will be usefull to setup CI/CD pipeline with Travis-CI.

Build Tiller executable and Docker image

Cross Compiling from Ubuntu Machine

First check the go setup. Fetch the code

Let’s check the go environment. See Setup GOLANG environment

which go

/usr/bin/go

go version

go version go1.10.1 linux/amd64

export GOPATH=$HOME/src
mkdir -p src/k8s.io
cd src/k8s.io

git clone -b release-2.9 git@github.com:kubernetes/helm.git

cd $HOME/src/k8s.io/helm
make clean bootstrap build-cross dist APP=helm VERSION=2.9 TARGETS=linux/arm

...

go build -o bin/protoc-gen-go ./vendor/github.com/golang/protobuf/protoc-gen-go
CGO_ENABLED=0 gox -parallel=3 -output="_dist/{{.OS}}-{{.Arch}}/{{.Dir}}" -osarch='linux/arm'  -tags '' -ldflags '-w -s -X k8s.io/helm/pkg/version.Version=2.9 -X k8s.io/helm/pkg/version.BuildMetadata= -X k8s.io/helm/pkg/version.GitCommit=20adb27c7c5868466912eebdf6664e7390ebe710 -X k8s.io/helm/pkg/version.GitTreeState=clean -extldflags "-static"' k8s.io/helm/cmd/helm
Number of parallel builds: 3

-->       linux/arm: k8s.io/helm/cmd/helm
( \
        cd _dist && \
        find * -type d -exec cp ../LICENSE {} \; && \
        find * -type d -exec cp ../README.md {} \; && \
        find * -type d -exec tar -zcf helm-2.9-{}.tar.gz {} \; && \
        find * -type d -exec zip -r helm-2.9-{}.zip {} \; \
)
  adding: linux-arm/ (stored 0%)
  adding: linux-arm/LICENSE (deflated 65%)
  adding: linux-arm/README.md (deflated 59%)
  adding: linux-arm/helm (deflated 67%)

Let’s transfer the helm binarie to RPI

scp -r _dist/linux-arm/ rpiuser@192.168.1.95:/home/rpiuser/helm_binaries

Cross Compiling and Image cration from Travis-CI

Helm

Cross Compiling from Ubuntu Machine

Let’s assume we cloned the code for helm already.

make clean bootstrap build-cross dist APP=tiller VERSION=2.9 TARGETS=linux/arm

...
go build -o bin/protoc-gen-go ./vendor/github.com/golang/protobuf/protoc-gen-go
CGO_ENABLED=0 gox -parallel=3 -output="_dist/{{.OS}}-{{.Arch}}/{{.Dir}}" -osarch='linux/arm'  -tags '' -ldflags '-w -s -X k8s.io/helm/pkg/version.Version=2.9 -X k8s.io/helm/pkg/version.BuildMetadata= -X k8s.io/helm/pkg/version.GitCommit=20adb27c7c5868466912eebdf6664e7390ebe710 -X k8s.io/helm/pkg/version.GitTreeState=clean -extldflags "-static"' k8s.io/helm/cmd/tiller
Number of parallel builds: 3

-->       linux/arm: k8s.io/helm/cmd/tiller
( \
        cd _dist && \
        find * -type d -exec cp ../LICENSE {} \; && \
        find * -type d -exec cp ../README.md {} \; && \
        find * -type d -exec tar -zcf helm-2.9-{}.tar.gz {} \; && \
        find * -type d -exec zip -r helm-2.9-{}.zip {} \; \
)
  adding: linux-arm/ (stored 0%)
  adding: linux-arm/LICENSE (deflated 65%)
  adding: linux-arm/README.md (deflated 59%)
  adding: linux-arm/tiller (deflated 67%)

Build the docker image on the remote PI

Create a Dockerfile called rootfs/Dockerfile.arm32v7

FROM debian:jessie-slim

RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates \
    && rm -rf /var/lib/apt/lists/*

ENV HOME /tmp

COPY _dist/linux-arm /

EXPOSE 44134

CMD ["/tiller"]

export DHUBREPO=jbrette/tiller-arm32v7
export VERSION=2.9
docker build -t $DHUBREPO:$VERSION -f rootfs/Dockerfile.arm32v7 .

Sending build context to Docker daemon  284.1MB
Step 1/6 : FROM debian:jessie-slim
 ---> d273fca45b31
Step 2/6 : RUN apt-get update     && apt-get install -y --no-install-recommends ca-certificates     && rm -rf /var/lib/apt/lists/*
 ---> Running in e51ea5c31ad7
Get:1 http://security.debian.org jessie/updates InRelease [44.9 kB]
Ign http://deb.debian.org jessie InRelease
Get:2 http://deb.debian.org jessie-updates InRelease [145 kB]
Get:3 http://deb.debian.org jessie Release.gpg [2420 B]
Get:4 http://deb.debian.org jessie Release [148 kB]
Get:5 http://security.debian.org jessie/updates/main armel Packages [576 kB]
Get:6 http://deb.debian.org jessie-updates/main armel Packages [23.7 kB]
Get:7 http://deb.debian.org jessie/main armel Packages [8902 kB]
Fetched 9843 kB in 49s (199 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  libssl1.0.0 openssl
The following NEW packages will be installed:
  ca-certificates libssl1.0.0 openssl
0 upgraded, 3 newly installed, 0 to remove and 1 not upgraded.
Need to get 1692 kB of archives.
After this operation, 3770 kB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security/ jessie/updates/main ca-certificates all 20141019+deb8u4 [185 kB]
Get:2 http://deb.debian.org/debian/ jessie/main libssl1.0.0 armel 1.0.1t-1+deb8u8 [852 kB]
Get:3 http://deb.debian.org/debian/ jessie/main openssl armel 1.0.1t-1+deb8u8 [655 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 1692 kB in 7s (235 kB/s)
Selecting previously unselected package libssl1.0.0:armel.
(Reading database ... 7451 files and directories currently installed.)
Preparing to unpack .../libssl1.0.0_1.0.1t-1+deb8u8_armel.deb ...
Unpacking libssl1.0.0:armel (1.0.1t-1+deb8u8) ...
Selecting previously unselected package openssl.
Preparing to unpack .../openssl_1.0.1t-1+deb8u8_armel.deb ...
Unpacking openssl (1.0.1t-1+deb8u8) ...
Selecting previously unselected package ca-certificates.
Preparing to unpack .../ca-certificates_20141019+deb8u4_all.deb ...
Unpacking ca-certificates (20141019+deb8u4) ...
Setting up libssl1.0.0:armel (1.0.1t-1+deb8u8) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/arm-linux-gnueabi/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/arm-linux-gnueabi/perl5/5.20 /usr/share/perl5 /usr/lib/arm-linux-gnueabi/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Setting up openssl (1.0.1t-1+deb8u8) ...
Setting up ca-certificates (20141019+deb8u4) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/arm-linux-gnueabi/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/arm-linux-gnueabi/perl5/5.20 /usr/share/perl5 /usr/lib/arm-linux-gnueabi/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Updating certificates in /etc/ssl/certs... 152 added, 0 removed; done.
Processing triggers for libc-bin (2.19-18+deb8u10) ...
Processing triggers for ca-certificates (20141019+deb8u4) ...
Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.
Removing intermediate container e51ea5c31ad7
 ---> 1dccf46769e7
Step 3/6 : ENV HOME /tmp
 ---> Running in 45de8c047bd3
Removing intermediate container 45de8c047bd3
 ---> baa6d8b15164
Step 4/6 : COPY _dist/linux-arm /
 ---> 59a6cc8bfdbe
Step 5/6 : EXPOSE 44134
 ---> Running in d2c6d9b3a3bc
Removing intermediate container d2c6d9b3a3bc
 ---> 28a331217c52
Step 6/6 : CMD ["/tiller"]
 ---> Running in 9760ea4dc6c5
Removing intermediate container 9760ea4dc6c5
 ---> 9384b4f39ab3
Successfully built 9384b4f39ab3
Successfully tagged jbrette/tiller-arm32v7:2.9

Cross Compiling and Image cration from Travis-CI

Install Tiller and Helm on RPI

Deploy Tiller POD

Let’s install tiller first. Note that tiller Docker image also contains the helm binary

cd $HOME
cp -r proj/kubedge/kubedge_utils/kube-deployment/ .
cd kube-deployment/tiller/


kubectl create -f tiller-serviceaccount.yaml
kubectl create -f tiller.yaml

Check the deployment state

kubectl get all -n kube-system

NAME                                        READY     STATUS             RESTARTS   AGE
pod/tiller-deploy-b59fcc885-66l7s           1/1       Running            0          5m

NAME                                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/tiller-deploy          1         1         1            1           5m

NAME                                              DESIRED   CURRENT   READY     AGE
replicaset.apps/tiller-deploy-b59fcc885           1         1         1         5m

Check the logs

kubectl logs pod/tiller-deploy-b59fcc885-66l7s -n kube-system

[main] 2018/07/15 18:08:10 Starting Tiller 2.9 (tls=false)
[main] 2018/07/15 18:08:10 GRPC listening on :44134
[main] 2018/07/15 18:08:10 Probes listening on :44135
[main] 2018/07/15 18:08:10 Storage driver is ConfigMap
[main] 2018/07/15 18:08:10 Max history per release is 0

Init the Helm client side

locate the node running the tiller pod

ssh <nodewithtillerip>
docker cp <tillercontainerid>:/helm .
scp helm <masterpid>:/home/pirate/helm
rm helm

Back on the master PI

chmod 755 helm
mv helm /usr/bin/helm

Initialize client side of Helm

helm init --client-only

Creating /home/rpiuser/.helm
Creating /home/rpiuser/.helm/repository
Creating /home/rpiuser/.helm/repository/cache
Creating /home/rpiuser/.helm/repository/local
Creating /home/rpiuser/.helm/plugins
Creating /home/rpiuser/.helm/starters
Creating /home/rpiuser/.helm/cache/archive
Creating /home/rpiuser/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /home/rpiuser/.helm.
Not installing Tiller due to 'client-only' flag having been set
Happy Helming!

Use helm on RPI

Let’s access the simple helm repo designed for

helm repo add kubedge1 'https://raw.githubusercontent.com/kubedge/helmrepos/arm32v7/kubedge1/'
helm repo update

helm repo list

NAME            URL
stable          https://kubernetes-charts.storage.googleapis.com
local           http://127.0.0.1:8879/charts
kubedge1        https://raw.githubusercontent.com/kubedge/helmrepos/arm32v7/kubedge1/

helm search kubedge1

NAME                            CHART VERSION   APP VERSION     DESCRIPTION
kubedge1/kubeplay-arm32v7       0.1.0           0.1.0           A Helm chart for Kubernetes

Install simple hello world helm chart from kubeplay

helm install --name demo kubedge1/kubeplay-arm32v7

kubectl get all

Let’s open a browser towards the URL on http://192.168.2.1:kubeplay_NortPort:

Conclusion

Need to cleanup the helm chart and docker image to ensure consistency of the internal
Need to improve the NodePort/ClusterIP/LoadBalancer handling.

Reference Links

Usage of remote API