Kubedge > DevOps > Kubedge SDK > Recompile Kubernetes components for Raspberry PI

Recompile Kubernetes components for Raspberry PI

During the installation of official Kubernetes 1.11.0 on RPI Cluster 1, encountered a bug on the controller manager preventing the controller-manager from starting. The problem here was to be able to cross compiled the latest version of Kubernetes 1.11.1 before the code was officially released and of course rebuild the images.

Key Aspects

The bug had been fixed by the Kubernetes kube-controller-manager - panic: runtime error: index out of range has been fixed and will be built as part of 1.11.1
The goal is to learn how to recompile Kubernetes from the source to be able to contribute when possible and address problems as soon as possible

Build Kubernetes executables for AMD64 and ARM

Cross Compiling from Ubuntu Machine

First check the go setup. Fetch the code

Let’s check the go environment. See Setup GOLANG environment

$ which go

/usr/bin/go

$ go version

go version go1.10.1 linux/amd64

Clone the code

$ export GOPATH=$HOME/src
$ mkdir -p src/k8s.io
$ cd src/k8s.io
$ git clone -b release-1.11 git@github.com:kubernetes/kubernetes.git
$ cd kubernertes

To save time, I removed platforms I had not use of and only kept amd64 and arm on linux

$ cat hack/lib/golang.sh

  # The server platform we are building on.
  readonly KUBE_SERVER_PLATFORMS=(
    linux/amd64
    linux/arm
  )

  # The node platforms we build for
  readonly KUBE_NODE_PLATFORMS=(
    linux/amd64
    linux/arm
  )

  # If we update this we should also update the set of platforms whose standard library is precompiled for in build/build-image/cross/Dockerfile
  readonly KUBE_CLIENT_PLATFORMS=(
    linux/amd64
    linux/arm
  )

  # Which platforms we should compile test targets for. Not all client platforms need these tests
  readonly KUBE_TEST_PLATFORMS=(
    linux/amd64
    linux/arm
  )

Double check your Docker setup is correct

$ docker run hello-world

Clean the directory

$ ./build/make-clean.sh

Rebuild the amd64 and arm executable

$ ./build/run.sh make cross

Check the executables transfered from the docker build container to your vm by rsync

$ cd _output/dockerized/bin/linux

Check the amd64 executables

ls -lt amd64

total 2322696
-rwxr-xr-x 1 xxxxxx yyyyyy 209961168 Jul 16 01:57 e2e_node.test
-rwxr-xr-x 1 xxxxxx yyyyyy  10645252 Jul 16 01:57 ginkgo
-rwxr-xr-x 1 xxxxxx yyyyyy 160051600 Jul 16 01:57 kubemark
-rwxr-xr-x 1 xxxxxx yyyyyy 231997872 Jul 16 01:55 genman
-rwxr-xr-x 1 xxxxxx yyyyyy  54099081 Jul 16 01:55 gendocs
-rwxr-xr-x 1 xxxxxx yyyyyy  54039865 Jul 16 01:55 genyaml
-rwxr-xr-x 1 xxxxxx yyyyyy   6694536 Jul 16 01:55 linkcheck
-rwxr-xr-x 1 xxxxxx yyyyyy   5481582 Jul 16 01:55 genswaggertypedocs
-rwxr-xr-x 1 xxxxxx yyyyyy 226061456 Jul 16 01:55 genkubedocs
-rwxr-xr-x 1 xxxxxx yyyyyy 173411368 Jul 16 01:55 e2e.test
-rwxr-xr-x 1 xxxxxx yyyyyy  55241186 Jul 16 01:52 kubectl
-rwxr-xr-x 1 xxxxxx yyyyyy  51912114 Jul 16 01:52 kube-proxy
-rwxr-xr-x 1 xxxxxx yyyyyy  57246829 Jul 16 01:51 kubeadm
-rwxr-xr-x 1 xxxxxx yyyyyy 162716256 Jul 16 01:51 kubelet
-rwxr-xr-x 1 xxxxxx yyyyyy  57908740 Jul 16 01:50 kube-aggregator
-rwxr-xr-x 1 xxxxxx yyyyyy 185152632 Jul 16 01:50 kube-apiserver
-rwxr-xr-x 1 xxxxxx yyyyyy   2330265 Jul 16 01:50 mounter
-rwxr-xr-x 1 xxxxxx yyyyyy 138048783 Jul 16 01:50 cloud-controller-manager
-rwxr-xr-x 1 xxxxxx yyyyyy 153798994 Jul 16 01:50 kube-controller-manager
-rwxr-xr-x 1 xxxxxx yyyyyy 227283184 Jul 16 01:50 hyperkube
-rwxr-xr-x 1 xxxxxx yyyyyy  59296776 Jul 16 01:50 apiextensions-apiserver
-rwxr-xr-x 1 xxxxxx yyyyyy  55471235 Jul 16 01:50 kube-scheduler
-rwxr-xr-x 1 xxxxxx yyyyyy   2835466 Jul 16 01:40 go-bindata
-rwxr-xr-x 1 xxxxxx yyyyyy  13630237 Jul 16 01:40 openapi-gen
-rwxr-xr-x 1 xxxxxx yyyyyy   7699847 Jul 16 01:40 conversion-gen
-rwxr-xr-x 1 xxxxxx yyyyyy   7669238 Jul 16 01:39 defaulter-gen
-rwxr-xr-x 1 xxxxxx yyyyyy   7695690 Jul 16 01:39 deepcopy-gen

Check the RPI executables

$ ls -lt arm

total 2089216
-rwxr-xr-x 1 xxxxxx yyyyyy 191189688 Jul 16 01:58 e2e_node.test
-rwxr-xr-x 1 xxxxxx yyyyyy   9286102 Jul 16 01:58 ginkgo
-rwxr-xr-x 1 xxxxxx yyyyyy 142111524 Jul 16 01:58 kubemark
-rwxr-xr-x 1 xxxxxx yyyyyy 210989712 Jul 16 01:55 genman
-rwxr-xr-x 1 xxxxxx yyyyyy  48169969 Jul 16 01:55 gendocs
-rwxr-xr-x 1 xxxxxx yyyyyy  48102745 Jul 16 01:55 genyaml
-rwxr-xr-x 1 xxxxxx yyyyyy   5847821 Jul 16 01:55 linkcheck
-rwxr-xr-x 1 xxxxxx yyyyyy   4927730 Jul 16 01:55 genswaggertypedocs
-rwxr-xr-x 1 xxxxxx yyyyyy 205458748 Jul 16 01:55 genkubedocs
-rwxr-xr-x 1 xxxxxx yyyyyy 155653412 Jul 16 01:55 e2e.test
-rwxr-xr-x 1 xxxxxx yyyyyy  49252169 Jul 16 01:52 kubectl
-rwxr-xr-x 1 xxxxxx yyyyyy  46156229 Jul 16 01:52 kube-proxy
-rwxr-xr-x 1 xxxxxx yyyyyy  50989514 Jul 16 01:52 kubeadm
-rwxr-xr-x 1 xxxxxx yyyyyy 144291264 Jul 16 01:52 kubelet
-rwxr-xr-x 1 xxxxxx yyyyyy  52024300 Jul 16 01:50 kube-aggregator
-rwxr-xr-x 1 xxxxxx yyyyyy 168751462 Jul 16 01:50 kube-apiserver
-rwxr-xr-x 1 xxxxxx yyyyyy   2231154 Jul 16 01:50 mounter
-rwxr-xr-x 1 xxxxxx yyyyyy 122701327 Jul 16 01:50 cloud-controller-manager
-rwxr-xr-x 1 xxxxxx yyyyyy 136828051 Jul 16 01:50 kube-controller-manager
-rwxr-xr-x 1 xxxxxx yyyyyy 206769888 Jul 16 01:50 hyperkube
-rwxr-xr-x 1 xxxxxx yyyyyy  53220784 Jul 16 01:50 apiextensions-apiserver
-rwxr-xr-x 1 xxxxxx yyyyyy  49501810 Jul 16 01:50 kube-scheduler
-rwxr-xr-x 1 xxxxxx yyyyyy   2649635 Jul 16 01:40 go-bindata
-rwxr-xr-x 1 xxxxxx yyyyyy  11885551 Jul 16 01:40 openapi-gen
-rwxr-xr-x 1 xxxxxx yyyyyy   6770458 Jul 16 01:40 conversion-gen
-rwxr-xr-x 1 xxxxxx yyyyyy   6764729 Jul 16 01:39 defaulter-gen
-rwxr-xr-x 1 xxxxxx yyyyyy   6770309 Jul 16 01:39 deepcopy-gen

Transfer the binaries to RPI machine

scp -r arm  rpiuser@192.168.1.94:/home/rpiuser/kubernetes_binaries

Build the docker image

$ mkdir -p images/kube-controller-manager
$ cd images/kube-controller-manager
$ cp $HOME/kubernetes_binaries/kube-controller-manager

Create a simple Dockerfile to update the executable

$ cat Dockerfile

FROM k8s.gcr.io/kube-controller-manager-arm:v1.11.0

COPY kube-controller-manager /usr/local/bin/kube-controller-manager

Build the image

$ docker build -t jbrette/kube-controller-manager-arm:v1.11.1

Because of the brute force approach the image is twice as big since the executable (130MB) is contained in two layers of the container image

$ docker image list

REPOSITORY                               TAG                 IMAGE ID            CREATED             SIZE
jbrette/kube-controller-manager-arm      v1.11.1             b7023eef8fdf        8 hours ago         275MB
k8s.gcr.io/kube-apiserver-arm            v1.11.0             383bd2c4314e        2 weeks ago         170MB
k8s.gcr.io/kube-controller-manager-arm   v1.11.0             5b25f8a97aec        2 weeks ago         138MB
k8s.gcr.io/kube-scheduler-arm            v1.11.0             555ee860fa3c        2 weeks ago         50.5MB
k8s.gcr.io/kube-proxy-arm                v1.11.0             d7ebe361fe95        2 weeks ago         89.1MB
k8s.gcr.io/coredns                       1.1.3               7ceeb40862fb        7 weeks ago         31.3MB
k8s.gcr.io/etcd-arm                      3.2.18              ae02bf7047c8        3 months ago        221MB
quay.io/coreos/flannel                   v0.10.0-arm         c663d02f7966        5 months ago        39.9MB
k8s.gcr.io/pause-arm                     3.1                 e11a8cbeda86        6 months ago        374kB
k8s.gcr.io/pause                         3.1                 e11a8cbeda86        6 months ago        374kB

First push the controller-manager docker image to the repo

$ docker login

$ docker push jbrette/kube-controller-manager-arm:v1.11.1

Updating Kubeadm and Kubernetes configuration to use the patched images

Because I only rebuild one image, I can not use the kubeadm feature which allows to point to a different repo. Have to edit manually the configuration.

$ more kube-controller-manager.yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ""
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --address=127.0.0.1
    - --allocate-node-cidrs=true
    - --cluster-cidr=10.244.0.0/16
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --node-cidr-mask-size=24
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --use-service-account-credentials=true
    image: jbrette/kube-controller-manager-arm:v1.11.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10252
        scheme: HTTP
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-controller-manager
    resources:
      requests:
        cpu: 200m
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/kubernetes/controller-manager.conf
      name: kubeconfig
      readOnly: true
    - mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
      name: flexvolume-dir
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/kubernetes/controller-manager.conf
      type: FileOrCreate
    name: kubeconfig
  - hostPath:
      path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
      type: DirectoryOrCreate
    name: flexvolume-dir
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
status: {}

Conclusion

Need to streamline the build process
Need to streamline the image patching processed

Reference Links

[Kubernetes cross build]()