Update Kubernetes to 1.11 on Ubuntu

Kubeadm is coming with an upgrade option. The goal of this study is to leverage the option.

Initial set up

Kubernetes 1.10.4 is installed

cat /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
sudo dpkg -l kubeadm

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                   Version                  Architecture             Description
+++-======================================-========================-========================-=================================================================================
ii  kubeadm                                1.10.4-00                amd64                    Kubernetes Cluster Bootstrapping Tool
sudo dpkg -l kubelet

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                   Version                  Architecture             Description
+++-======================================-========================-========================-=================================================================================
ii  kubelet                                1.10.4-00                amd64                    Kubernetes Node Agent
sudo dpkg -l kubectl

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                   Version                  Architecture             Description
+++-======================================-========================-========================-=================================================================================
ii  kubectl                                1.10.4-00                amd64                    Kubernetes Command Line Tool
kubectl version

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Update kubelet and kubectl

Update aptitude

sudo apt-get update

Hit:2 http://ppa.launchpad.net/longsleep/golang-backports/ubuntu xenial InRelease
Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Get:4 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [511 kB]
Get:5 http://security.ubuntu.com/ubuntu xenial-security/main i386 Packages [452 kB]
Get:6 http://security.ubuntu.com/ubuntu xenial-security/main amd64 DEP-11 Metadata [67.7 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/main DEP-11 64x64 Icons [68.0 kB]
Get:8 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [355 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/universe i386 Packages [303 kB]
Get:10 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [133 kB]
Get:11 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 DEP-11 Metadata [107 kB]
Get:12 http://security.ubuntu.com/ubuntu xenial-security/universe DEP-11 64x64 Icons [147 kB]
Hit:13 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Get:14 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:15 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:16 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [796 kB]
Get:17 http://us.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages [728 kB]
Get:18 http://us.archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [329 kB]
Get:19 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 DEP-11 Metadata [318 kB]
Get:20 http://us.archive.ubuntu.com/ubuntu xenial-updates/main DEP-11 64x64 Icons [228 kB]
Get:21 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [640 kB]
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]
Get:22 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages [585 kB]
Get:23 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [17.9 kB]
Get:24 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [257 kB]
Get:25 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 DEP-11 Metadata [246 kB]
Get:26 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe DEP-11 64x64 Icons [331 kB]
Get:27 http://us.archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 DEP-11 Metadata [5,964 B]
Get:28 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [6,740 B]
Get:29 http://us.archive.ubuntu.com/ubuntu xenial-backports/main i386 Packages [6,732 B]
Get:30 http://us.archive.ubuntu.com/ubuntu xenial-backports/main Translation-en [4,180 B]
Get:31 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 DEP-11 Metadata [3,328 B]
Get:32 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [7,412 B]
Get:33 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe i386 Packages [7,104 B]
Get:34 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe amd64 DEP-11 Metadata [5,100 B]
Get:35 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe DEP-11 64x64 Icons [1,789 B]
Fetched 7,000 kB in 9s (757 kB/s)

Run the apt-get upgrade command. Notice how the kubeadm is “kept back”

sudo apt-get upgrade


Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  linux-headers-4.13.0-26 linux-headers-4.13.0-26-generic linux-image-4.13.0-26-generic linux-image-extra-4.13.0-26-generic
Use 'sudo apt autoremove' to remove them.
The following packages have been kept back:
  kubeadm
The following packages will be upgraded:
  amd64-microcode console-setup console-setup-linux cpp-5 desktop-file-utils ebtables g++-5 gcc-5 gcc-5-base gir1.2-javascriptcoregtk-4.0 gir1.2-webkit2-4.0 gnome-software
  gnome-software-common keyboard-configuration kubectl kubelet libasan2 libatomic1 libcc1-0 libcilkrts5 libgcc-5-dev libgcrypt20 libgomp1 libitm1 libjasper1
  libjavascriptcoregtk-4.0-18 libldap-2.4-2 liblsan0 libmpx0 libplymouth4 libquadmath0 libssl-dev libssl1.0.0 libstdc++-5-dev libstdc++6 libtsan0 libubsan0
  libwebkit2gtk-4.0-37 libwebkit2gtk-4.0-37-gtk2 linux-firmware openssl plymouth plymouth-label plymouth-theme-ubuntu-logo plymouth-theme-ubuntu-text rfkill snapd
  ubuntu-core-launcher ubuntu-software update-notifier update-notifier-common wireless-regdb
52 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
sudo apt-get upgrade
sudo apt-get autoremove
sudo reboot
sudo dpkg -l kubectl
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                   Version                  Architecture             Description
+++-======================================-========================-========================-=================================================================================
ii  kubectl                                1.11.0-00                amd64                    Kubernetes Command Line Tool
sudo dpkg -l kubeadm
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                   Version                  Architecture             Description
+++-======================================-========================-========================-=================================================================================
ii  kubeadm                                1.10.4-00                amd64                    Kubernetes Cluster Bootstrapping Tool
sudo dpkg -l kubelet
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                   Version                  Architecture             Description
+++-======================================-========================-========================-=================================================================================
ii  kubelet                                1.11.0-00                amd64                    Kubernetes Node Agent
kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Upgrading kubeadm

Install new kubeadm and associated new cri-tools

sudo apt-get install kubeadm
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  cri-tools
The following NEW packages will be installed:
  cri-tools
The following packages will be upgraded:
  kubeadm
1 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 14.7 MB of archives.
After this operation, 70.9 MB disk space will be freed.
Do you want to continue? [Y/n] y
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.11.0-00 [5,309 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.11.0-00 [9,422 kB]
Fetched 14.7 MB in 5s (2,780 kB/s)
Selecting previously unselected package cri-tools.
(Reading database ... 225649 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.11.0-00_amd64.deb ...
Unpacking cri-tools (1.11.0-00) ...
Preparing to unpack .../kubeadm_1.11.0-00_amd64.deb ...
Unpacking kubeadm (1.11.0-00) over (1.10.4-00) ...
Setting up cri-tools (1.11.0-00) ...
Setting up kubeadm (1.11.0-00) ...
Installing new version of config file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf ...
kubeadm version

kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

Upgrading your Kubernetes cluster

Cluster does not seems be in such a great shape

NAME                                           READY     STATUS    RESTARTS   AGE
pod/calico-etcd-j8zqp                          1/1       Running   22         11d
pod/calico-kube-controllers-679568f47c-wgz7n   1/1       Running   27         11d
pod/calico-node-knstj                          2/2       Running   39         11d
pod/etcd-xxxxxxx                               0/1       Pending   0          1h
pod/kube-apiserver-xxxxxxx                     0/1       Pending   0          1h
pod/kube-controller-manager-xxxxxxx            0/1       Pending   0          1h
pod/kube-dns-86f4d74b45-fs7bq                  3/3       Running   66         11d
pod/kube-proxy-mpbf7                           1/1       Running   22         11d
pod/kube-scheduler-xxxxxxx                     0/1       Pending   0          1h
pod/tiller-deploy-5c688d5f9b-6cl5n             1/1       Running   23         11d

NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
service/calico-etcd     ClusterIP   10.96.232.136   <none>        6666/TCP        11d
service/kube-dns        ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP   11d
service/tiller-deploy   ClusterIP   10.97.44.240    <none>        44134/TCP       11d

NAME                         DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
daemonset.apps/calico-etcd   1         1         0         1            0           node-role.kubernetes.io/master=   11d
daemonset.apps/calico-node   1         1         0         1            0           <none>                            11d
daemonset.apps/kube-proxy    1         1         0         1            0           <none>                            11d

NAME                                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/calico-kube-controllers   1         1         1            0           11d
deployment.apps/kube-dns                  1         1         1            0           11d
deployment.apps/tiller-deploy             1         1         1            0           11d

NAME                                                 DESIRED   CURRENT   READY     AGE
replicaset.apps/calico-kube-controllers-679568f47c   1         1         0         11d
replicaset.apps/kube-dns-86f4d74b45                  1         1         0         11d
replicaset.apps/tiller-deploy-5c688d5f9b             1         1         0         11d
sudo kubeadm upgrade apply --dry-run 1.11.0 > /tmp/traces
[upgrade/health] FATAL: [preflight] Some fatal errors occurred:
        [ERROR MasterNodesReady]: there are NotReady masters in the cluster: [xxxxxxx]
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

TBD