Canonical Voices

What KTsakalozos @ Medium talks about

K.Tsakalozos

In our last post we discussed the steps required to build the Canonical Distribution of Kubernetes (CDK). That post should give you a good picture of the components coming together to form a release. Some of these components are architecture agnostic, some not. Here we will update CDK to support IBM’s s390x architecture.

The CDK bundle is made of charms in python running on Ubuntu. That means we are already in a pretty good shape in terms of running on multiple architectures.

However, charms deploy binaries that are architecture specific. These binaries are of two types:

  1. snaps and
  2. juju resources

Snap packages are cross-distribution but unfortunately they are not cross-architecture. We need to build snaps for the architecture we are targeting. Snapped binaries include Kubernetes with its addons as well as etcd.

There are a few other architecture specific binaries that are not snapped yet. Flannel and CNI plugins are consumed by the charms as Juju resources.

Build snaps for s390x

CDK uses snaps to deploy a) Kubernetes binaries b) add-on services and,
c) etcd binaries.

Snaps with Kubernetes binaries

Kubernetes snaps are built using the branch in, https://github.com/juju-solutions/release/tree/rye/snaps/snap. You will need to login to your s390x machine, clone the repository and checkout the right branch:

On your s390x machine:
> git clone https://github.com/juju-solutions/release.git
> cd release
> git checkout rye/snaps
> cd snap

At this point we can build the Kubernetes snaps:

On your s390x machine:
> make KUBE_ARCH=s390x KUBE_VERSION=v1.7.4 kubectl kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy kubeadm kubefed

The above will fetch the Kubernetes released binaries from upstream and package them in snaps. You should see a bunch of *_1.7.4_-s390x.snap files in the snap directory.

Snap Kubernetes addons

Apart from the Kubernetes binaries CDK also packages the Kubernetes addons as snaps. The process of building the cdk-addons_1.7.4_s390x.snap is almost identical to the Kubernetes snaps. Clone the cdk-addons repository and make the addons based on the Kubernetes version:

On your s390x machine:
> git clone https://github.com/juju-solutions/cdk-addons.git
> cd cdk-addons
> make KUBE_VERSION=v1.7.4 KUBE_ARCH=s390x

Snap with etcd binaries

The last snap we will need is the snap for etcd. This time we do not have fancy Makafiles but the build process is still very simple:

> git clone https://github.com/tvansteenburgh/etcd-snaps
> cd etcd-snaps/etcd-2.3
> snapcraft — target-arch s390x

At this point you should have etcd_2.3.8_s390x.snap. This snap as well as the addons one and one snap for each of the kubectl kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy kubeadm kubefed will be attached to the Kubernetes charms upon release.

Make sure you go through this great post on using Kubernetes snaps.

Build juju resources for s390x

CDK needs to deploy the binaries for CNI and Flannel. These two binaries are not packaged as snaps (they might be in the future), instead they are provided as Juju resources. As these two binaries are architecture specific we need to build them for s390x.

CNI resource

For CNI you need to clone the respective repository, checkout the version you need and compile using a docker environment:

> git clone https://github.com/containernetworking/cni.git cni 
> cd cni
> git checkout -f v0.5.1
> docker run — rm -e “GOOS=linux” -e “GOARCH=s390x” -v ${PWD}/cni:/cni golang /bin/bash -c “cd /cni && ./build”

Have a look at how CI creates the tarball needed with the CNI binaries: https://github.com/juju-solutions/kubernetes-jenkins/blob/master/resources/build-cni.sh

Flannel resource

Support for s390x was added from version v0.8.0. Here is the usual cloning and building the repository.

> git clone https://github.com/coreos/flannel.git flannel
> cd flannel
> checkout -f v0.8.0
> make dist/flanneld-s390x

If you look at the CI script for Flannel resource you will see that the tarball also contains the CNI binaries and the etcd client: https://github.com/juju-solutions/kubernetes-jenkins/blob/master/resources/build-flannel.sh

Building etcd:

> git clone https://github.com/coreos/etcd.git etcd 
> cd etcd
> git checkout -f v2.3.7
> docker run --rm -e "GOOS=linux" -e "GOARCH=s390x -v ${PWD}/etcd:/etcd golang /bin/bash -c "cd /etcd &&./build"

Update charms

As we already mentioned, charms are written in python so they run on any platform/architecture. We only need to make sure they are fed the right architecture specific binaries to deploy and manage.

Kubernetes charms have a configuration option called channel. Channel points to a fallback snap channel where snaps are fetched from. It is a fallback because snaps are also attached to the charms when releasing them, and priority is given to those attached snaps. Let me a explain how this mechanism works. When releasing a Kubernetes charm you need to provide a resource file for each of the snaps the charm will deploy. If you upload a zero sized file (.snap) the charm will consider this as a dummy snap and try to snap-install the respective snap from the official snap repository. Grabbing the snaps from the official repository works towards a single multi arch charm since the arch specific repository is always available.

For the non snapped binaries we build above (cni and flannel) we need to patch the charms to make sure those binaries are available as Juju resources. In this pull request we add support for multi arch non snapped resources. We add an additional Juju resource for cni built s390x arch.

# In the metadata.yaml file of Kubernetes worker charm
cni-s390x: type: file
etype: file
filename: cni.tgz
description: CNI plugins for amd64

The charm will concatenate “cni-” and the architecture of the system to form the name of the right cni resource. On an amd64 the resource to be used is cni-amd64, on an s390x the cni-s390x is used.

We follow the same approach for the flannel charm. Have a look at the respective pull request.

Build and release

We would need to build the charms, push them to the store, attach the resources and release them. Lets trace these steps for kubernetes-worker:

> cd <path_to_kubernetes_worker_layer>
> charm build
> cd <output_directory_probably_$JUJU_REPOSITORY/builds/kubernetes-worker>
> charm push . cs:~kos.tsakalozos/kubernetes-worker-s390x

You will get a revision number for your charm. In my case it was 0. Lets list the resources we have for that charm:

> charm list-resources cs:~kos.tsakalozos/kubernetes-worker-s390x-0

And attach the resources to the uploaded charm:

> cd <where_your_resources_are>
> charm attach ~kos.tsakalozos/kubernetes-worker-s390x-0 kube-proxy=./kube-proxy.snap
> charm attach ~kos.tsakalozos/kubernetes-worker-s390x-0 kubectl=./kubectl.snap
> charm attach ~kos.tsakalozos/kubernetes-worker-s390x-0 kubelet=./kubelet.snap
> charm attach ~kos.tsakalozos/kubernetes-worker-s390x-0 cni-s390x=./cni-s390x.tgz
> charm attach ~kos.tsakalozos/kubernetes-worker-s390x-0 cni-amd64=./cni-amd64.tgz

Notice how we have to attach one resource per architecture for the cni non-snapped resource. You can try to provide a valid amd64 build for cni but since we are not building a multi arch charm it wouldn’t matter as it will not be used on the targeted s390x platform.

Now let’s release and grant read visibility to everyone:

> charm release cs:~kos.tsakalozos/kubernetes-worker-s390x-0 — channel edge -r cni-s390x-0 -r cni-amd64-0 -r kube-proxy-0 -r kubectl-0 -r kubelet-0
> charm grant cs:~kos.tsakalozos/kubernetes-worker-s390x-0 everyone

We omit building and releasing rest of the charms for brevity.

Summing up

Things have lined up really nice for CDK to support multiple hardware architectures. The architecture specific binaries are well contained into Juju resources and most of them are now shipped as snaps. Building each binary is probably the toughest part in the process outlined above. Each binary has its own dependencies and build process but the scripts linked here reduce the complexity a lot.

In the future we plan to support other architectures. We should end-up with a single bundle that would deploy on any architecture via a juju deploy canonical-kubernetes. You can follow our progress on our trello board. Do not hesitate to reach out and suggest improvements and tell us what architecture you think we should target next.

Read more
K.Tsakalozos

Patch CDK #1: Build & Release

Happens all the time. You often come across a super cool open source project you would gladly contribute but setting up the development environment and learning to patch and release your fixes puts you off. The Canonical Distribution of Kubernetes (CDK) is not an exception. This set of blog posts will shed some light on the most dark secrets of CDK.

Welcome to the CDK patch journey!

What is your Build & Release workflow? (Figure from xkcd)

Build CDK from source

Prerequisits

You would need to have Juju configured and ready to build charms. We will not be covering that in this blog post. Please, follow the official documentation to setup your environment and build you own first charm with layers.

Build the charms

CDK is made of a few charms, namely:

To build each charm you need to spot the top level charm layer and do a `charm build` on it. The links on the above list will get you to the github repository you will need to clone and build. Lets try this out for easyrsa:

> git clone https://github.com/juju-solutions/layer-easyrsa
Cloning into ‘layer-easyrsa’…
remote: Counting objects: 55, done.
remote: Total 55 (delta 0), reused 0 (delta 0), pack-reused 55
Unpacking objects: 100% (55/55), done.
Checking connectivity… done.
> cd ./layer-easyrsa/
> charm build
build: Composing into /home/jackal/workspace/charms
build: Destination charm directory: /home/jackal/workspace/charms/builds/easyrsa
build: Processing layer: layer:basic
build: Processing layer: layer:leadership
build: Processing layer: easyrsa (from .)
build: Processing interface: tls-certificates
proof: OK!

The above builds the easyrsa charm and prints the output directory (/home/jackal/workspace/charms/builds/easyrsa in this case).

Building the kubernetes-* charms is slightly different. As you might already know the kubernetes charm layers are already upstream under cluster/juju/layers. Building the respective charms requires you to clone the kubernetes repository and pass the path to each layer to your invocation of charm build. Let’s build the kubernetes worker layer here:

> git clone https://github.com/kubernetes/kubernetes
Cloning into ‘kubernetes’…
remote: Counting objects: 602553, done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 602553 (delta 18), reused 20 (delta 15), pack-reused 602481
Receiving objects: 100% (602553/602553), 456.97 MiB | 2.91 MiB/s, done.
Resolving deltas: 100% (409190/409190), done.
Checking connectivity… done.
> cd ./kubernetes/
> charm build cluster/juju/layers/kubernetes-worker/
build: Composing into /home/jackal/workspace/charms
build: Destination charm directory: /home/jackal/workspace/charms/builds/kubernetes-worker
build: Processing layer: layer:basic
build: Processing layer: layer:debug
build: Processing layer: layer:snap
build: Processing layer: layer:nagios
build: Processing layer: layer:docker (from ../../../workspace/charms/layers/layer-docker)
build: Processing layer: layer:metrics
build: Processing layer: layer:tls-client
build: Processing layer: layer:nvidia-cuda (from ../../../workspace/charms/layers/nvidia-cuda)
build: Processing layer: kubernetes-worker (from cluster/juju/layers/kubernetes-worker)
build: Processing interface: nrpe-external-master
build: Processing interface: dockerhost
build: Processing interface: sdn-plugin
build: Processing interface: tls-certificates
build: Processing interface: http
build: Processing interface: kubernetes-cni
build: Processing interface: kube-dns
build: Processing interface: kube-control
proof: OK!

During charm build all layers and interfaces referenced recursively starting by the top charm layer are fetched and merged to form your charm. The layers needed for building a charm are specified in a layer.yaml file on the root of the charm’s directory. For example, looking at cluster/juju/layers/kubernetes-worker/layer.yaml we see that the kubernetes worker charm uses the following layers and interfaces:

- 'layer:basic'
- 'layer:debug'
- 'layer:snap'
- 'layer:docker'
- 'layer:metrics'
- 'layer:nagios'
- 'layer:tls-client'
- 'layer:nvidia-cuda'
- 'interface:http'
- 'interface:kubernetes-cni'
- 'interface:kube-dns'
- 'interface:kube-control'

Layers is an awesome way to share operational logic among charms. For instance, the maintainers of the nagios layer have a better understanding of the operational needs of nagios but that does not mean that the authors of the kubernetes charms cannot use it.

charm build will recursively lookup each layer and interface at http://interfaces.juju.solutions/ to figure out where the source is. Each repository is fetched locally and squashed with all the other layers to form a single package, the charm. Go ahead a do a charm build with “-l debug” to see how and when a layer is fetched. It is important to know that if you already have a local copy of a layer under $JUJU_REPOSITORY/layers or interface under $JUJU_REPOSITORY/interfaces charm build will use those local forks instead of fetching them from the registered repositories. This enables charm authors to work on cross layer patches. Note that you might need to rename the directory of your local copy to match exactly the name of the layer or interface.

Building Resources

Charms will install Kubernetes but to do so they need to have the Kubernetes binaries. We package these binaries in snaps so that they are self-contained and deployed in any Linux distribution. Building such binaries is pretty straight forward as long as you know where to find them :)

Here is the repository holding the Kubernetes snaps: https://github.com/juju-solutions/release.git. The branch we want is rye/snaps:

> git clone https://github.com/juju-solutions/release.git
Cloning into ‘release’…
remote: Counting objects: 1602, done.
remote: Total 1602 (delta 0), reused 0 (delta 0), pack-reused 1602
Receiving objects: 100% (1602/1602), 384.69 KiB | 236.00 KiB/s, done.
Resolving deltas: 100% (908/908), done.
Checking connectivity… done.
> cd release
> git checkout rye/snaps
Branch rye/snaps set up to track remote branch rye/snaps from origin.
Switched to a new branch ‘rye/snaps’

Have a look at the README.md inside the snap directory to see how to build the snaps:

> cd snap/
> ./docker-build.sh KUBE_VERSION=v1.7.4

A number of .snap files should be available after the build.

In similar fashion you can build the snap package holding Kubernetes addons. We refer to this package as cdk-addons and it can be found at: https://github.com/juju-solutions/cdk-addons.git

> git clone https://github.com/juju-solutions/cdk-addons.git
Cloning into ‘cdk-addons’…
remote: Counting objects: 408, done.
remote: Total 408 (delta 0), reused 0 (delta 0), pack-reused 408
Receiving objects: 100% (408/408), 51.16 KiB | 0 bytes/s, done.
Resolving deltas: 100% (210/210), done.
Checking connectivity… done.
> cd cdk-addons/
> make

The last resource you will need (which is not packaged as a snap) is the container network interface (cni). Lets grab the repository and get to a release tag:

> git clone https://github.com/containernetworking/cni.git
Cloning into ‘cni’…
remote: Counting objects: 4048, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 4048 (delta 0), reused 2 (delta 0), pack-reused 4043
Receiving objects: 100% (4048/4048), 1.76 MiB | 613.00 KiB/s, done.
Resolving deltas: 100% (1978/1978), done.
Checking connectivity… done.
> cd cni
> git checkout -f v0.5.1

Build and package the cni resource:

> docker run — rm -e “GOOS=linux” -e “GOARCH=amd64” -v `pwd`:/cni golang /bin/bash -c “cd /cni && ./build”
Building API
Building reference CLI
Building plugins
flannel
tuning
bridge
ipvlan
loopback
macvlan
ptp
dhcp
host-local
noop
> cd ./bin
> tar -cvzf ../cni.tgz *
bridge
cnitool
dhcp
flannel
host-local
ipvlan
loopback
macvlan
noop
ptp
tuning

You should now have a cni.tgz in the root folder of the cni repository.

Two things to note here:
- We do have a CI for building, testing and releasing charms and bundles. In case you want to follow each step of the build process, you can find our CI scripts here: https://github.com/juju-solutions/kubernetes-jenkins
- You do not need to build all resources yourself. You can grab the resources used in CDK from the Juju store. Starting from the canonical-kubernetes bundle you can navigate to any charm shipped. Select one from the very end of the bundle page and then look for the “resources” sidebar on the right. Download any of them, rename it properly and you are ready to use it in your release.

Releasing Your Charms

After patching the charms to match your needs, please, consider submitting a pull request to tell us what you have been up to. Contrary to many other projects you do not need to wait for your PR to get accepted before you can actually make your work public. You can immediately release your work under your own namespace on the store. This is described in detail on the official charm authors documentation. The developers team is often using private namespaces to test PoCs and new features. The main namespace where CDK is released from is “containers”.

Yet, there is one feature you need to be aware of when attaching snaps to your charms. Snaps have their own release cycle and repositories. If you want to use the officially released snaps instead of attaching them to the charms, you can use a dummy zero sized file with the correct extension (.snap) in the place of each snap resource. The snap layer will see that that resource is empty and will try to grab the snap from the official repositories. Using the official snaps is recommended, however, in network restricted environments you might need to attach your own snaps while you deploy the charms.

Why is it this way?

Building CDK is of average difficulty as long as you know how where to look. It is not perfect by any standards and it will probably continue this path. The reason is that there are opposing forces shaping the build processes. This should come as no surprise. As Kubernetes changes rapidly and constantly expands, the build and release process should be flexible enough to include any new artefacts. Consider for example the switch from flannel cni to calico. In our case it is a resource and a charm that need to be updated. A monolithic build script would have been more “elegant” to the outsiders (eg, make CDK), but we would have been hiding a lot under the carpet. CI should be part of the culture of any team and should be owned by the team or else you get disconnected from the end product causing delays and friction. Our build and release process might look a bit “dirty” with a lot of moving parts but it really is not that bad! I managed to highlight the build and release steps in a single blog. Positive feedback also comes from our field engineers. Most of the time CDK deploys out of the box. When our field engineers are called it is either because our customers have a special requirement from the software or they have a “unconventional” environment in which Kubernetes needs to be deployed. Having such a simple and flexible build and release process enables our people to solve a problem on-site and release it tothe Juju store within a couple of hours.

Next steps

This blog post serves as foundation work for what is coming up. The plan is to go over some easy patches so we further demystify how CDK works. Funny thing this post was originally titled “Saying goodbye to my job security”. Cheers!

Read more
K.Tsakalozos

Sometimes we miss the forest for the trees.

It’s all about portability

Take a step back and think about how much of your effort goes into your product’s portability. A lot, I’d wager. You don’t wait to see what hardware your customer is on before you start coding your app. Widely accepted architectures give you a large customer base, so you target those. You skip optimisations in order to maximize compatibility. The language you are using ensures you can move to another platform/architecture.

Packaging your application is also important if you want to broaden your audience. The latest trend here is to use containers such as docker. Essentially containers (much like VMs) are another layer of abstraction for the sake of portability. You can package your app such that it will run anywhere.

https://medium.com/media/e87190de95f6f33f07f75beb4c8e7f39/href

Things go wrong

When you first create a PoC, even a USB stick is enough to distribute it! After that you should take things more seriously. I see products “dockerised” so that they run everywhere, yet the deployment story is limited to a single cloud. You do all the work to deliver your app everywhere only to fail when you actually need to deliver it! It doesn’t make sense — or to put it more accurately it only makes sense in the short term.

The overlooked feature

Canonical Distribution of Kubernetes (CDK) will deploy exactly the same Kubernetes in all major clouds (AWS, GCE, Azure, VMware, Openstack, Joynet), bare metal, or even your local machine (using lxd). When you first evaluate CDK you may think: “I do not need this”, or even “I hate these extra two lines where I decide the cloud provider because I already know I am going full AWS”. At that point you are tempted to take the easy way out and get pinned to a single Kubernetes provider. However, soon you will find yourself in need of feature F from cloud C or Kubernetes version vX.Y.Z. Worse is when your potential customer is on a different platform and you discover the extent of dependencies you have on your provider.

Before you click off this tab thinking I am exaggerating, think about this: the best devops are done by the devs themselves. How easy is it to train all devs to use AWS for spawning containers? It is trivial, right? Suppose you get two more customers, one on Azure and one on GCE. How easy is it to train your team to perform devops against two additional clouds? You now have to deal with three different monitoring and logging mechanisms. That might still be doable since AWS, GCE, and Azure are all well behaved. What if a customer wants their own on-prem cluster? You are done — kiss your portability and velocity goodbye. Your team will be constantly firefighting, chasing after misaligned versions, and emulating features across platforms.

How is it done

Under the hood of CDK you will find Juju. Juju abstracts the concepts of IaaS. It will provision everything from bare metal machines in your rack, to VMs on any major cloud, to lxd containers on your laptop. On top of these units Juju will deploy Kubernetes thus keeping the deployment agnostic to the underlying provider. Two points here:

  1. Juju has been here for years and is used both internally and externally by Canonical to successfully deliver solutions such as Openstack and Big Data.
  2. Since your Kubernetes is built on IaaS cloud offerings you are able to tailor node properties to match your needs. You may also find it is more cost effective.

Also check out Conjure-up, an easy-to-use wizard that will get your Kubernetes cluster up-and-running in minutes.

To conclude

I am sure you have your reasons to “dockerize” your application and then deliver it on a single cloud. It is just counter-intuitive. If you want to make the most from your portability decisions you should consider CDK.

References:

Read more
K.Tsakalozos

Introduction

In this post we discus our efforts to setup a Federation of on-prem Kubernetes Clusters using CoreDNS. The Kubernetes Cluster version used is 1.6.2. We use Juju to deliver clusters on AWS, yet the clusters should be considered on-prem since they do not integrate with any of the cloud’s features. The steps described here are repeatable on any pool of resources you may have available (cloud or bare metal). Note that this is still a work in progress and should not be used on a production environment.

What is Federation

Cluster Federation (Fig. from http://blog.kubernetes.io/2016/07/cross-cluster-services.html)

Cluster federation allows you to “connect” your Kubernetes clusters and manage certain functionality from a central point. The need for central management arises when you have more than one clusters possibly spread across different locations. You may need a globally distributed infrastructure for performance, availability or even legal reasons. You might want to have isolated clusters satisfying the needs of different teams. Whatever the case may be, federation can assist in some of the ops tasks. You can find more on the benefits of federation in the official documentation.

Why Cluster Federation with Juju

Juju makes delivery of Kubernetes clusters dead simple! Juju builds an abstraction over the pool of resources you have and allows us to deliver Kubernetes to all the main public and private clouds (AWS, Azure, GCE, Openstack, Oracle etc) and bare metal physical clusters. Juju also enables you to experiment with small scale deployments stood up on your local machine using lxc containers.

Having the option to deploy a cluster with the ease Juju offers, federation comes as a natural next step. Deployed clusters should be considered on-prem clusters as they are exactly the same as if they were deployed on bare metal. This is simply because even if you deploy on a cloud Juju will provision machines (virtual machines in this case) and deploy Kubernetes as if it were a local on-prem deployment. You see, Juju is not magic after all!

There is a excellent blog post showing how you can place Juju clusters under a federation hosting cluster setup on GCE. Here we focus on a federation that has no dependencies on a specific cloud provider. To this end, we go with CoreDNS as a federation DNS provider.

Let’s Federate

Deploy clusters

Our setup is made of 3 clusters:

  • f1: will host the federation controller place
  • c1, c2: federated clusters

At the time of this writing (2nd of June 2017) cluster federation is in beta and is not yet integrated into the official Kubernetes charms. Therefore we will be using a patched version of charms and bundles (source of charms and cdk-addons, charms master and worker, and bundle).

We assume you already have Juju installed and a controller in place. If not have a look at the Juju installation instructions.

Let’s deploy these three clusters.

#> juju add-model f1
#> juju deploy cs:~kos.tsakalozos/bundle/federated-kubernetes-core-2
#> juju add-model c1
#> juju deploy cs:~kos.tsakalozos/bundle/federated-kubernetes-core-2
#> juju add-model c2
#> juju deploy cs:~kos.tsakalozos/bundle/federated-kubernetes-core-2

Here is what the bundle.yaml looks like: http://paste.ubuntu.com/24746868/

Let’s copy and merge the kubeconfig files.

#> juju switch f1
#> juju scp kubernetes-master/0:config ~/.kube/config
#> juju switch c1
#> juju scp kubernetes-master/0:config ~/.kube/config.c1
#> juju switch c2
#> juju scp kubernetes-master/0:config ~/.kube/config.c2

Using this script you can merge the configs

#> ./merge-configs.py config.c1 c1
#> ./merge-configs.py config.c2 c2

To enable CoreDNS on f1 you just need to:

#> juju switch f1
#> juju config kubernetes-master enable-coredns=True

Have a look at your three contexts:

#> kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
f1 f1 ubuntu
c1 c1-cluster c1-user
c2 c2-cluster c2-user

Create the federation

At this point you are ready to deploy the federation control plane on f1. To do so you will need a call to kubefed init. Kubefed init will deploy the required services and it will also perform a health check ping to see everything is ready. During this last check kubefed init will try to reach a service on a port that is not open. At the time of this writing we have a patch waiting upstream to make the port configurable so you can open it before triggering the federation creation. If you do not want to wait for the patch to be merged you can call kubefed init with -v 9 to see what port the services were appointed to and expose them though Juju while keeping kubefed running. This will become clear shortly.

Create a coredns-provider.conf file with the etc endpoint of etcd of f1 and your zone. You can look at the Juju status to find out the IP of the etcd machine:

#> cat ./coredns-provider.conf 
[Global]
etcd-endpoints = http://54.211.106.117:2379
zones = juju-fed.com.

Call kubefed

#> juju switch f1
#> kubefed init federation — host-cluster-context=f1 — dns-provider=”coredns” — dns-zone-name=”juju-fed.com.” — dns-provider-config=coredns-provider.conf — etcd-persistent-storage=false — api-server-service-type=”NodePort” -v 9 — api-server-advertise-address=34.224.101.147 — image=gcr.io/google_containers/hyperkube-amd64:v1.6.2

Note here that the api-server-advertise-address=34.224.101.147 is the IP of the worker on your f1 cluster. You selected NodePort so the federation service is available from all the worker nodes under a random port.

Kubefed will end up trying to do a health check. You should see on your console something similar to this: “GET https://34.224.101.147:30229/healthz” . The port 30229 is randomly selected and will differ in your case. While keeping kubefed init running, go ahead and open this port from a second terminal:

#> juju run — application kubernetes-worker “open-port 30229”

The federation context should be present in the list of contexts you have.

#> kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
f1 f1 ubuntu
federation federation federation
c1 c1-cluster c1-user
c2 c2-cluster c2-user

Now you are ready to have other clusters join the federation:

#> kubectl config use-context federation
#> kubefed join c1 — host-cluster-context=f1
#> kubefed join c2 — host-cluster-context=f1

Two clusters should soon appear ready:

#> kubectl get clusters
NAME STATUS AGE
c1 Ready 3m
c2 Ready 1m

Congratulations! You are all set, your federation is ready to use.

What works

Setup your default namespace:

#> kubectl — context=federation create ns default

Get this replica set and create it.

#> kubectl — context=federation create -f nginx-rs.yaml

See your replica set created:

#> kubectl — context=federation describe rs/nginx
Name: nginx
Namespace: default
Selector: app=nginx
Labels: app=nginx
type=demo
Annotations: <none>
Replicas: 1 current / 1 desired
Pods Status: error in fetching pods: the server could not find the requested resource
Pod Template:
Labels: app=nginx
Containers:
frontend:
Image: nginx
Port: 80/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -
26s 26s 1 federated-replicaset-controller Normal CreateInCluster Creating replicaset in cluster c2

Create a service.

#> kubectl — context=federation create -f beacon.yaml

Scale and open ports so the service is available

#> kubectl — context=federation scale rs/nginx — replicas 3

What does not work yet

Federation does not seem to handle NodePort services. To see that you can deploy a busybox and try to resolve the name of the service you deployed right above.

Use this yaml.

#> kubectl — context=c2 create -f bbox-4-test.yaml
#> kubectl — context=c2 exec -it busybox — nslookup beacon

This returns a local cluster IP which is rather unexpected.

There are some configuration options that did not have the effect I was expecting. First, we would want to expose the CoreDNS service as described in the docs. Assigning the same NodePort for both UDP and TCP had no effect; this is rather unfortunate since the issue is already addressed. The configuration described here and here (although it should be automatically applied I thought I would give it a try) did not seem to have any effect.

Open issues:

Conclusions

Federation on on-prem clusters has gone a long way and is getting significant improvements with every release. Some of the issues described here have already been addressed, I am waiting anxiously for the upcoming v1.7.0 release to see the fixes in action.

Do not hesitate to reach out to the sig-federation group, but most importantly I would be extremely happy and grateful if you pointed out some bug I have on my configs/code/thoughts ;)

References and Links

  1. Official federations docs, https://kubernetes.io/docs/concepts/cluster-administration/federation/
  2. Juju docs, https://jujucharms.com/
  3. Canonical Distribution of Kubernetes, https://jujucharms.com/canonical-kubernetes/
  4. Federation with host on GCE, https://medium.com/google-cloud/experimenting-with-cross-cloud-kubernetes-cluster-federation-dfa99f913d54
  5. CDK-addons source, https://github.com/juju-solutions/cdk-addons/tree/feature/coredns
  6. Kubernetes charms, https://github.com/juju-solutions/kubernetes/tree/feature/coredns
  7. Kubernetes master charm, https://jujucharms.com/u/kos.tsakalozos/kubernetes-master/31
  8. Kubernetes worker charm, https://jujucharms.com/u/kos.tsakalozos/kubernetes-worker/17
  9. Kubernetes bundle used for federation, https://jujucharms.com/u/kos.tsakalozos/federated-kubernetes-core/bundle/0
  10. Juju getting started, https://jujucharms.com/docs/stable/getting-started
  11. Patch kubefed init, https://github.com/kubernetes/kubernetes/pull/46283
  12. CoreDNS setup instructions https://kubernetes.io/docs/tasks/federation/set-up-coredns-provider-federation/#setup-coredns-server-in-nameserver-resolvconf-chain
  13. CoreDNS yaml file https://github.com/juju-solutions/cdk-addons/blob/feature/coredns/coredns/coredns-rbac.yaml#L95
  14. Nodeport UDP/TCP issue, https://github.com/kubernetes/kubernetes/issues/20092
  15. Federation DNS on-prem configuration https://kubernetes.io/docs/admin/federation/#kubernetes-15-passing-federations-flag-via-config-map-to-kube-dns

Read more
K.Tsakalozos

Juju Made the Deadline

Ok… I am exaggerating. Juju did not make the deadline, Panagiotis and his co-authors with their hard work made the deadline. Ah, ok you caught me lying again… One of Panagiotis co-authors (me) did not work hard, actually he did not work on the paper at all! Normally, I would publicly apologize, but let me first explain why I am in the author’s list.

Panagiotis is a PhD researcher at the University of Athens. He is also a member of the Madgik lab, which is where I know him from. Panagiotis is interested in Graphs, so he is into Apache Giraph and GraphX for Apache Spark. Knowing how frustrating his work may get, I felt obliged to introduce him to Juju. The goal was to save him some time from deploying and configuring infrastructures and have him focus on real work.

Juju offers an often overlooked feature that proves to be immensely useful to researchers and people who just want to experiment with some software without committing to it. You can deploy an infrastructure on your local machine in less than 10 minutes, take it for a test drive, and then when you are happy, move to a cloud and test at scale. In the case of the Madgik lab, where Panagiotis is, getting cloud resources includes contacting the IT department and wait for them to find time and resources. I think I saw a spark in Panagiotis eyes when I showed him my laptop, or was it the reflection of the Spark infrastructure running in LXC containers, I don’t remember. He immediately showed me a Spark deployment of his own with a bunch of worker nodes that took him a week to set up. After all this time, he was still not sure if that configuration was appropriate for running tests and publishing results. What if he had misconfigured something? What if a minor config change (e.g.: cache size) would skew the results? That is not to say Juju has a magic way to optimally tune the infrastructure for your needs, but we try to choose sensible configuration variables based on the feedback we get from the community.

Panagiotis office looked chaotic, computer towers, screens and keyboards all over the place. Funny how he had prepared a VM for us to work on! Setting up Juju was flawless. Deploying a mapreduce processing bundle and scaling it within the limits of 48GB of RAM and 16 cores was a piece of cake. We even double checked that after a reboot all nodes would come up. All this under the eye of George, the head of the lab’s IT. After an hour or so, we said our goodbyes.

A few weeks later, I received an email titled SocInf 2016 submission and my name in the author’s list. Surprised, I asked, “Why?” The response was that, “without Juju we wouldn’t have made the deadline!”

So… you will find us at the 2nd International Workshop on Social Influence Analysis, co-located with the International Joint Conference on Artificial Intelligence (IJCAI 2016) in NY on the 9th of July. The paper is titled Pinpointing Influence in Pinterest. Kudos to Panagiotis Liakos, Katia Papakonstantinopoulou, Michael Sioutis, Alex Delis and the Juju Big Data team.

Original article

Originally published at insights.ubuntu.com on June 15, 2016.

Read more