Canonical Voices

Posts tagged with 'kubernetes'

K. Tsakalozos

I read the Hacker News post Heptio Contour and I thought “Cool! A project from our friends at Heptio, lets see what they got for us”. I wont lie to you, at first I was a bit disappointed because there was no special mention for Canonical Distribution of Kubernetes (CDK) but I understand, I am asking too much :). Let me cover this gap here.

Deploy CDK

To deploy CDK on Ubuntu you need to just do a:

> sudo snap install conjure-up --classic
> conjure-up kubernetes-core

Using ‘kubernetes-core’ will give you a minimal k8s cluster — perfect for our use case. For a larger, more robust cluster, try ‘canonical-kubernetes.’

Deploy Contour and Demo App

CDK already comes with an ingress solution so you need to disable it and deploy Contour. Here we also deploy the demo kuard application:

> juju config kubernetes-worker ingress=false
> kubectl --kubeconfig=/home/jackal/.kube/config apply -f http://j.hept.io/contour-deployment-norbac
> kubectl --kubeconfig=/home/jackal/.kube/config apply -f http://j.hept.io/contour-kuard-example

Get Your App

The Contour service will be on a port that (depending on the cloud you are targeting) might be closed, so you need to open it before accessing kuard:

> kubectl --kubeconfig=/home/jackal/.kube/config get service -n heptio-contour  contour -o wide                                                                                                        
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
contour LoadBalancer 10.152.183.201 <pending> 80:31226/TCP 2m app=contour
juju run --application kubernetes-worker open-port 31226

And here it is running on AWS:

Instead of opening the ports to the outside world you could set the right DNS entries. However, this is specific to the cloud you are deploying to.

As the Contour README says “On AWS, create a CNAME record that maps the host in your Ingress object to the ELB address.”
“If you have an IP address instead (on GCE, for example), create an A record.”
For a localhost deployment your ports should not be blocked and you can fake a DNS entry by editing /etc/hosts.

Thank you Heptio. Keep it up!

Resources

Read more
K.Tsakalozos

In our last post we discussed the steps required to build the Canonical Distribution of Kubernetes (CDK). That post should give you a good picture of the components coming together to form a release. Some of these components are architecture agnostic, some not. Here we will update CDK to support IBM’s s390x architecture.

The CDK bundle is made of charms in python running on Ubuntu. That means we are already in a pretty good shape in terms of running on multiple architectures.

However, charms deploy binaries that are architecture specific. These binaries are of two types:

  1. snaps and
  2. juju resources

Snap packages are cross-distribution but unfortunately they are not cross-architecture. We need to build snaps for the architecture we are targeting. Snapped binaries include Kubernetes with its addons as well as etcd.

There are a few other architecture specific binaries that are not snapped yet. Flannel and CNI plugins are consumed by the charms as Juju resources.

Build snaps for s390x

CDK uses snaps to deploy a) Kubernetes binaries b) add-on services and,
c) etcd binaries.

Snaps with Kubernetes binaries

Kubernetes snaps are built using the branch in, https://github.com/juju-solutions/release/tree/rye/snaps/snap. You will need to login to your s390x machine, clone the repository and checkout the right branch:

On your s390x machine:
> git clone https://github.com/juju-solutions/release.git
> cd release
> git checkout rye/snaps
> cd snap

At this point we can build the Kubernetes snaps:

On your s390x machine:
> make KUBE_ARCH=s390x KUBE_VERSION=v1.7.4 kubectl kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy kubeadm kubefed

The above will fetch the Kubernetes released binaries from upstream and package them in snaps. You should see a bunch of *_1.7.4_-s390x.snap files in the snap directory.

Snap Kubernetes addons

Apart from the Kubernetes binaries CDK also packages the Kubernetes addons as snaps. The process of building the cdk-addons_1.7.4_s390x.snap is almost identical to the Kubernetes snaps. Clone the cdk-addons repository and make the addons based on the Kubernetes version:

On your s390x machine:
> git clone https://github.com/juju-solutions/cdk-addons.git
> cd cdk-addons
> make KUBE_VERSION=v1.7.4 KUBE_ARCH=s390x

Snap with etcd binaries

The last snap we will need is the snap for etcd. This time we do not have fancy Makafiles but the build process is still very simple:

> git clone https://github.com/tvansteenburgh/etcd-snaps
> cd etcd-snaps/etcd-2.3
> snapcraft — target-arch s390x

At this point you should have etcd_2.3.8_s390x.snap. This snap as well as the addons one and one snap for each of the kubectl kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy kubeadm kubefed will be attached to the Kubernetes charms upon release.

Make sure you go through this great post on using Kubernetes snaps.

Build juju resources for s390x

CDK needs to deploy the binaries for CNI and Flannel. These two binaries are not packaged as snaps (they might be in the future), instead they are provided as Juju resources. As these two binaries are architecture specific we need to build them for s390x.

CNI resource

For CNI you need to clone the respective repository, checkout the version you need and compile using a docker environment:

> git clone https://github.com/containernetworking/cni.git cni 
> cd cni
> git checkout -f v0.5.1
> docker run — rm -e “GOOS=linux” -e “GOARCH=s390x” -v ${PWD}/cni:/cni golang /bin/bash -c “cd /cni && ./build”

Have a look at how CI creates the tarball needed with the CNI binaries: https://github.com/juju-solutions/kubernetes-jenkins/blob/master/resources/build-cni.sh

Flannel resource

Support for s390x was added from version v0.8.0. Here is the usual cloning and building the repository.

> git clone https://github.com/coreos/flannel.git flannel
> cd flannel
> checkout -f v0.8.0
> make dist/flanneld-s390x

If you look at the CI script for Flannel resource you will see that the tarball also contains the CNI binaries and the etcd client: https://github.com/juju-solutions/kubernetes-jenkins/blob/master/resources/build-flannel.sh

Building etcd:

> git clone https://github.com/coreos/etcd.git etcd 
> cd etcd
> git checkout -f v2.3.7
> docker run --rm -e "GOOS=linux" -e "GOARCH=s390x -v ${PWD}/etcd:/etcd golang /bin/bash -c "cd /etcd &&./build"

Update charms

As we already mentioned, charms are written in python so they run on any platform/architecture. We only need to make sure they are fed the right architecture specific binaries to deploy and manage.

Kubernetes charms have a configuration option called channel. Channel points to a fallback snap channel where snaps are fetched from. It is a fallback because snaps are also attached to the charms when releasing them, and priority is given to those attached snaps. Let me a explain how this mechanism works. When releasing a Kubernetes charm you need to provide a resource file for each of the snaps the charm will deploy. If you upload a zero sized file (.snap) the charm will consider this as a dummy snap and try to snap-install the respective snap from the official snap repository. Grabbing the snaps from the official repository works towards a single multi arch charm since the arch specific repository is always available.

For the non snapped binaries we build above (cni and flannel) we need to patch the charms to make sure those binaries are available as Juju resources. In this pull request we add support for multi arch non snapped resources. We add an additional Juju resource for cni built s390x arch.

# In the metadata.yaml file of Kubernetes worker charm
cni-s390x: type: file
etype: file
filename: cni.tgz
description: CNI plugins for amd64

The charm will concatenate “cni-” and the architecture of the system to form the name of the right cni resource. On an amd64 the resource to be used is cni-amd64, on an s390x the cni-s390x is used.

We follow the same approach for the flannel charm. Have a look at the respective pull request.

Build and release

We would need to build the charms, push them to the store, attach the resources and release them. Lets trace these steps for kubernetes-worker:

> cd <path_to_kubernetes_worker_layer>
> charm build
> cd <output_directory_probably_$JUJU_REPOSITORY/builds/kubernetes-worker>
> charm push . cs:~kos.tsakalozos/kubernetes-worker-s390x

You will get a revision number for your charm. In my case it was 0. Lets list the resources we have for that charm:

> charm list-resources cs:~kos.tsakalozos/kubernetes-worker-s390x-0

And attach the resources to the uploaded charm:

> cd <where_your_resources_are>
> charm attach ~kos.tsakalozos/kubernetes-worker-s390x-0 kube-proxy=./kube-proxy.snap
> charm attach ~kos.tsakalozos/kubernetes-worker-s390x-0 kubectl=./kubectl.snap
> charm attach ~kos.tsakalozos/kubernetes-worker-s390x-0 kubelet=./kubelet.snap
> charm attach ~kos.tsakalozos/kubernetes-worker-s390x-0 cni-s390x=./cni-s390x.tgz
> charm attach ~kos.tsakalozos/kubernetes-worker-s390x-0 cni-amd64=./cni-amd64.tgz

Notice how we have to attach one resource per architecture for the cni non-snapped resource. You can try to provide a valid amd64 build for cni but since we are not building a multi arch charm it wouldn’t matter as it will not be used on the targeted s390x platform.

Now let’s release and grant read visibility to everyone:

> charm release cs:~kos.tsakalozos/kubernetes-worker-s390x-0 — channel edge -r cni-s390x-0 -r cni-amd64-0 -r kube-proxy-0 -r kubectl-0 -r kubelet-0
> charm grant cs:~kos.tsakalozos/kubernetes-worker-s390x-0 everyone

We omit building and releasing rest of the charms for brevity.

Summing up

Things have lined up really nice for CDK to support multiple hardware architectures. The architecture specific binaries are well contained into Juju resources and most of them are now shipped as snaps. Building each binary is probably the toughest part in the process outlined above. Each binary has its own dependencies and build process but the scripts linked here reduce the complexity a lot.

In the future we plan to support other architectures. We should end-up with a single bundle that would deploy on any architecture via a juju deploy canonical-kubernetes. You can follow our progress on our trello board. Do not hesitate to reach out and suggest improvements and tell us what architecture you think we should target next.

Read more
K.Tsakalozos

Patch CDK #1: Build & Release

Happens all the time. You often come across a super cool open source project you would gladly contribute but setting up the development environment and learning to patch and release your fixes puts you off. The Canonical Distribution of Kubernetes (CDK) is not an exception. This set of blog posts will shed some light on the most dark secrets of CDK.

Welcome to the CDK patch journey!

What is your Build & Release workflow? (Figure from xkcd)

Build CDK from source

Prerequisits

You would need to have Juju configured and ready to build charms. We will not be covering that in this blog post. Please, follow the official documentation to setup your environment and build you own first charm with layers.

Build the charms

CDK is made of a few charms, namely:

To build each charm you need to spot the top level charm layer and do a `charm build` on it. The links on the above list will get you to the github repository you will need to clone and build. Lets try this out for easyrsa:

> git clone https://github.com/juju-solutions/layer-easyrsa
Cloning into ‘layer-easyrsa’…
remote: Counting objects: 55, done.
remote: Total 55 (delta 0), reused 0 (delta 0), pack-reused 55
Unpacking objects: 100% (55/55), done.
Checking connectivity… done.
> cd ./layer-easyrsa/
> charm build
build: Composing into /home/jackal/workspace/charms
build: Destination charm directory: /home/jackal/workspace/charms/builds/easyrsa
build: Processing layer: layer:basic
build: Processing layer: layer:leadership
build: Processing layer: easyrsa (from .)
build: Processing interface: tls-certificates
proof: OK!

The above builds the easyrsa charm and prints the output directory (/home/jackal/workspace/charms/builds/easyrsa in this case).

Building the kubernetes-* charms is slightly different. As you might already know the kubernetes charm layers are already upstream under cluster/juju/layers. Building the respective charms requires you to clone the kubernetes repository and pass the path to each layer to your invocation of charm build. Let’s build the kubernetes worker layer here:

> git clone https://github.com/kubernetes/kubernetes
Cloning into ‘kubernetes’…
remote: Counting objects: 602553, done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 602553 (delta 18), reused 20 (delta 15), pack-reused 602481
Receiving objects: 100% (602553/602553), 456.97 MiB | 2.91 MiB/s, done.
Resolving deltas: 100% (409190/409190), done.
Checking connectivity… done.
> cd ./kubernetes/
> charm build cluster/juju/layers/kubernetes-worker/
build: Composing into /home/jackal/workspace/charms
build: Destination charm directory: /home/jackal/workspace/charms/builds/kubernetes-worker
build: Processing layer: layer:basic
build: Processing layer: layer:debug
build: Processing layer: layer:snap
build: Processing layer: layer:nagios
build: Processing layer: layer:docker (from ../../../workspace/charms/layers/layer-docker)
build: Processing layer: layer:metrics
build: Processing layer: layer:tls-client
build: Processing layer: layer:nvidia-cuda (from ../../../workspace/charms/layers/nvidia-cuda)
build: Processing layer: kubernetes-worker (from cluster/juju/layers/kubernetes-worker)
build: Processing interface: nrpe-external-master
build: Processing interface: dockerhost
build: Processing interface: sdn-plugin
build: Processing interface: tls-certificates
build: Processing interface: http
build: Processing interface: kubernetes-cni
build: Processing interface: kube-dns
build: Processing interface: kube-control
proof: OK!

During charm build all layers and interfaces referenced recursively starting by the top charm layer are fetched and merged to form your charm. The layers needed for building a charm are specified in a layer.yaml file on the root of the charm’s directory. For example, looking at cluster/juju/layers/kubernetes-worker/layer.yaml we see that the kubernetes worker charm uses the following layers and interfaces:

- 'layer:basic'
- 'layer:debug'
- 'layer:snap'
- 'layer:docker'
- 'layer:metrics'
- 'layer:nagios'
- 'layer:tls-client'
- 'layer:nvidia-cuda'
- 'interface:http'
- 'interface:kubernetes-cni'
- 'interface:kube-dns'
- 'interface:kube-control'

Layers is an awesome way to share operational logic among charms. For instance, the maintainers of the nagios layer have a better understanding of the operational needs of nagios but that does not mean that the authors of the kubernetes charms cannot use it.

charm build will recursively lookup each layer and interface at http://interfaces.juju.solutions/ to figure out where the source is. Each repository is fetched locally and squashed with all the other layers to form a single package, the charm. Go ahead a do a charm build with “-l debug” to see how and when a layer is fetched. It is important to know that if you already have a local copy of a layer under $JUJU_REPOSITORY/layers or interface under $JUJU_REPOSITORY/interfaces charm build will use those local forks instead of fetching them from the registered repositories. This enables charm authors to work on cross layer patches. Note that you might need to rename the directory of your local copy to match exactly the name of the layer or interface.

Building Resources

Charms will install Kubernetes but to do so they need to have the Kubernetes binaries. We package these binaries in snaps so that they are self-contained and deployed in any Linux distribution. Building such binaries is pretty straight forward as long as you know where to find them :)

Here is the repository holding the Kubernetes snaps: https://github.com/juju-solutions/release.git. The branch we want is rye/snaps:

> git clone https://github.com/juju-solutions/release.git
Cloning into ‘release’…
remote: Counting objects: 1602, done.
remote: Total 1602 (delta 0), reused 0 (delta 0), pack-reused 1602
Receiving objects: 100% (1602/1602), 384.69 KiB | 236.00 KiB/s, done.
Resolving deltas: 100% (908/908), done.
Checking connectivity… done.
> cd release
> git checkout rye/snaps
Branch rye/snaps set up to track remote branch rye/snaps from origin.
Switched to a new branch ‘rye/snaps’

Have a look at the README.md inside the snap directory to see how to build the snaps:

> cd snap/
> ./docker-build.sh KUBE_VERSION=v1.7.4

A number of .snap files should be available after the build.

In similar fashion you can build the snap package holding Kubernetes addons. We refer to this package as cdk-addons and it can be found at: https://github.com/juju-solutions/cdk-addons.git

> git clone https://github.com/juju-solutions/cdk-addons.git
Cloning into ‘cdk-addons’…
remote: Counting objects: 408, done.
remote: Total 408 (delta 0), reused 0 (delta 0), pack-reused 408
Receiving objects: 100% (408/408), 51.16 KiB | 0 bytes/s, done.
Resolving deltas: 100% (210/210), done.
Checking connectivity… done.
> cd cdk-addons/
> make

The last resource you will need (which is not packaged as a snap) is the container network interface (cni). Lets grab the repository and get to a release tag:

> git clone https://github.com/containernetworking/cni.git
Cloning into ‘cni’…
remote: Counting objects: 4048, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 4048 (delta 0), reused 2 (delta 0), pack-reused 4043
Receiving objects: 100% (4048/4048), 1.76 MiB | 613.00 KiB/s, done.
Resolving deltas: 100% (1978/1978), done.
Checking connectivity… done.
> cd cni
> git checkout -f v0.5.1

Build and package the cni resource:

> docker run — rm -e “GOOS=linux” -e “GOARCH=amd64” -v `pwd`:/cni golang /bin/bash -c “cd /cni && ./build”
Building API
Building reference CLI
Building plugins
flannel
tuning
bridge
ipvlan
loopback
macvlan
ptp
dhcp
host-local
noop
> cd ./bin
> tar -cvzf ../cni.tgz *
bridge
cnitool
dhcp
flannel
host-local
ipvlan
loopback
macvlan
noop
ptp
tuning

You should now have a cni.tgz in the root folder of the cni repository.

Two things to note here:
- We do have a CI for building, testing and releasing charms and bundles. In case you want to follow each step of the build process, you can find our CI scripts here: https://github.com/juju-solutions/kubernetes-jenkins
- You do not need to build all resources yourself. You can grab the resources used in CDK from the Juju store. Starting from the canonical-kubernetes bundle you can navigate to any charm shipped. Select one from the very end of the bundle page and then look for the “resources” sidebar on the right. Download any of them, rename it properly and you are ready to use it in your release.

Releasing Your Charms

After patching the charms to match your needs, please, consider submitting a pull request to tell us what you have been up to. Contrary to many other projects you do not need to wait for your PR to get accepted before you can actually make your work public. You can immediately release your work under your own namespace on the store. This is described in detail on the official charm authors documentation. The developers team is often using private namespaces to test PoCs and new features. The main namespace where CDK is released from is “containers”.

Yet, there is one feature you need to be aware of when attaching snaps to your charms. Snaps have their own release cycle and repositories. If you want to use the officially released snaps instead of attaching them to the charms, you can use a dummy zero sized file with the correct extension (.snap) in the place of each snap resource. The snap layer will see that that resource is empty and will try to grab the snap from the official repositories. Using the official snaps is recommended, however, in network restricted environments you might need to attach your own snaps while you deploy the charms.

Why is it this way?

Building CDK is of average difficulty as long as you know how where to look. It is not perfect by any standards and it will probably continue this path. The reason is that there are opposing forces shaping the build processes. This should come as no surprise. As Kubernetes changes rapidly and constantly expands, the build and release process should be flexible enough to include any new artefacts. Consider for example the switch from flannel cni to calico. In our case it is a resource and a charm that need to be updated. A monolithic build script would have been more “elegant” to the outsiders (eg, make CDK), but we would have been hiding a lot under the carpet. CI should be part of the culture of any team and should be owned by the team or else you get disconnected from the end product causing delays and friction. Our build and release process might look a bit “dirty” with a lot of moving parts but it really is not that bad! I managed to highlight the build and release steps in a single blog. Positive feedback also comes from our field engineers. Most of the time CDK deploys out of the box. When our field engineers are called it is either because our customers have a special requirement from the software or they have a “unconventional” environment in which Kubernetes needs to be deployed. Having such a simple and flexible build and release process enables our people to solve a problem on-site and release it tothe Juju store within a couple of hours.

Next steps

This blog post serves as foundation work for what is coming up. The plan is to go over some easy patches so we further demystify how CDK works. Funny thing this post was originally titled “Saying goodbye to my job security”. Cheers!

Read more
K.Tsakalozos

Sometimes we miss the forest for the trees.

It’s all about portability

Take a step back and think about how much of your effort goes into your product’s portability. A lot, I’d wager. You don’t wait to see what hardware your customer is on before you start coding your app. Widely accepted architectures give you a large customer base, so you target those. You skip optimisations in order to maximize compatibility. The language you are using ensures you can move to another platform/architecture.

Packaging your application is also important if you want to broaden your audience. The latest trend here is to use containers such as docker. Essentially containers (much like VMs) are another layer of abstraction for the sake of portability. You can package your app such that it will run anywhere.

https://medium.com/media/e87190de95f6f33f07f75beb4c8e7f39/href

Things go wrong

When you first create a PoC, even a USB stick is enough to distribute it! After that you should take things more seriously. I see products “dockerised” so that they run everywhere, yet the deployment story is limited to a single cloud. You do all the work to deliver your app everywhere only to fail when you actually need to deliver it! It doesn’t make sense — or to put it more accurately it only makes sense in the short term.

The overlooked feature

Canonical Distribution of Kubernetes (CDK) will deploy exactly the same Kubernetes in all major clouds (AWS, GCE, Azure, VMware, Openstack, Joynet), bare metal, or even your local machine (using lxd). When you first evaluate CDK you may think: “I do not need this”, or even “I hate these extra two lines where I decide the cloud provider because I already know I am going full AWS”. At that point you are tempted to take the easy way out and get pinned to a single Kubernetes provider. However, soon you will find yourself in need of feature F from cloud C or Kubernetes version vX.Y.Z. Worse is when your potential customer is on a different platform and you discover the extent of dependencies you have on your provider.

Before you click off this tab thinking I am exaggerating, think about this: the best devops are done by the devs themselves. How easy is it to train all devs to use AWS for spawning containers? It is trivial, right? Suppose you get two more customers, one on Azure and one on GCE. How easy is it to train your team to perform devops against two additional clouds? You now have to deal with three different monitoring and logging mechanisms. That might still be doable since AWS, GCE, and Azure are all well behaved. What if a customer wants their own on-prem cluster? You are done — kiss your portability and velocity goodbye. Your team will be constantly firefighting, chasing after misaligned versions, and emulating features across platforms.

How is it done

Under the hood of CDK you will find Juju. Juju abstracts the concepts of IaaS. It will provision everything from bare metal machines in your rack, to VMs on any major cloud, to lxd containers on your laptop. On top of these units Juju will deploy Kubernetes thus keeping the deployment agnostic to the underlying provider. Two points here:

  1. Juju has been here for years and is used both internally and externally by Canonical to successfully deliver solutions such as Openstack and Big Data.
  2. Since your Kubernetes is built on IaaS cloud offerings you are able to tailor node properties to match your needs. You may also find it is more cost effective.

Also check out Conjure-up, an easy-to-use wizard that will get your Kubernetes cluster up-and-running in minutes.

To conclude

I am sure you have your reasons to “dockerize” your application and then deliver it on a single cloud. It is just counter-intuitive. If you want to make the most from your portability decisions you should consider CDK.

References:

Read more
Dustin Kirkland

Earlier this month, I spoke at ContainerDays, part of the excellent DevOpsDays series of conferences -- this one in lovely Portland, Oregon.

I gave a live demo of Kubernetes running directly on bare metal.  I was running it on an 11-node Ubuntu Orange Box -- but I used the exact same tools Canonical's world class consulting team uses to deploy Kubernetes onto racks of physical machines.
You see, the ability to run Kubernetes on bare metal, behind your firewall is essential to the yin-yang duality of Cloud Native computing.  Sometimes, what you need is actually a Native Cloud.
Deploying Kubernetes into virtual machines in the cloud is rather easy, straightforward, with dozens of tools now that can handle that.

But there's only one tool today, that can deploy the exact same Kubernetes to AWS, Azure, GCE, as well as VMware, OpenStack, and bare metal machines.  That tools is conjure-up, which acts as a command line front end to several essential Ubuntu tools: MAAS, LXD, and Juju.

I don't know if the presentation was recorded, but I'm happy to share with you my slides for download, and embedded here below.  There are a few screenshots within that help convey the demo.




Cheers,
Dustin

Read more
Dustin Kirkland

Thank you to Oracle Cloud for inviting me to speak at this month's CloudAustin Meetup hosted by Rackspace.

I very much enjoyed deploying Canonical Kubernetes on Ubuntu in the Oracle Cloud, and then exploring Kubernetes a bit, how it works, the architecture, and a simple workload within.  I'm happy to share my slides below, and you can download a PDF here:


If you're interested in learning more, check out:
It was a great audience, with plenty of good questions, pizza, and networking!

I'm pleased to share my slide deck here.

Cheers,
Dustin

Read more
K.Tsakalozos

Introduction

In this post we discus our efforts to setup a Federation of on-prem Kubernetes Clusters using CoreDNS. The Kubernetes Cluster version used is 1.6.2. We use Juju to deliver clusters on AWS, yet the clusters should be considered on-prem since they do not integrate with any of the cloud’s features. The steps described here are repeatable on any pool of resources you may have available (cloud or bare metal). Note that this is still a work in progress and should not be used on a production environment.

What is Federation

Cluster Federation (Fig. from http://blog.kubernetes.io/2016/07/cross-cluster-services.html)

Cluster federation allows you to “connect” your Kubernetes clusters and manage certain functionality from a central point. The need for central management arises when you have more than one clusters possibly spread across different locations. You may need a globally distributed infrastructure for performance, availability or even legal reasons. You might want to have isolated clusters satisfying the needs of different teams. Whatever the case may be, federation can assist in some of the ops tasks. You can find more on the benefits of federation in the official documentation.

Why Cluster Federation with Juju

Juju makes delivery of Kubernetes clusters dead simple! Juju builds an abstraction over the pool of resources you have and allows us to deliver Kubernetes to all the main public and private clouds (AWS, Azure, GCE, Openstack, Oracle etc) and bare metal physical clusters. Juju also enables you to experiment with small scale deployments stood up on your local machine using lxc containers.

Having the option to deploy a cluster with the ease Juju offers, federation comes as a natural next step. Deployed clusters should be considered on-prem clusters as they are exactly the same as if they were deployed on bare metal. This is simply because even if you deploy on a cloud Juju will provision machines (virtual machines in this case) and deploy Kubernetes as if it were a local on-prem deployment. You see, Juju is not magic after all!

There is a excellent blog post showing how you can place Juju clusters under a federation hosting cluster setup on GCE. Here we focus on a federation that has no dependencies on a specific cloud provider. To this end, we go with CoreDNS as a federation DNS provider.

Let’s Federate

Deploy clusters

Our setup is made of 3 clusters:

  • f1: will host the federation controller place
  • c1, c2: federated clusters

At the time of this writing (2nd of June 2017) cluster federation is in beta and is not yet integrated into the official Kubernetes charms. Therefore we will be using a patched version of charms and bundles (source of charms and cdk-addons, charms master and worker, and bundle).

We assume you already have Juju installed and a controller in place. If not have a look at the Juju installation instructions.

Let’s deploy these three clusters.

#> juju add-model f1
#> juju deploy cs:~kos.tsakalozos/bundle/federated-kubernetes-core-2
#> juju add-model c1
#> juju deploy cs:~kos.tsakalozos/bundle/federated-kubernetes-core-2
#> juju add-model c2
#> juju deploy cs:~kos.tsakalozos/bundle/federated-kubernetes-core-2

Here is what the bundle.yaml looks like: http://paste.ubuntu.com/24746868/

Let’s copy and merge the kubeconfig files.

#> juju switch f1
#> juju scp kubernetes-master/0:config ~/.kube/config
#> juju switch c1
#> juju scp kubernetes-master/0:config ~/.kube/config.c1
#> juju switch c2
#> juju scp kubernetes-master/0:config ~/.kube/config.c2

Using this script you can merge the configs

#> ./merge-configs.py config.c1 c1
#> ./merge-configs.py config.c2 c2

To enable CoreDNS on f1 you just need to:

#> juju switch f1
#> juju config kubernetes-master enable-coredns=True

Have a look at your three contexts:

#> kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
f1 f1 ubuntu
c1 c1-cluster c1-user
c2 c2-cluster c2-user

Create the federation

At this point you are ready to deploy the federation control plane on f1. To do so you will need a call to kubefed init. Kubefed init will deploy the required services and it will also perform a health check ping to see everything is ready. During this last check kubefed init will try to reach a service on a port that is not open. At the time of this writing we have a patch waiting upstream to make the port configurable so you can open it before triggering the federation creation. If you do not want to wait for the patch to be merged you can call kubefed init with -v 9 to see what port the services were appointed to and expose them though Juju while keeping kubefed running. This will become clear shortly.

Create a coredns-provider.conf file with the etc endpoint of etcd of f1 and your zone. You can look at the Juju status to find out the IP of the etcd machine:

#> cat ./coredns-provider.conf 
[Global]
etcd-endpoints = http://54.211.106.117:2379
zones = juju-fed.com.

Call kubefed

#> juju switch f1
#> kubefed init federation — host-cluster-context=f1 — dns-provider=”coredns” — dns-zone-name=”juju-fed.com.” — dns-provider-config=coredns-provider.conf — etcd-persistent-storage=false — api-server-service-type=”NodePort” -v 9 — api-server-advertise-address=34.224.101.147 — image=gcr.io/google_containers/hyperkube-amd64:v1.6.2

Note here that the api-server-advertise-address=34.224.101.147 is the IP of the worker on your f1 cluster. You selected NodePort so the federation service is available from all the worker nodes under a random port.

Kubefed will end up trying to do a health check. You should see on your console something similar to this: “GET https://34.224.101.147:30229/healthz” . The port 30229 is randomly selected and will differ in your case. While keeping kubefed init running, go ahead and open this port from a second terminal:

#> juju run — application kubernetes-worker “open-port 30229”

The federation context should be present in the list of contexts you have.

#> kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
f1 f1 ubuntu
federation federation federation
c1 c1-cluster c1-user
c2 c2-cluster c2-user

Now you are ready to have other clusters join the federation:

#> kubectl config use-context federation
#> kubefed join c1 — host-cluster-context=f1
#> kubefed join c2 — host-cluster-context=f1

Two clusters should soon appear ready:

#> kubectl get clusters
NAME STATUS AGE
c1 Ready 3m
c2 Ready 1m

Congratulations! You are all set, your federation is ready to use.

What works

Setup your default namespace:

#> kubectl — context=federation create ns default

Get this replica set and create it.

#> kubectl — context=federation create -f nginx-rs.yaml

See your replica set created:

#> kubectl — context=federation describe rs/nginx
Name: nginx
Namespace: default
Selector: app=nginx
Labels: app=nginx
type=demo
Annotations: <none>
Replicas: 1 current / 1 desired
Pods Status: error in fetching pods: the server could not find the requested resource
Pod Template:
Labels: app=nginx
Containers:
frontend:
Image: nginx
Port: 80/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -
26s 26s 1 federated-replicaset-controller Normal CreateInCluster Creating replicaset in cluster c2

Create a service.

#> kubectl — context=federation create -f beacon.yaml

Scale and open ports so the service is available

#> kubectl — context=federation scale rs/nginx — replicas 3

What does not work yet

Federation does not seem to handle NodePort services. To see that you can deploy a busybox and try to resolve the name of the service you deployed right above.

Use this yaml.

#> kubectl — context=c2 create -f bbox-4-test.yaml
#> kubectl — context=c2 exec -it busybox — nslookup beacon

This returns a local cluster IP which is rather unexpected.

There are some configuration options that did not have the effect I was expecting. First, we would want to expose the CoreDNS service as described in the docs. Assigning the same NodePort for both UDP and TCP had no effect; this is rather unfortunate since the issue is already addressed. The configuration described here and here (although it should be automatically applied I thought I would give it a try) did not seem to have any effect.

Open issues:

Conclusions

Federation on on-prem clusters has gone a long way and is getting significant improvements with every release. Some of the issues described here have already been addressed, I am waiting anxiously for the upcoming v1.7.0 release to see the fixes in action.

Do not hesitate to reach out to the sig-federation group, but most importantly I would be extremely happy and grateful if you pointed out some bug I have on my configs/code/thoughts ;)

References and Links

  1. Official federations docs, https://kubernetes.io/docs/concepts/cluster-administration/federation/
  2. Juju docs, https://jujucharms.com/
  3. Canonical Distribution of Kubernetes, https://jujucharms.com/canonical-kubernetes/
  4. Federation with host on GCE, https://medium.com/google-cloud/experimenting-with-cross-cloud-kubernetes-cluster-federation-dfa99f913d54
  5. CDK-addons source, https://github.com/juju-solutions/cdk-addons/tree/feature/coredns
  6. Kubernetes charms, https://github.com/juju-solutions/kubernetes/tree/feature/coredns
  7. Kubernetes master charm, https://jujucharms.com/u/kos.tsakalozos/kubernetes-master/31
  8. Kubernetes worker charm, https://jujucharms.com/u/kos.tsakalozos/kubernetes-worker/17
  9. Kubernetes bundle used for federation, https://jujucharms.com/u/kos.tsakalozos/federated-kubernetes-core/bundle/0
  10. Juju getting started, https://jujucharms.com/docs/stable/getting-started
  11. Patch kubefed init, https://github.com/kubernetes/kubernetes/pull/46283
  12. CoreDNS setup instructions https://kubernetes.io/docs/tasks/federation/set-up-coredns-provider-federation/#setup-coredns-server-in-nameserver-resolvconf-chain
  13. CoreDNS yaml file https://github.com/juju-solutions/cdk-addons/blob/feature/coredns/coredns/coredns-rbac.yaml#L95
  14. Nodeport UDP/TCP issue, https://github.com/kubernetes/kubernetes/issues/20092
  15. Federation DNS on-prem configuration https://kubernetes.io/docs/admin/federation/#kubernetes-15-passing-federations-flag-via-config-map-to-kube-dns

Read more
Dustin Kirkland



Yesterday, I delivered a talk to a lively audience at ContainerWorld in Santa Clara, California.

If I measured "the most interesting slides" by counting "the number of people who took a picture of the slide", then by far "the most interesting slides" are slides 8-11, which pose an answer the question:
"Should I run my PaaS on top of my IaaS, or my IaaS on top of my PaaS"?
In the Ubuntu world, that answer is super easy -- however you like!  At Canonical, we're happy to support:
  1. Kubernetes running on top of Ubuntu OpenStack
  2. OpenStack running on top of Canonical Kubernetes
  3. Kubernetes running along side OpenStack
In all cases, the underlying substrate is perfectly consistent:
  • you've got 1 to N physical or virtual machines
  • which are dynamically provisioned by MAAS or your cloud provider
  • running stable, minimal, secure Ubuntu server image
  • carved up into fast, efficient, independently addressable LXD machine containers
With that as your base, we'll easily to conjure-up a Kubernetes, an OpenStack, or both.  And once you have a Kubernetes or OpenStack, we'll gladly conjure-up one inside the other.


As always, I'm happy to share my slides with you here.  You're welcome to download the PDF, or flip through the embedded slides below.



Cheers,
Dustin

Read more