Kubernetes manages containerised applications. The container images are found either locally, or fetched from a remote registry. We recently released MicroK8s and noticed that some of our users were not comfortable with configuring containerd with image registries. In this blog we go through a few workflows most people are following.

What you should know already

We discuss how to consume local images, or images fetched from public and private registries in Kubernetes configured with containerd.

To get one such cluster simply:

sudo snap install microk8s --classic

Familiarity with building, pushing and tagging container images will be helpful. The examples that follow use Docker but you can use your preferred container tool chain.

To install Docker on Ubuntu 18.04:

sudo apt-get install docker.io

Add the user to the docker group:

sudo usermod -aG docker ${USER}

Open a new shell for the user, with updated group membership:

su - ${USER}

The Dockerfile we will be using is:

FROM nginx:alpine

To build the image tagged with mynginx:local, go to the directory where the Dockerfile is and run:

docker build . -t mynginx:local

Working with locally built images without a registry

When an image is built it is cached on the Docker daemon used during the build. Having run the docker build . -t mynginx:localcommand, you can see the newly built image by running:

docker images

This will list the images currently known to Docker, for example:

REPOSITORY          TAG                 IMAGE ID             SIZE
mynginx local 0be75340bd9b 16.1MB

The image we created is known to Docker. Kubernetes is not aware of the newly built image as your local Docker daemon is not part of the MicroK8s Kubernetes cluster. We can export the built image from the local Docker daemon and “inject” it into the MicroK8s image cache like this:

docker save mynginx > myimage.tar
microk8s.ctr -n k8s.io image import myimage.tar

Note that when we import the image to MicroK8s we do so under the k8s.io namespace (the -n k8s.io argument).

Now we can list the images present in MicroK8s:

microk8s.ctr -n k8s.io images ls

At this point we are ready to microk8s.kubectl apply -f a deployment with this image:

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: mynginx:local
ports:
- containerPort: 80

We reference the image with image: mynginx:local. Kubernetes will behave as if there is an image in docker.io (the Dockerhub registry) for which it already has a cached copy. Note here, that containerd will not cache images with the latest tag so make sure you do not use that.

Working with public registries

After building an image with docker build . -t mynginx:local, it can be pushed to one of the mainstream public registries. You will need to create an account and register a username with the registry provider. For this example, we created an account with https://hub.docker.com/ and we log in as kjackal.

First we run the login command:

docker login

Docker will ask for a Docker ID and password to complete the login.

Login with your Docker ID to push and pull images from Docker Hub. If you don’t have a Docker ID, head over to https://hub.docker.com to create one.
Username: kjackal
Password: *******

Pushing to the registry requires that the image is tagged with your-hub-username/image-name:tag. We can either add proper tagging during build:

docker build . -t kjackal/mynginx:public

Or tag an already existing image using the image ID. Obtain the ID by running:

docker images

The ID is listed in the output:

REPOSITORY          TAG                 IMAGE ID            SIZE
mynginx local 0be75340bd9b 16.1MB

Then use the tag command:

docker tag 0be75340bd9b kjackal/mynginx:public

Now that the image is tagged correctly, it can be pushed to the registry:

docker push kjackal/mynginx

At this point we are ready to microk8s.kubectl apply -f a deployment with our image:

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: kjackal/mynginx:public
ports:
- containerPort: 80

We refer to the image as image:kjackal/mynginx:public. Kubernetes will search for the image in its default registry, docker.io.

Working with MicroK8s’ registry add-on

Having a private Docker registry can significantly improve your productivity by reducing the time spent in uploading and downloading images. The registry shipped with MicroK8s is hosted within the Kubernetes cluster and is exposed as a NodePort service on port 32000 of the localhost. Note that this is an insecure registry and you may need to take extra steps to limit access to it.

You can install the registry with:

microk8s.enable registry

The add-on registry is backed up by a 20Gi persistent volume claimed for storing images. To satisfy this claim the storage add-on is also enabled along with the registry.

The containerd daemon used by MicroK8s is configured to trust this insecure registry. To upload images we have to tag them with localhost:32000/your-mage before pushing them:

We can either add proper tagging during build:

docker build . -t localhost:32000/mynginx:registry

Or tag an already existing image using the image ID. Obtain the ID by running:

docker images

The ID is listed in the output:

REPOSITORY                TAG             IMAGE ID       SIZE
localhost:32000/mynginx registry 0be75340bd9b 16.1MB

Then use the tag command:

docker tag 0be75340bd9b localhost:32000/mynginx:registry

Now that the image is tagged correctly, it can be pushed to the registry:

docker push localhost:32000/mynginx

Pushing to this insecure registry may fail in some versions of Docker unless the daemon is explicitly configured to trust it. To address this we need to edit /etc/docker/daemon.json and add:

{
"insecure-registries" : ["localhost:32000"]
}

The new configuration should be loaded with a Docker daemon restart:

sudo systemctl restart docker

At this point we are ready to microk8s.kubectl apply -f a deployment with our image:

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: localhost:32000/mynginx:registry
ports:
- containerPort: 80

What if MicroK8s runs inside a VM?
Often MicroK8s is placed in a VM while the development process takes place on the host machine. In this setup pushing container images to the in-VM registry requires some extra configuration.

Let’s assume the IP of the VM running MicroK8s is 10.141.241.175. When we are on the host the Docker registry is not on localhost:32000 but on 10.141.241.175:32000. As a result the first thing we need to do is to tag the image we are building on the host with the right registry endpoint:

docker build . -t 10.141.241.175:32000/mynginx:registry

If we immediately try to push the mynginx image we will fail because the local Docker does not trust the in-VM registry. Here is what happens if we try a push:

> docker push 10.141.241.175:32000/mynginx
The push refers to repository [10.141.241.175:32000/mynginx]
Get https://10.141.241.175:32000/v2/: http: server gave HTTP response to HTTPS client

We need to be explicit and configure the Docker daemon running on the host to trust the in-VM insecure registry. Add the registry endpoint in /etc/docker/daemon.json:

{
"insecure-registries" : ["10.141.241.175:32000"]
}

Then restart the docker daemon on the host to load the new configuration:

sudo systemctl restart docker

We can now docker push 10.141.241.175:32000/mynginx and see the image getting uploaded. During the push our Docker client instructs the in-host Docker daemon to upload the newly built image to the 10.141.241.175:32000 endpoint as marked by the tag on the image. The Docker daemon sees (on /etc/docker/daemon.jason) that it trusts the registry and proceeds with uploading the image.

Consuming the image from inside the VM involves no changes:

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: localhost:32000/mynginx:registry
ports:
- containerPort: 80

We reference the image with localhost:32000/mynginx:registry since the registry runs inside the VM so it is on localhost:32000.

Working with a private registry

Often organisations have their own private registry to assist collaboration and accelerate development. Kubernetes (and thus MicroK8s) need to be aware of the registry endpoints before being able to pull container images.

Insecure registry
Let’s assume the private insecure registry is at 10.141.241.175 on port 32000. The images we build need to be tagged with the registry endpoint:

docker build . -t 10.141.241.175:32000/mynginx:registry

Pushing the mynginx image at this point will fail because the local Docker does not trust the private insecure registry. The docker daemon used for building images should be configured to trust the private insecure registry. This is done by marking the registry endpoint in /etc/docker/daemon.json:

{
"insecure-registries" : ["10.141.241.175:32000"]
}

Restart the Docker daemon on the host to load the new configuration:

sudo systemctl restart docker

Now running

docker push 10.141.241.175:32000/mynginx

…should succeed in uploading the image to the registry.

Attempting to pull an image in MicroK8s at this point will result in an error like this:

Warning Failed 1s (x2 over 16s) kubelet, jackal-vgn-fz11m Failed to pull image “10.141.241.175:32000/mynginx:registry”: rpc error: code = Unknown desc = failed to resolve image “10.141.241.175:32000/mynginx:registry”: no available registry endpoint: failed to do request: Head https://10.141.241.175:32000/v2/mynginx/manifests/registry: http: server gave HTTP response to HTTPS client

We need to edit /var/snap/microk8s/current/args/containerd-template.toml and add the following under [plugins] -> [plugins.cri.registry] -> [plugins.cri.registry.mirrors]:

  [plugins.cri.registry.mirrors.”10.141.241.175:32000"]
endpoint = [“http://10.141.241.175:32000"]

See the full file here.

Restart MicroK8s to have the new configuration loaded:

microk8s.stop
microk8s.start

The image can now be deployed with:

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: 10.141.241.175:32000/mynginx:registry
ports:
- containerPort: 80

Note that the image is referenced with 10.141.241.175:32000/mynginx:registry.

Secure registry
There are a lot of ways to setup a private secure registry that may slightly change the way you interact with it. Instead of diving into the specifics of each setup we provide here two pointers on how you can approach the integration with Kubernetes.

  • The official Kubernetes documentation describe how to create a secret from the Docker login credentials and use it to access the secure registry. To achieve this, imagePullSecrets is used as part of the container spec.
  • MicroK8s v1.14 and onwards uses containerd. As described here configuring containerd involves editing /var/snap/microk8s/current/args/containerd-template.toml and reloading the new configuration via a microk8s.stop, microk8s.start cycle.

Further Reading


Working with image registries and containerd in Kubernetes was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Read more