Container Registries, Minikube and Authorization

Categories: Programming, Cloud

Introduction

This article came about because I started Minikube on my (Linux) laptop, then wanted to use the Maven JIB plugin to build a Java application into a container image and push it to an image registry accessible to Minikube.

That didn’t work well initially; I just got:

Build image failed, perhaps you should use a registry that supports HTTPS or set the configuration parameter 'allowInsecureRegistries'

Searching on the internet for that error-message led to a whole bunch of pages that offered wildly different advice, and no context - ie no description of why they were suggesting any specific solution.

I did eventually solve my problem. This article is an attempt to describe not only my solution, but also the reason why it worked for me. Even if you have a slightly different problem, the context presented here might help lead to a solution for your specific issue.

I have possibly included too much background info in this article, including things that most Docker/Minikube users will already know. However it seems easier to skip over info than guess what was omitted..

The article assumes that Docker is used as the container engine within Minikube, and the set of client tools that you are using. There are of course other container engines, but this is the most common setup - and hopefully the background info will be at least partially applicable to other setups.

The official Docker docs are of course the best resource for understanding Docker; I certainly haven’t tried to duplicate that level of detail.

Containers vs Images

A container image is a simple file consisting of:

  • a reference to a parent image (optional)
  • a set of files
  • some metadata

The image has a core identifier which is the hash-value of the image. There can also be multiple “tags” which point to the image.

An image can be stored in an “image registry”. It can also be stored in a container-engine’s image cache.

A container is an “instantiation” of an image. It has:

  • a filesystem, which initially is defined by its base image but can be modified by write-operations the container makes as it runs.
  • a set of port-mappings
  • a concrete ENTRY-POINT and CMD (which may vary from the default values provided by the base image)

The container may be running or not. When not running, its filesystem image still exists; that is only removed when the container is “deleted”.

Dockerd, Docker commandline, etc

In a typical Linux-based development environment with Docker installed, you have:

  • a bunch of Docker commandline tools in the default path, primarily the “docker” CLI application
  • an instance of Docker server process dockerd (aka docker daemon) running as root and listening on a filesystem-socket at /var/run/docker.sock
  • and your user is a member of group “docker” (to allow Docker commandline tools to access dockerd)

When a user executes a command like “docker pull”, and $DOCKER_HOST is not set then the Docker client tool just opens filesystem socket /var/run/docker.sock and sends a request to perform the required action. Similarly, “docker ps” just sends a request to the central server and receives back a list of all containers that exist. The filesystem socket is only accessible to members of group docker; dockerd does no other validation of the caller.

When $DOCKER_HOST is set, then the client opens a TCP socket to the specified address; the dockerd instance that is listening on that address should be configured to perform mutual TLS authentication - ie the daemon provides a signed TLS certificate, and expects the client connecting to it to do the same. The client “identity” is then extracted from the provided client certificate; optionally an “authentication plugin” can be configured in the server to allow/deny specific operations depending on the client identity. A dockerd instance which opens a network port, but does not validate clients, is effectively “world open”.

In a simple setup, the client application and the dockerd process are running in the same operating system as different users.

Docker Run

When a user starts a new container (eg via “docker run ..”), the client tool run by the user just sends that request to the central service. It is the central daemon that:

  • communicates with the Linux kernel to create new PID, filesystem, network, and other namespaces
  • creates a filesystem that consists of all the “layers” that the container’s base image reference and mounts that into the newly created filesystem namespace
  • creates a new CGROUP which is a child of the CGROUP in which dockerd is running
  • starts the process specified by the containers ENTRYPOINT and CMD properties (where paths in these values are relative to the container’s filesystem namespace), assigning that process to the created CGROUP

Similarly, when a user executes “docker build ..” to process a dockerfile, the client tool opens a socket to dockerd and sends all necessary data to it. It is the server itself which executes the build, and then saves the resulting image to its local cache.

Container Filesystem Layers

A container image has a parent and a set of files (as a .tar.gzip archive). The complete filesystem that a container sees when first run is the union of all files provided by all ancestor images.

It is the container runtime (eg dockerd) that is responsible for making this complete filesystem available to the code running in the container.

The simplest solution for creating a complete filesystem from a chain of images is just to allocate a directory on the local filesystem then, starting from the oldest ancestor image, unpack the files from each image into that directory. However this is be relatively time-consuming, generates lots of disk-io on container launch, and uses unnecessary disk capacity when multiple running containers use the same image, or at least share ancestor images.

Linux has provided several “union file systems” over the years, including AUFS (deprecated) and Overlayfs. The BTRFS and ZFS filesystems can also perform “union” operations natively. All of these solutions are kernel-level code, ie the container engine can use them only if the host operating system happens to have that feature installed. With this approach, each image’s files is unpacked just once on each host, into a cache. When a container needs to be started, a union-filesystem is created which references the unpacked copy of each ancestor image without copying it. The filesystem also points to an empty directory which forms the “writable upper layer”; anything that the running container writes to disk is placed there instead of modifying the underlying layer.

The Docker container engine supports all of the filesystems listed above (including the “just copy everything into one directory” approach which it calls “vfs storage”), and chooses one at runtime based on what is available in the underlying kernel (unless explicitly configured).

Container Image Registries

A container image registry is a process that provides a network API over which images can be uploaded and downloaded. The dockerd server caches images, and uses them to launch new containers, but is not itself a registry and does not offer this network API.

Dockerhub offers an image that provides a container image registry; see below.

By convention, an image registry offers its API on port 5000.

Image Names

The “one true id” of an image is its hash-value. However it us usually referred to by a name which is a string of form “{registry/}{repository}{:tag}” - ie a name is an alias for a specific image.

The {registry} is optional; if present, it is expected to be of form “{hostname}{:port}?”. If not present then it defaults to the host:port of dockerhub’s image registry server. Note that either the hostname needs to include a dot, or :port needs to be present; if neither of these is used then the string is assumed to be part of the “repository”, and the default registry is used.

The {repository} part is mandatory and is a string of form “{group/}*{name}”; this identifies the “logical artifact”. Characters ‘.’ and ‘:’ are not allowed in group or name fields.

The {tag} part is optional; if present it is of form “:tagname”. If not present then it defaults to “:latest”.

When “pulling” an image from a remote registry with “docker pull {name}”, dockerd checks its local cache to see whether an image-file with the specified name is already available (unless “pull=always” is enabled). If not, then the name includes the network address of the registry from which it should be downloaded.

The docker push .. command uses a slightly unusual approach (at least for those used to maven/gradle/etc). Rather than specify “what to push” and “where to push it” separately, you instead ensure that the images you want to push have “names” that indicate what registry they should be stored in - and then use docker push to upload all matching images to “the registry they belong to”.

There are several very-well-known image registries which provide a large selection of public images (ie clients do not need authentication) as well as images for which authentication is required:

  • dockerhub (docker’s image registry) - the primary location for images released by open-source projects
  • gcr.io (google’s image registry) - hosts many useful containers for projects led by Google

The Amazon ECR and Microsoft Azure image registries mostly provide images that are useful only on those specific platforms.

Running an Image Registry under Docker

If you have a Docker environment, then starting an image registry is trivial:

docker run -d -p 5000:5000 --name registry registry:2

This downloads the image “registry:2” (version 2.x of image registry) and starts it as a container named “registry”. It also maps port localhost:5000 to port 5000 of the started container, ie the service is available from the host at two IP addresses:

  • $(container-ip-address):5000
  • localhost:5000

The registry service is set up to perform no validation of users - ie any application that can reach either of these ports can read/write images. However given that the container’s IP address is on a “bridge network” that is only accessible from the host on which the container is running, and the “localhost” address is also only accessible from the same host, that risk is relatively low.

The registry service provides its services over HTTPS, but by default uses a self-signed server TLS certificate. Any client that tries to verify this certificate will therefore not be successful; solutions for this include:

  • dockerd does not bother to validate the server’s TLS certificate for addresses of form “localhost:someport”
  • dockerd can be told to ignore TLS certificates (or even allow http instead of https) for specific registries via config item insecure-registries
  • various non-docker client tools (eg JIB) can be told to ignore TLS certificates for all registries via an option called “allowInsecureRegistries=true” or similar
  • or you can ensure that the dockerd instance that is communicating with the remote registry has that registry’s self-signed certificate in their trust-domain
  • or you can install a properly-signed certificate for your newly-set-up registry

For dockerd, in file /etc/docker/daemon.json configuration item insecure-registries=[..] tells it which remote registries are allowed to be insecure - ie where TLS certificates should not be validated. It also even allows the remote registry to be running HTTP rather than HTTPS. Obviously, enabling this for arbitrary addresses is dangerous but it is sometimes necessary for internal registries.

Remember that when running “docker pull ..” and similar, it is actually some dockerd instance that really communicates with the specified remote image registry..

Minikube and Image Registries

Minikube sets up a VM running Linux and Docker. Command minikube start creates the VM, starts Docker, and launches a bunch of processes and containers that provide a full (but single-node) kubernetes environment.

Command “minikube docker-env” outputs the environment-vars needed to get local Docker commandline tools to talk to the dockerd instance within the VM, rather than the local one (if any). Thus, after “eval $(minikube docker-env)” a call to “docker ps” will show all containers running within the minikube VM rather than containers running on the local host.

The “minikube start” command also creates (or updates) file ~/.kube/config which contains the address of the VM minikube created. If you run the standard kubectl tools provided by kubernetes, they will therefore by default talk to the kube-apiserver instance running within the minikube VM.

Minikube does not start an image registries process by default. However starting one is such a common requirement that Minikube offers an “addon” for that purpose:

minikube addon enable registry

This starts a pod in namespace “kube-system”, and registers a service-resource named “registry” in the same namespace. You can see the service with:

kubectl --namespace kube-system describe service registry

and the pods on which it is running with:

kubectl --namespace kube-system get pods --selector kubernetes.io/minikube-addons=registry

The registry service resource definition ensures that port 5000 on the VM’s network interface is forwarded to port 80 on each pod that is running the registry image (only one).

The pod is running exactly the same “registry:2” image described in the above section on running an image registry directly under Docker. It also does no validation of clients, ie allows any application that can reach its network port to read/write any image. However given that the Minikube VM is running on a “bridge network” with an IP address that is only accessible from the host on which the minikube VM is running, that risk is relatively low.

As with image-registry-on-docker, the minikube registry service supports HTTPS but uses a self-signed TLS certificate; any client that tries to verify this certificate will therefore not be successful. The solutions that are documented above for image-registry-on-docker also apply:

  • dockerd does not bother to validate the server’s TLS certificate for addresses of form “localhost:someport”
  • dockerd can be told to ignore TLS certificates (or even allow http instead of https) for specific registries via config-item insecure-registries - so add your minikube registry’s address to this list
  • various non-docker client tools (eg JIB) can be told to ignore TLS certificates for all registries via an option called “allowInsecureRegistries=true” or similar
  • or you can ensure that the dockerd instance that is communicating with the Minikube-hosted registry has that registry’s certificate in their trust-domain
  • or you can install a properly-signed certificate for the Minikube-hosted registry service (ie add the cert to the filesystem for that pod)

If you need to have dockerd on your development system push images to the image registry within Minikube, then taking advantage of the localhost-exception is possibly the easiest workaround. The minikube-hosted image registry can be made to appear to be at a localhost address by setting up kubectl port-forwarding:

# Sadly, service "registry" does not yet define a named service-port, so the "raw" exported port-number must be used. 
kubectl port-forward --namespace kube-system service/registry 5000:80

As an alternative, plain SSH-port-forwarding could be used.

JIB and Image Registries

The JIB build plugin (for maven, gradle, etc) creates an image directly rather than talking to a Docker daemon. It then “uploads” the image to a specified registry using the normal network protocols.

Note: JIB can be configured to talk to use docker build instead of creating the image itself. This solves the problem of pushing to a remote registry when using minikube, as you just need to do eval $(minikube docker-env) and then the build is actually done by dockerd within minikube and then stored directly without needing to talk to a registry. An invalid TLS cert is therefore not a problem.

This leads to the problem mentioned at the start of this article: when using $(minikube ip):5000 as the registry to write to, the operation fails because that registry has a self-signed TLS certificate.

There are two easy solutions:

  • in the JIB plugin config, set allowInsecureRepositories=true
  • or use kubectl port-forward .. as described above to map localhost:someport to the registry-service inside minikube, then use “localhost:someport” as the target registry for JIB

The second solution relies on the fact that, like dockerd, JIB automatically enables “allowInsecureRepositories” for registries with address at localhost. Note that when running JIB in its usual mode (without docker-build) the connection is simply opened by your build-tool, and therefore the port-forwarding needs to be on the server on which the build runs (not on the one on which dockerd is running).

Summary

Wow - so many words for such simple solutions to the JIB build problem. I hope you found it worth-while to read. I’m sure I’ll find it useful to have these notes available to me if I face the same problem again in a few years!

Terminology

Just a brief mention of some confusing word-usage:

  • a container registry is a service that stores images
  • a repository is a set of related images (usually different versions of the same application)

In other words, a “repository” is a string of form “group/name”; repository+tag (ie group/name:tag) identifies a specific image. However sometimes the word “tag” is used as a synonym for the whole repository:tag string.

References