Overview
When running a set of docker containers, with some services being “clustered” services, the hostnames exported by the clustered services can be “unresolvable” from the host point-of-view. Below is some simple code to fix this for Java-based client apps.
The Problem Details
Normally, when opening a socket the JRE deals with resolving the specified hostname to an IP address; it first looks in /etc/hosts
and then consults a DNS server. However I recently had a situation where a library was doing the calls to open a socket, and I knew the right host-to-ip mapping, but it wasn’t in the usual places.
To be specific, I am working on a Java application which uses Kafka and the Confluent Schema Registry. In order to conveniently develop locally, I start all the services the app needs via a docker-compose file (Zookeeper, Kafka, Schema Registry). In addition, I have developed integration-tests which use docker-compose to run all necessary dependencies; it’s a quite elegant way to ensure application code really does communicate with its runtime infrastructure correctly, and I hope to have time to write an article about it soon.
Docker-compose has some nice networking features. By default it creates a network entry within the host environment with name ${project}_default
where project is by default the name of the directory in which the docker-compose.yaml
file exists. Each container (“service”) defined in the docker-compose file is given an IP address on this network, and docker sets up dns-like domain-name resolution so that code within one container can address other containers using hostname=service-name. In addition, specific ports on specific services can be exported into the host network.
Unfortunately, this nice DNS-resolution works only for container-to-container communication; code running within the host environment can reach services within containers via ip-address:port
or via localhost:exported-port
but not servicename:port
as code within containers can.
Normally, this is no big deal; code in the host just uses the localhost address and the exported port. However it is a real problem for some clusterable applications running in containers, eg a Kafka message broker. A Kafka client first connects to any node in the bootstrapServers list, and asks it for the full list of all broker nodes. The first step is fine - for code in the host, the bootstrapServers list can contain localhost:port
while for code in a container the bootstrapServers list can be servicename:port
. However the returned list of addresses are the ADVERTISED_LISTENERS
addresses, ie addresses that Kafka thinks clients should address it at. If the advertised addresses are set to “localhost” then (a) multiple broker nodes are not possible, and (b) client code in other containers will not be able to see the broker nodes, as the docker “exported ports” are only exported to the host, not other containers. If the advertised addresses are set to the service-name, then (a) and (b) are solved - but code in the host environment cannot connect, because those service-names are not valid in the host environment; they are not in /etc/hosts
nor in a DNS server that the host looks at.
There are a couple of tools which effectively modify the host /etc/hosts
file when a docker container is created:
However this can lead to name conflicts, eg on CI servers when multiple builds are running concurrently. It is also an additional application that must be run. And as far as I can see it also works only on Linux machines - while I am on MacOS.
A Solution for Java Client Apps
I don’t need to solve this problem in general, eg allow access “by name” to containers from a commandline; I just need my Java Spring application which uses the Kafka client libraries to be able to handle the fact that the “node addresses” returned by the initial Kafka query are of form “some-docker-servicename:port”. In particular, in my case I have only one Kafka node, and its port is already exported into the host network, so I simply need to remap lookups of a specific servicename into “localhost”. Note that changing the ADVERTISED_LISTENER
setting for the Kafka node to localhost
breaks other containers that need to talk to Kafka, and in particular the Confluent Schema Registry service.
So, finally, here is a class that allows adding custom mappings into the standard JRE hostname resolution process:
// Copyright Simon Kitching 2017. Available under the Apache License 2.0
package net.vonos.network.util;
import sun.net.spi.nameservice.NameService;
import java.lang.reflect.Field;
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.*;
import java.util.stream.Collectors;
/**
* Test/Development utility which intercepts the JRE logic which maps domain-names to addresses, allowing programmer-provided mappings.
* <p>
* Mappings looked-up by the JRE before this class is initialised are already cached in the JDK, ie customised mappings made too late will have no effect.
* </p>
* <p>
* Example Usage:
* <pre>
* NetworkNameMapper.install(NetworkNameMapper.mappingsFromSystemProperties());
* </pre>
* </p>
*/
public class NetworkNameMapper {
// Disable constructor
private NetworkNameMapper() {
}
/**
* Build a mapping suitable for passing to method NetworkNameMapper.install.
* <p>
* System properties of form "dns.map.somename=alias" will cause lookups of "somename" to return the alias.
* The alias may be an IP-address or a name; names will be looked up using the mappings existing at the time this method is called.
* </p>
*/
public static Map<String, InetAddress> mappingsFromSystemProperties() {
try {
Map<String, InetAddress> mappings = new HashMap<>();
for (Map.Entry<?, ?> e : System.getProperties().entrySet()) {
String pname = e.getKey().toString();
if (pname.startsWith("dns.map.") && pname.length() > 8) {
String name = pname.substring(8);
String value = e.getValue().toString();
InetAddress addr = InetAddress.getByName(value);
mappings.put(name, addr);
}
}
return mappings;
} catch(UnknownHostException e) {
// This NetworkNameMapper class is intended primarily for development/testing purposes, so it seems better
// to not clutter calling code with exception-handling.
throw new IllegalArgumentException("Unable to install network mappings", e);
}
}
/**
* Implement the redirects specified in the mapping.
* <p>
* The specified mappings are copied, ie mutating them after this install method has been called has no effect.
* </p>
*/
public static void install(Map<String, InetAddress> mappings) {
if (mappings.isEmpty()) {
return;
}
Map<String, InetAddress[]> internalMappings = mappings.entrySet().stream().collect(
Collectors.toMap(e -> e.getKey(), e -> new InetAddress[]{e.getValue()}));
// Avoid race-conditions with other instances of this type. Unfortunately, there is no way to guarantee no
// race-conditions with other code that is also manipulating the internal InetAddress fields, but that is
// rather unlikely. Class INetAddress itself initializes field nameServices _very_ early, and never modifies it,
// so there are no races with INetAddress itself.
synchronized(NetworkNameMapper.class) {
try {
installInternal(internalMappings);
} catch(NoSuchFieldException|IllegalAccessException e) {
// Can happen only if a later JRE implementation changes the INetAddress class, or if a
// SecurityManager class forbids private-field-access.
throw new RuntimeException("Internal error", e);
}
}
}
/**
* Use reflection to add a custom NameService instance to the global list of NameServices used by INetAddress.
*/
private static void installInternal(Map<String, InetAddress[]> mappings) throws NoSuchFieldException, IllegalAccessException {
// Use reflection to get access to the list of NameService objects deep within the JRE implementation.
Class<InetAddress> ic = InetAddress.class;
Field f = ic.getDeclaredField("nameServices");
f.setAccessible(true);
Object currentList = f.get(null);
@SuppressWarnings("unchecked")
List<NameService> origNS = (List<NameService>) currentList;
// Create a new list of NameService objects containing our custom MyNS class followed by all existing NameService objects
NameService myNs = new MyNS(mappings);
List<NameService> newNS = new ArrayList<>(origNS.size() + 1);
newNS.add(myNs);
newNS.addAll(origNS);
// Replace the static field with our new class. Updating a reference is atomic, ie INetAddress will never be corrupted.
// Races with other instances of this type are prevented by method initIntenal. Races with other code that is also
// manipulating this same field is not prevented, but very unlikely.
f.set(null, newNS);
}
// ====================================================================================================
/** Simple implementation of the standard NameService interface. */
private static class MyNS implements NameService {
private final Map<String, InetAddress[]> mappings;
private MyNS(Map<String, InetAddress[]> mappings) {
this.mappings = mappings;
}
@Override
public InetAddress[] lookupAllHostAddr(String s) throws UnknownHostException {
InetAddress[] result = mappings.get(s);
if (result != null) {
return result;
}
// cause INetAddress.getAllByName to try the next NameService
throw new UnknownHostException();
}
// Reverse lookups not supported for now..
@Override
public String getHostByAddr(byte[] bytes) throws UnknownHostException {
// cause INetAddress.getAllByName to try the next NameService
throw new UnknownHostException();
}
}
}
In the application main-method, I then just call:
NetworkNameMapper.install(NetworkNameMapper.fromSystemProperties());
which allows me to start my Java app in development environments with arguments like “-Ddns.map.kafkaservice=localhost
”.
For integration-tests, I just add:
@BeforeClass
public static void initDNS() throws Exception {
NetworkNameMapper.install(Collections.singletonMap("kafkaservice", "localhost"));
}
Yes, this NetworkNameMapper class is using reflection to poke around in parts of the JRE. But it works!
Further Reading
- locally managed dns - a similar approach to mine, but somewhat more complicated.
- stackoverflow: local hostnames for docker containers - one of many similar questions on this topic
- github: add container hostnames - another similar question
- github: use ip addresses and DNS names for docker containers - yet another