OpenStack Tech

Network namespaces and containers

Openstack can be deployed in several ways. For example, for testing and development, devstack is used. For production environments, Kolla-Ansible provides a set of tools to define, create and deploy a set of containers running the needed services. TripleO is another deployment tool that can be configured to run the services on containers.

Neutron, the networking orchestrator project, makes use of the network namespaces to isolate some operations. For example, the DHCP agent creates a namespace per network; this namespace is connected to other namespace via a veth pair. Inside this DHCP namespace, a dnsmasq process is running to attend all DHCP requests from any registered device (the dnsmasq daemon has a list of authorized MAC addresses).

Example of DHCP namespace:

$ ip netns exec qdhcp-2365b6b7-8532-4db2-ab7a-e5a6f9000db1 ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host 
valid_lft forever preferred_lft forever
28: tap0c79968e-a5: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:e8:c7:1d brd ff:ff:ff:ff:ff:ff
inet brd scope global tap0c79968e-a5
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fee8:c71d/64 scope link 
valid_lft forever preferred_lft forever

What happens when the DHCP agent is running inside a container? Well, in this case the container will create and modify, in the host, the network namespace to hold the dnsmasq process. Here is where the game begins.

How a network namespace is mounted in the filesystem.

Just as a reference, this is the code iproute2 and pyroute2 implement to add a new namespace. pyroute2 implementation is a translation from iproute2 code.

First, the directory is created; for example, /run/netns/ns01. Then the mounting point, common for all namespaces in the system, is mounted with flags MS_SHARED and MS_REC. This is very important because it makes possible for network namespace mounts to propagate between mount namespaces. After that the mounting point is bound recursively. At this point, the resource can be found in the filesystem as a shared mounted point:

TARGET                                SOURCE                 FSTYPE     PROPAGATION
│ └─/run/netns                        tmpfs[/netns]          tmpfs      shared

Once the common mounting point is created (this will be done only once in the system), the namespace mounting point is created; in our case /run/netns/ns01. Then the network namespace itself is created, by creating new IPv4 and IPv6 stacks, using unshare.

The last process is to bind the network namespace to /proc/self/ns/net. The directory /proc/[pid]/ns is a subdirectory containing one entry for each namespace that supports being manipulated by setns.

A network namespace lifespan, created with the ip command is, at most, until the system is restarted. A network namespace does not persist across reboots. When a network namespace is created with the unshare or clone system calls, it is bound to the life of the current process, so if your process exits, the network namespace is removed. Only a privileged process can make their network namespace persistent when there are no processes in it.

When is deleted using the ip command, the namespace is removed when the last process running inside it stops.

Let’s create some network namespaces.

In a rebooted system there are no namespaces created. The common mounting point for the network namespaces, /run/netns, is still not created.

TARGET                                SOURCE                 FSTYPE     PROPAGATION
├─/run                                tmpfs                  tmpfs      shared

Once we create the first namespace, two new mounting points appear in the filesystem: the netns directory and the namespace itself.

TARGET                                SOURCE                 FSTYPE     PROPAGATION
├─/run                                tmpfs                  tmpfs      shared
│ ├─/run/netns                        tmpfs[/netns]          tmpfs      shared
│ │ └─/run/netns/ns01                 nsfs[net:[4026532276]] nsfs       shared
│ └─/run/netns/ns01                   nsfs[net:[4026532276]] nsfs       shared

If the namespace is delete, the netns directory is left to be used by other namespaces and it’s keep as “shared”. What can happen if the propagation status is not shared? Let’s play a game: let’s create a private copy of the mount namespace, not shared with any other process, using unshare:

$ unshare --mount
TARGET                                SOURCE                 FSTYPE     PROPAGATION
├─/run                                tmpfs                  tmpfs      private

If, from this unshared mount namespace, we create a new network namespace, we’ll have a weird situation: both mount namespaces will share it’s own network namespace, the directory will be present in the filesystem but the mounting point won’t be shared between them.

TARGET                               SOURCE                 FSTYPE     PROPAGATION
├─/run                               tmpfs                  tmpfs      private
│ └─/run/netns                       tmpfs[/netns]          tmpfs      shared
│   └─/run/netns/ns02                nsfs[net:[4026532335]] nsfs       shared

├─/run                               tmpfs                  tmpfs      shared
│ ├─/run/netns                       tmpfs[/netns]          tmpfs      shared
│ │ └─/run/netns/ns01                nsfs[net:[4026532277]] nsfs       shared
│ └─/run/netns/ns01                  nsfs[net:[4026532277]] nsfs       shared

That will lead to a situation where the second network namespace is inaccessible:

$ ll /run/netns/
total 0
drwxr-xr-x  2 root root ?   80 Mar  8 20:31 ./
drwxr-xr-x 36 root root ? 1200 Mar  8 20:21 ../
-r--r--r--  1 root root ?    0 Mar  8 20:30 ns01
----------  1 root root ?    0 Mar  8 20:31 ns02
root@osdev18:~# ip netns 
Error: Peer netns reference is invalid.
Error: Peer netns reference is invalid.
root@osdev18:~# ip netns exec ns01 ip a
1: lo:  mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
root@osdev18:~# ip netns exec ns02 ip a
setting the network namespace "ns02" failed: Invalid argument

And now from inside a container.

The goal is to be able to create a network namespace in the host. By default all files created inside a container are stored on a writeable container layer. Basically it means from the container filesystem there is no way to modify the host filesystem; that makes sense if one of the goals of a container is to isolate the running processes from the rest of the system.

Without any previous consideration, let’s create a container and then create a network namespace.

$ podman pull
$ podman run -d --name my_container -it fedora
$ podman exec -it my_container bash

# Now from inside the container
$ yum update -y; yum install iproute -y
$ ip netns add ns01
mount --make-shared /var/run/netns failed: Operation not permitted

Ok, something is not working: we need first to share the network namespace mounting point with the container. This is done using volumes. Due to the nature of this operation (needs to have root permissions), we also need to give extended privileges to the container.

$ podman run -d --name my_container -v /var/run/netns:/var/run/netns:shared --privileged -it fedora

Now it’s possible to install iproute and then create successfully a network namespace that will be accessible from the container and the host (or any other container with the correct privileges).

A little bug, big problems!

I’ve used podman in this example to spawn the container. I recommend you to read a bit about the container networking using podman (I’m still doing it!). When the package is installed, a default network configuration is installed into /etc/cni/net.d. This network configuration will define how rootfull and rootless containers can communicate between each other and the host. The default network configuration will create a network namespace to isolate the pod networking. That means, when using podman, the host directory /run/netns will be always created.

But that was not happening with docker. When the container was started, the directory /run/netns was still not created in the host. This is not a problem… unless you have an unpatched version of iproute2 or pyroute2 (big drama!). From the patch commit message (and the pyroute2 equivalent):

“When ip netns {add|delete} is first run, it bind-mounts /var/run/netns on top of itself, then marks it as shared. However, if there are already bind-mounts in the directory from other tools, these would not be propagated. Fix this by recursively bind-mounting.”

When a network namespace is created from inside a container, the namespace directories is created without the correct propagation flag. From the host, that is how the filesystem looks like:

TARGET                                SOURCE                 FSTYPE     PROPAGATION
│ ├─/run/netns                        tmpfs[/netns]          tmpfs      shared 
│ │ └─/run/netns/ns02                 nsfs[net:[4026532443]]
│ ├─/run/netns/ns02                   nsfs[net:[4026532443]]

The host, therefore, does not have access to the namespace created by this container:

$ ll /run/netns/
total 0
drwxr-xr-x  2 root root ?   80 Mar  9 20:31 ./
drwxr-xr-x 36 root root ? 1200 Mar  9 20:21 ../
----------  1 root root ?    0 Mar  9 20:31 ns02
root@osdev18:~# ip netns
Error: Peer netns reference is invalid.

This problem was solved in iproute2 v4.13.0 and pyroute2 v0.5.7.

Leave a Reply

Your email address will not be published. Required fields are marked *