Real out of band connectivity with network namespaces
This post explains how to configure on a Linux server a second and totally independent network interface with its own connectivity. It can be useful to access the server when the regular connectivity is broken.
This can happen thanks to network namespaces, a virtualization feature available in recent kernels.
We need to create a simple script to be run at boot time which will create and configure the namespace. First, move in the new namespace the network interface which will be dedicated to it:
ip netns add oob ip link set eth2 netns oob
And then configure it as usual with iproute, by executing it in the new namespace with ip netns exec:
ip netns exec oob ip link set lo up ip netns exec oob ip link set eth2 up ip netns exec oob ip addr add 192.168.1.2/24 dev eth2 ip netns exec oob ip route add default via 192.168.1.1
The interface must be configured manually because ifupdown does not support namespaces yet, and it would use the same /run/network/ifstate file which tracks the interfaces of the main namespace (this is also a good argument in favour of something persistent like Network Manager...).
Now we can start any daemon in the namespace, just make sure that they will not interfere with the on-disk state of other instances:
ip netns exec oob /usr/sbin/sshd -o PidFile=/run/sshd-oob.pid
Netfilter is virtualized as well, so we can load a firewall configuration which will be applied only to the new namespace:
ip netns exec oob iptables-restore < /etc/network/firewall-oob-v4
As documented in ip-netns(8), iproute netns add will also create a mount namespace and bind mount in it the files in /etc/netns/$NAMESPACE/: this is very useful since some details of the configuration, like the name server IP, will be different in the new namespace:
mkdir -p /etc/netns/oob/ echo 'nameserver 8.8.8.8' > /etc/netns/oob/resolv.conf
If we connect to the second SSH daemon, it will create a shell in the second namespace. To enter the main one, i.e. the one used by PID 1, we can use a simple script like:
#!/bin/sh -e exec nsenter --net --mount --target 1 "$@"
To reach the out of band namespace from the main one we can use instead:
#!/bin/sh -e exec nsenter --net --mount --target $(cat /var/run/sshd-oob.pid) "$@"
Scripts like these can also be used in fun ssh configurations like:
Host 10.2.1.* ProxyCommand ssh -q -a -x -N -T server-oob.example.net 'nsenter-main nc %h %p'
Since systemd release 242 the NetworkNamespacePath directive can be used to automatically start a service unit (and its own socket units, if any) in the specified namespame, e.g.:
[Service] NetworkNamespacePath=/run/netns/oob