Linux: Advanced Networking Under Linux
The sheer amount of networking possibilities under the Linux operating system I think go a bit under noticed as Windows is the dominant platform for end-users, including end-users who might be using various emulation and virtualization software. A lot of documentation for networking under Windows is usually in the form of a standard bridged network. This sort of networking option is fine for most projects and end-users, but with more interesting options, it can enable some rather interesting scenarios, which are only possible under the Linux operating system. I've been using Linux for quite sometime, and while I was sort of aware of the TAP/TUN interfaces, which are usually used by various VPN Software, and the popular Bridged via the brctl, I never was really exposed to either of these much until I decided to move from the VirtualBox virtualization software I was using since the late 2000s, to the open source alternative KVM through the very accessible libvirt virtualization library. While using VirtualBox, I was very accustomed to their Host-Mode Networking I believe it was called. This is where a virtual machine can be placed on the same network as the host, and have an IP address on my home network, it made it very trivial to share resources from a virtual server on my network without needing to resort to NAT and port forwarding. libvirt has a similar alternative called MacVLAN, but the catch was that the host cannot see the virtual machines, and the virtual machines cannot see the host. This added the requirement of needing a secondary network adapter on the virtual machine so that if I needed to say mount an NFS Share between the two, I still could. This led me into looking at all the possible networking options libvirt had to offer.
In this post, I will present a couple years of research and solid use of various advanced forms of networking I've discovered under Linux while trying to find a decent alternative to the Host-Mode Networking offered by VirtualBox.
Routed Networking
By far the most interesting networking option which I have discovered was that of Routed Networking, which can be created as a new network in libvirt or in the popular Virtual Machine Manager. If configured correctly, it can really expand your home lab in rather interesting ways. Gone are any limitations of only using IP addresses in a single subnet your network might be on, but can expand it to multiple subnets. This can work in a few different ways, depending on how your network and home lab is configured.
By far the easiest way to implement this networking model is if you run your own router, and can install libvirt directly onto it. Once you create a new Routed Network, it will configure the needed firewall and routing rules to make sure that packets get delivered where they need to, and because it is on your main router, any computer or system connected through this router will immediately have access to this new subnet with zero configuration. If your libvirt server is on a separate machine, and you can at least add custom routes to your router, then you can add the appropriate routes to it, which will also make the new subnet appear mostly seamless, but it would be best to add these routes to your DHCP Server configuration instead, so that each client connecting to the router will obtain the new routes, rather than having to go through the router. The less easy method is to add the manual routes for the new subnet on each system or device that needs to access the virtual systems on that subnet, which isn't too difficult.
Using this sort of configuration for virtual machines in a home lab, it can allow some separation from your physical network devices, and the virtual ones, while still enabling the virtual systems to have fully route-able IP addresses.
Internal Network
Another option available in libvirt, but also in VirtualBox. It is very simple, and in the case of libvirt, you can choose to either have it create it's own internal DHCP and DNS server to allocate addresses, or you can turn those features off for a fully isolated internal private network.
At first glance this sort of network might sound counterproductive, as these systems cannot access any external resources, such as the Internet, at most, the systems on this network can still communicate to the host, but that's it. Where this networking option shines is with retro operating systems that you don't want on the Internet, as they are too outdated, but you still want to have them on a network of sorts to either share files easily with a samba server, or with other virtual systems. Personally, I find it easier to use a retro system's native file sharing capabilities than it is to mount it's disc on my host, as mounting it's disc image on my host also requires that the virtual system is offline. Other options for sharing files are mostly to these retro systems, usually by floppy disk images, or CD images, or sometimes by turning a directory into a virtual CD-ROM for it. These don't allow data to be easily transferred back out, the floppy is size limited, and mounting the entire disk requires that the VM is turned off.
In a section later on in this document, I will also explain how to combine something called VDE Networking and this Internal Network for some rather interesting results.
Network Bridges and TAPs
Before I can dive further into some interesting examples, let's first explore what we can do with Network Bridges and TAPs, and how they can be created.
A Network Bridge can be normally created via the brctl command on Linux, but did you also know that each of the virtual networks you create in libvirt are also the same type of network interface? With libvirt, they are bridges with the names of virbr0, virbr1, etc... You can actually use the standard brctl command to do some interesting things with these bridges, if you needed to.
A TAP can be created via the ip tuntap command, as seen here:
ip tuntap add mode tap name tap0 user <username>
ip addr add 192.168.4.1/24 dev tap0
ip link set tap0 up
This will create a new tap0 interface with the IP address of 192.168.4.1 on your system. Alone, it's fairly useless. However, when paired correctly with Network Bridges and VDE Networking, you have some of the most powerful networking tools at your disposal!
As a way to whet your interest, here is how you can add this newly created tap to a bridge: brctl addif virbr1 tap0. We'll see why this is useful soon.
Adding Docker Containers
Wouldn't it be awesome if you could say mix lightweight Docker containers with the power of libvirt leveraging both KVMs and LXCs when appropriate? This is actually possible during a couple different methods, which I will explain in this section.
The first method, is to create a new bridged network in libvirt which connects directly into one of your docker made networks, here is a network definition you can define via virsh net-define docker0.xml:
<network>
<name>docker0</name>
<forward mode="bridge"/>
<bridge name="docker0"/>
</network>
This will create a new network for any libvirt system to use, either a full KVM, or an LXC. It can chosen as usually when creating either of these system types. It will then be a part of that network in docker, and be-able to access any of the containers that are running within that network. This can be useful for such situations, where you want to explore the container network from within the container network itself. So, if you install a desktop OS, you'll be-able to point to any IP address on that container network and see it's resources.
The second method for connecting up various container and virtualization technology is to use a macvlan network in Docker. This means using the docker tools to create a new docker managed network using the macvlan driver. The interesting part about MacVLan is that you can connect it to a TAP! I have one docker network I created aptly called tapnet, which uses the MacVLAN docker driver to connect to tap0. So now, perhaps some of the more interesting uses of a TAP/TUN are now coming to light? Any docker container you spin up in this docker network will be on that new shiny tap0 network interface you created.
Enter VDE Networking
Okay, so thus far, all of the networking we've discussed is fairly primitive to what I am going to dive into next. This next variation on advanced Linux networking is most useful with emulators and virtualization software that supports it. One such example where I use it loyally is with 86Box.
Here's the scenario and goal. We want to host a samba server capable of speaking with Retro Windows guests so that we can easily share files between various retro systems that do no support modern CIFS protocols, and exposing it on a real network would otherwise be a security risk. Here is how that can be done with VDE Networking. This can also connect retro systems running in 86Box to both KVM and LXCs as well, so you can say run an older version of Linux in virtualization, as running Linux in 86Box can be a bit sluggish, but running retro Linux in libvirt works really well. It's always good to leverage whatever technology you can for the best outcomes.
First, you'll need to configure that tapnet docker network I mentioned above, create a macvlan docker network connected into your new tap0 network interface. Next, you'll need a samba configuration which will allow older versions of Windows to speak with, here is the configuration you can add to modern samba when installing a samba server in a docker container:
[global]
server min protocol = NT1
lanman auth = yes
ntlm auth = yes
This will allow very old clients to connect to the samba server, this includes the MS-DOS clients, as well as the Windows 3.1/9x clients. I have this in a repository, which you can check-out and use to build the docker container with.
Next, you'll need to create what is called the VDE Switch, this can be easily done after installing vde via: vde_plug tap://tap0 switch:///tmp/retro.
This will run in the foreground and create a FIFO in the /tmp/retro directory for VDE compatible programs to connect through. With that, just create a new retro system in 86Box using the VDE Networking option, and place vde:///tmp/retro into the box as the network. Start the retro system, configure it's networking manually to have an IP in the same subnet as you configured for the docker network tapnet, and your retro system should be-able to communicate with the less security samba server running in the isolated docker container and network.
If you'd like to have full DHCP available so that you don't need to manually set the IP in your retro systems, then you'll need to bridge the tap with a libvirt network bridge: brctl addif virbr1 tap0. Replace virbr1 with a newly created Internal Network with DHCP you created in libvirt. The added side-effect of this bridge should be clear, any KVM or LXC you add into this network will also be accessible to the retro systems, adding for some interesting use-cases. This could enable say, running an HTTP Proxy Server in a KVM that has access to the Internet, ensuring that your retro system remains mostly safe from the Internet. You could also use it to host period Internet software applications that your retro systems can now access without the Internet. You can even install Database Software which still has supported client libraries and software for your retro systems, allowing them to do more than they could before. This can really open the door to some really interesting things you can now do with your retro systems, that you couldn't do before.
Connecting with Fellow Retro Friends
An idea I had, if I managed to make any friends into retro computing was to allow us to say play multiplayer games over the Internet using a private VDE network. Let me explain how this would work in this section.
One scenario I currently have configured is that on one of my Raspberry Pis, I have libvirt running with several LXCs. One of these LXCs is in an isolated network which can only be accessed in a very specific way. It hosts some of my backups for my retro systems, allowing me to use native retro backup software, along with native file sharing of the operating system to perform backups. This is how it can be done, and this same idea can be applied for being able to communicate across the Internet as well, and not just to a locally connected device.
On my Raspberry Pi, I have a libvirt network, which is local to the Pi, has no external access. Let's say this is called virbr1. Then I have an LXC which is running samba with the above configuration to allow retro systems to connect into it, and access it's shares. This LXC is on this internal network. For this to work, the Pi will also need to have a configured tap0, please refer to the above lines to create that on the Pi. Next, the tap needs to be bridged to the internal network, so follow the commands above for that as well.
Next, on your PC with your retro systems running in 86Box, perform the VDE networking configuration, but instead of running the VDE switch as mentioned above, run this command instead: vde_plug switch:///tmp/retro = ssh mypi vde_plug tap://tap0
A few assumptions here, firstly, the VDE software also needs to be installed on your Pi. Next, your username on both systems are the same, although they don't need to be, just change mypi to username@mypi. You should also configure SSH Key authentication so that you can SSH to your Pi without needing to enter in your password. Otherwise, what this line does, is connect a local VDE Switch to a tap device on a remote system over SSH! Isn't that so neat? It also becomes possible to connect this to your local TAP still via: vde_plug vde:///tmp/retro tap://tap0. Of course in another terminal tab, which will then connect the tap on your Pi with the tap on your PC. Just make sure there is only one DHCP Server running...
Using this sort of networking with friends whom are also into retro computing can allow for multiplayer games over the Internet using any protocol, not just the routable TCP/IP, and no additional software is needed in the retro systems, just connect as you would have in the 90s as a peer-to-peer connection. It can also allow for any other retro protocol to work in long distances, as I mentioned I use this to easily store retro backups directly on my Pi, so native retro file sharing is 100% possible using this method as well. The only barrier is that each person will need to be running Linux and use the VDE Networking, and at least one will need to expose an SSH server. An alternative being to host the TAP network on a cloud server, then use the VDE networking, which will still be a requirement to connect the VDE Switch with this cloud server TAP, but a local SSH server will no longer be required.
VDE Networking has a few other interesting network types as well, so you are also not required to use SSH either, as other methods are also available to connect multiple VDE Switches and taps together. In fact, the TAP is also entirely optionally! For example, you could do something like this as well to connect to VDE Switches without a tap: vde_plug switch:///tmp/localsw = ssh mybuddy vde_plug switch:///tmp/buddysw.
I highly recommend reading more into VDE Networking if this has made you more curious about it's potential. It is a rather powerful networking toolkit that has a lot of options.