Linux: Immutable and Atomic Debian
A few years ago, I made the choice to begin running Debian as an immutable distro with atomic updates. This isn't officially supported by the Debian project, and there doesn't exist any how-tos or guides on the subject. While there does exist other distros which natively support both of these features, I choose to roll my own using Debian, as I have been a dedicated and loyal Debian Linux user since the mid-2000s. This also demonstrates the sheer power of Linux in general, in that if a specific feature does not exist, and if you have the knowledge, you can in theory do anything you put your mind to!
In this post, I will explain my methods for turning Debian into both an immutable system, and to have fully atomic updates.
Immutable Linux
First, let's begin with a basic explanation of what immutable means in the context of an operating system. The easiest way to explain this is to have someone boot what is called a Linux Live CD/DVD/USB image. A lot of modern Linux installation images and environments are based on a Live System, where you boot into it, and a majority of the operating system is being run from a combination of this Live Media and RAM. This has been a staple Linux feature since even before I began using Linux, with it's popularity being the highest in the old Slackware distribution that is still around til this day. The original idea behind this distribution, was it was a fully live Linux environment which a potential Linux user can boot from a compact disc, and contained a lot of useful Linux software that the user can try without needing to ever install anything. If they wanted to use the software and save some of their data, they can easily mount their actual hard disc drive to save the files to their DOS or Windows partition. It was the perfect introduction for new potential Linux users. I have used my fair share of these Live Linux environments in my day. This is essentially an immutable operating system, as anything you did not save to an actual disc, will be completely lost once you shutdown or reboot your PC.
Atomic Linux
The idea is simple, but usually difficult to execute. It is to ensure that all of your system package updates are all updated in one single go, and ideally in a completely separate image or partition. One clear example of this is with the Android operating system. Where it usually has an A/B update system. When an update image is downloaded from the Internet, it is placed into a secondary system partition, and when the user then clicks on the update, Android will reboot into this second partition which has all the updates, and if there was an issue, it can easily fallback to the working version of the system. SteamOS also handles updates in this fashion as well.
Since Debian doesn't natively support Atomic updates like this, I made it work by pairing the immutable feature with it. I will explain how it all works in the next section.
Enter the Debian Live Project
The closest existing solution to what I was trying to achieve existed in another project, the Debian Live project. The main purpose of this project is to create the Debian Live CD/DVD/USB images that Debian itself officially distributes as it's Live and Installation environments. This project has changed over the years, and so older code from some past versions no longer work. I still have some old snippets locally, and once before on my BitBucket.org repository which is no longer with us.
If configured correctly, you can use a Debian Live image as both an immutable and atomic distro. In normal use-cases, you'd take the created .ISO file and either burn it onto a DVD, or dd it to a USB stick to boot as a Live image. This isn't actually required at all, and in fact, you can easily place the filesystem.squash along with the Kernel and RAM Disk onto your hard disc and simply have GRUB boot it as it would have from a DVD or USB. This creates the perception of it booting as though it was a locally installed Debian, but in fact, is booting more like how a Live DVD/USB would boot, but rather from the hard disc.
In most cases, a Debian Live DVD/USB should not be programmed to be specific to a machine, but it can contain whatever software and scripts that your heart might desire. You can customize the final Live Debian system in however you'd like, just as you would a regular Debian system, there really is no actually restrictions here.
Persistence in Many Forms
The next thing to making it work more along the lines like a typical distro, is to ensure that the end-user can still save their important documents and projects as easily as possible. This can come in several different shapes. The Debian Live project actually has a method built into itself for persistence, and can be easily enabled and can work in several different ways. I will explore the Debian Live method first, as this can allow a single Debian Live immutable image to be used in several different hardware configurations, and not just tied to a single device.
Enabling the persistence in Debian Live is as simple as either formatting a local partition or external USB stick with a standard Linux compatible file system, and giving it the label of persistence, here is a simple example:
mkfs.ext3 -L persistence /dev/vda
This is one I use when testing these images under virtualization, it will format the virtual block device of /dev/vda with the extended 3 file system to have a label of persistence. The next step is to mount this newly created partition and to create a file called persistence.conf at it's root. Here is an example to say persist a home directory:
/home
That's all that's needed, each line can have a different location you'd wish to persist. And yes, this can even include system related locations such as /var/lib/docker as a fairly popular choice, enabling full docker use in your Debian Live system! If you postfix it with union, you can even persist an entire file system, and only the files and settings you change will persist:
/ union
That will sort of turn off the immutability, but also not. With this, any changes you make to the system will be placed onto your USB Stick, and this also includes packages, so you can customize a system in this way fairly easily. If you want to revert back to the base system, then remove the persistence from your Kernel command-line, and it will no longer use the USB Stick, bringing you back to that immutability.
I used this persistence originally, but about a year and a half or so ago, I moved onto a new machine/device specific method. This method will make the Debian Live system tied to the hardware which it is programmed to work on, and may struggle to even boot outside of it. This method can allow for more traditional Linux storage configurations, and really shouldn't be used with the above Debian Live persistence method.
In order to use this method, you will need an understanding of the Linux boot process, and how file systems get mounted. This method can also support the same LUKS encryption that most Linux distributions provide during their installation. Disc encryption is generally recommended on portable devices, which is where I mainly tend to use it. Another use of data encryption is that of data integrity. If the keys are not known, then no data can be easily injected into the storage device, thus ensuring that the data is only read and written by someone who is otherwise authorized to do so.
To enable this method of persistence, you will need to know your hardware well, and then create the appropriate systemd unit files under /etc/systemd/system/, and then during the actual system build process, ensure that they are enabled. That's really all there is to it, but is much more advanced than the Debian Live persistence method, and is tied to your unique hardware. Using this method, you can then mount actual partitions from your local disc as you would in any Linux distribution. This means that you can have your usual home partition, but since the entire system is immutable otherwise, you will also need to create partitions for other parts of your system you may also wish to persist. This can include various directories under /var/lib for example, or you could even create a huge /var partition to make life easier. This method does require enough knowledge of how Linux works and functions internally, or you could create a potentially unbootable system if you do not configure it correctly. Other immutable and atomic distros will handle all of these mount points and such for you, but with this Debian Live method, everything is on you to configure.
Hacker's Edge
A few summers ago, I was able to launch a new Linux distribution based on Debian, and what I've learned. I had a website and everything up for it, and began creating some sample disc images which anyone could place onto a DVD or USB to boot themselves. The initial images I created were for the Plasma desktop with some additional useful software, and a gaming focused Live Image which contained Plasma, nVidia drivers, and of course Steam and OBS all ready to go. It was still in it's early stages, and at this point I was still homeless and hopeful that this project might get me out of that situation. About a few weeks after I launched the website, an previous co-worker contacts me about a job position in my field, and since I was homeless and money was running low, I felt like I had no choice but to accept the job offer, and so I did, and put this project into the back burner. I later began working on it for a bit when I had time, but once I was able to finally move into a place to live, this job offered me a full-time position, where prior I was just on contract. In the terms, it mentioned specifically about side-projects, and so I ultimately abandoned Hacker's Edge, and haven't returned to it much since, besides the odd YouTube video that is now no longer accessible.
However, not all was lost from this project, as I created some neat software to make building these custom Debian Live type systems very easy, trivial and less error-prone. I created a program, which I still use til this day to manage my various Debian immutable and atomic systems. It works using a unique scripting dialect I created, along with various mods as I called them, each injects a unique configuration into the final Debian Live configuration directory while allowing for multiple build configurations. One mod for example is specific for nvidia and another is specific to steam, and each of these ensures that specific Debian packages, repositories, and other system changes are performed to ensure that each of these is 100% reproducible and successful, oh, and of course transferable to other Debian Live configurations I might create.
This unique software I created for example, allows me to have 2 unique systems for my laptop, one for my everyday Debian productivity, and another specific for gaming and Steam. Both use the same nvidia mod I mentioned above, so it's easy as that to share similar machine configurations between systems I might build. Each time I need to atomically update either system, I run my software, point to the system I want to build, and well... use Debian Live to build it! Once the system is built, I copy the new filesystem image into place, and place the kernel and ramdisk, then boot into it.
At one point, I had ambitions to turn this into a business, where a company that would like to transition over to Linux, or even has specific uses internally for Linux on select systems can utilize my services to obtain an easy to use and build Debian Live configuration tree which they can either further customize themselves, or just build on-site to produce a working Debian Live .ISO which they can then either boot from a USB Stick, or just copy over to the machine's hard disc, providing them with all the needed software in an immutable and atomic form.
Another idea I had was to also create easy to use Debian Linux variants using this system, where say a small business would like to just run a simple internal Samba server to serve files to Windows clients, but don't want the hassle of installing Linux, and doing all the configuration themselves. This Debian Live ISO would use the persistence system built into Debian Live to store their specific Samba server configuration, while also providing a nice UI for someone without much or any Linux skills to still easily configure Samba to their specifications. The system won't even need any Internet access, as they can easily update the entire system by merely just downloading the latest ISO, and booting into that, which will then use their existing persistence, so their Samba server doesn't need to be reconfigured.
I'd still love to start such a business, but I lack the skills needed to actually start a business.