Kevin's Research Blog

Linux: My Secretly Kept Workflow Revealed

It's no secret that the only Linux distro I've been using for at least the past 2 decades has been none other than Debian GNU/Linux. I only briefly distro hopped in the very early days, to explore what else was available, but also because in the mid-2000s, being stuck on Debian 3 Sarge was very limiting, especially at a time when Linux software was literally exploding. As a result, I briefly used SimplyMEPIS, which is in itself, based on Debian Testing at the time. Now a disfunct distro and no longer being developed.

Anyways, this post isn't about what past distros other than Debian I used as a daily driver, but rather my overall Linux workflow that I don't tend to overly share that much.

Debian GNU/Linux, Why Exactly?

Based on this past situation where I found myself left behind in terms of software, why would I still stick with Debian then and not use a distro which does provide more recent versions of software?

This is actually a bit interesting. Over time, Open Source Software on the Linux desktop began to slow down in development quite a bit, much less than it was in the mid-2000s at least. As a result, using Debian as a desktop workstation in 2026, is much different than using Debian as a workstation back in 2006. Debian now only just ever so slightly uses older versions, but these days I see it more of an advantage, than a deficit. The reason being is that this provides Debian with it's incredibly rock solid stability, as by the time the latest packages finally reach the latest Debian stable, they are indeed now classified as stable, as all the bugs which were in the original release have been all ironed out and found by the poor people using those more bleeding-edge distros which include the absolute latest versions of everything. I let all of these Linux users test the software that I will eventually use in Debian, so thank you for testing all of this amazing software before the Debian people package it all up for me to consume. Now, if you feel this is a fallacy, because I'm using an older version of said software, so it must contain the bugs from that release, you'd be wrong. The Debian maintainers are amazing at ensuring that all downstream patches are fully applied to the older version. So, while I might be using an older version of something, it will still include the patches and security fixes from upstream fully integrated.

However, this isn't the only reason I've stuck with Debian over all these years. Since the first version of Debian I've used, Woody, it has been the distribution with the most official packages available in it's repos, meaning I spend less time searching the Internet for an open source program I want to use, and just need to literally apt-get install it. Let me put this package count into a bit of perspective on how large this collection of free and open source software has grown throughout the years. Woody had roughly 8,500 packages available to effortlessly install with a single command-line or via dselect. When I previously distro hopped to some other distros I got from a Linux magazine, such as Mandrake, yeah before Mandriva, it didn't have that much software to install and try out... For some one just transitioning from Windows 98 to Linux at the time, it made it rather difficult to find software, Debian took that struggle away. I still recall spending countless nights just scrolling through dselect many software categories, reading the various descriptions and installing whatever sounded fascinating. I feel like this is the best way anyone brand-new to Linux should be experiencing Linux for the first time, as it provides a much more positive feedback loop in the mind. Each time I found an interesting open source software, I got a nice hit of dopamine, which reinforces my need to continue scrolling through dselect to find even more fun software to try out. This is all before the concept of the modern App Store was a thing, as I feel that the Debian repo was in fact an early concept of what an App Store can be, except for open source software. Today, with Debian Trixie, the official package count is now at 69,830! That is all free and open source software that you can just install with a single click in most modern Linux package management tools, which is rather incredibly! On top of this super large official package count, Debian also supports a huge range of CPUs, so my Debian skills can be immediately transferred and used on other systems, which I have done multiple times throughout my life. I've installed Debian on an old SPARCstation, which is actually the place where I ended up compiling the Firefox browser for the first time. Debian just works anywhere you'd want to have it run, which is what makes it so powerful, and the natural go-to distro for me for absolutely everything. Debian is quite literally the Universal Operating System, which has been their slogan for as long as I can recall, and it holds very true.

Another aspect about Debian, that I'm not even sure that a lot of people even know, is that they provide the software in their repo without branding, and entirely from the core developer themselves. This means, that a lot of software, as long as the original developer configuration works with Debian, that original developer configuration is used on Debian. This means that with Debian, you get the experience that the developer of that open source software wanted as a base experience. A lot of other distros tend to customize each application's configuration instead, and add branding. My KDE launcher icon isn't a Debian logo, although I can change it to a Debian logo if I wanted, it is the original KDE Plasma icon provided by the KDE Plasma team. This branding thing took an odd turn many years ago with Debian renaming software like Firefox to Iceweasel to debrand Firefox... This is no longer the case, but it really shows how dedicated the Debian team is at preserving and ensuring that the open source software you install from it's official repos are in fact the experience the original developer of that software package wanted you to have, except for Firefox for awhile I guess...

Okay, now that the reasons for Debian are out of the way, let's continue...

Customized Workflow

My early days with Debian were rather interesting, as at this time in my life, my main programming skills included either BASIC or PHP, and that's it... When trying to customize my Debian systems back in the day, I was really limited, at least when compared to what I was able to do on Windows 98 with programming tools like RapidQ, which eventually got a Linux port later which I did end up using. It's a Visual Basic-like dialect, but it was free to use.

As a result, my early Debian systems were not very customized as they are these days, I tended to use BOCHS, and then later on QEMU 0.9.1 to run other PC operating systems in Linux, at first I stuck with MS-DOS due to my more limited hardware at the time, but later on, began using early versions of Windows. This isn't really customization whatsoever, but I did create many .desktop files to be-able to easily launch these virtual systems, and might see about adding a screenshot here if I can locate one later on within my older backups. A lot of my early customizations were on the Linux desktop as a result, rather than anything within the system itself.

Once I began to dive into Python back in 2010, this is when things changed a bit. I still wasn't able to really create many GUI applications as I once did with Windows and RapidQ, but I was finally able to write background scripts and services in Python, to allow for automation and customization. Oh, I almost forgot, before Python I did use tools like Zenity which has a really nice command-line interface perfect for use in shell scripts. A lot of my early Linux GUIs were all made using this, including after learning Python, as I can launch Python scripts from a shell script which called a Zenity dialog box.

Since most of my skills are adapted from Linux desktop usage, rather than having a server focus, a lot of my peers usually see my overall Linux skills as a bit lacking, especially when it comes to some command-line and shell scripting capabilities. Personally, it doesn't bother me much, because if I need to quickly remember how to do something very specific in either sed or awk, prefer sed btw, I can just bring up the man page via man sed, and quickly remind myself. I admit that I haven't fully memorized how to use each of these commands, but also don't feel like I actually need to, as the man pages are always a quick command away, and can easily be search and scanned with minimal effort. I have memorized some of the more common operations with sed, such as being able to perform a global string replacement, and know grep well enough to get my tasks done. Another command I also find myself using a lot is find. I've also memorized very common Linux utilities such as tar, gzip, bzip, and of course ssh. In fact, almost every core system utility has been memorized at this point, as I tend to use these everyday, whereas I don't use sed everyday, maybe once a month if that.

The parts about Linux that I probably know the most isn't really the commands, but rather the system itself, where to find various components, and what each special file system has and where I can find certain things. For example, if I want to control various LEDs or my backlight on my laptop, there is a nice place under /sys/ that has all of this. I've deeply explored the various Linux system components, and know my way through a Linux system fairly well. I do have a slight obsession with file systems, so that is perhaps why it was easier for me to learn these aspects of Linux over say using commands like awk.

Hopefully that ramble explains my Linux skills a bit to people who may have previously misunderstood me, as I feel like people don't understand how I could have been using Linux for this long, but still need to review a man page for some commands which well experienced Linux users should just know. If there's an easy to access document available, then why take the extra brain effort to learn it? Linux commands are amazing at the help it can provide via man ..., info ..., or even [command] --help. I memorize what I feel is most important, and leave the rest to just look-up adhoc when I need to remember, it's just the way my brain works, so get over it.

Modern Workflow

Now, let's get to the more juicy details, now that I've perhaps over explained why I prefer Debian, and how my Linux skills probably differ than the next guys...

My modern workflow is a mix of custom command-line, services, and desktop applications, which tend to all communicate with each other. Since getting back into Pascal many years ago, and finally being able to effortlessly create full Linux GUI applications for the desktop, a lot has changed in my Linux workflow.

Prior to running a fully immutable Debian, I initially provided these customizations through a blunt copy with no central configuration. This obviously led to many issues with consistency and being able to reproduce my configuration on another system... Least to say, it had it's problems. Once I began to learn SaltStack after attending a PyCon, I became hooked on this rather amazing Configuration Management Tool, and everything in my home was configured through it, and later my cloud server. I finally had a central configuration for all my customizations.

Fast forward to me going homeless in 2023 I think it was, I cannot even recall the exact year anymore oddly, and I don't care about looking it up... However, this upcoming situation of mine prompted me to look into how to make Debian immutable, and I recalled the Debian Live, which isn't really meant for this purpose, but I was able to make it work. I had some previous experience in creating custom Debian Live images as far back as 2010, where I didn't use for immutability, but rather to just try running Debian from USB and RAM. It was an interesting experience and I was able to do quite a lot with it, such as embedding the Eclipse editor with Python support enabled right into a Live image. Before going homeless, I jumped back into it, and refined a new Debian Live image that wouldn't boot from an external device, but rather, it would be placed onto a boot partition of sorts where it can be booted by my local GRUB. My Thinkpad had a decently large root partition, which was also the boot partition where GRUB and such were located. Modern PCs don't really need a separate boot partition for MBR booting as was the case for older BIOSes, however, now with EFI booting, a dedicated boot partition is once again required. However, my Thinkpad is a traditional BIOS and uses MBR booting. Over several weeks, I fine-tuned a Debian Live image with KDE Plasma, and all the software packages which I was currently using on my main PC at the time, and kept testing and iterating until I had a perfectly working immutable Debian system containing all of my favourite open source software.

Why did I want to create an immutable Debian before going homeless? Glad you asked! Since being homeless, I will lack a lot of resources that I'd normally have, and so if my Linux system had a bad update, bad configuration, or I made a mistake somewhere, it would be incredibly frustrating to fix while in a homeless state. I wanted a super stable, reliable, and quite literally, an unbreakable system. This system wasn't lacking either in it's capabilities, when people normally think of a Live Linux distribution, they think of the limitations it poses, as most Live Linux systems aren't meant to be used as a daily driver, and most exist solely for the purpose to host a GUI installer, and so the software selection in this live environment is barely productive. Here's a shortened list of popular software I had integrated into my own immutable Debian, and still have on my current immutable Debian:

There is a lot more within my immutable Debian system, this is just a bit of what's fully integrated and configured for use. You now might be wondering, well persistence, so it's not fully immutable. It's immutable where it counts, in that if something within the persistence breaks, I can always safely boot into a working system ALWAYS.

How does this persistence work? Well, this is yet another complicated story, but I'll save it for a future section. I first want to take a small step back to just before this immutable Debian which I currently use.

Virtualization and Compartmentalization

A past co-worker once questioned me why when I'm streaming or recording my desktop with OBS Studio, why I was always recording the desktop of a virtual machine, rather then the desktop of my host. It was really odd for him to even point this out, like it isn't really his business, and felt really awkward for a past co-worker to even be so judgemental, least to say, I no longer associate with let's call him, Mr. Jenkins. Friends that purposely try to push judgement only to make you feel bad, aren't really friends at all... Anyways... The reason for this was actually two-fold, I wanted to record my actual Linux desktop, but I actually couldn't... You see, either the nVidia card or drivers at the time on this PC had a weird bug when it came to recording the entire desktop, in that 25% of the top of the screen during recording would be tearing, and be very distracting, so in order to create quality video recordings for people to watch, I couldn't record my full desktop, or they would be subjected to this awful recording glitch which would only distract and cause people to comment about it... So, while OBS Studio did run on my host, I usually selected a single window, usually the window of a virtual machine running in libvirt, and then showing all my tasks, such as programming inside there to alleviate the problem. I say this turned two-fold, as it was this situation which actually got me to experiment more with compartmentalization, where I'd dedicate specific virtual machines for certain purposes, such as the development of different programs, or running certain games. In retrospect, this is how things are going in software anyways now, I was just a bit ahead of the curve because of my odd nVidia video recording bug. Here are some of the development systems I put together back then as an example:

I still continue this overall practice today, except now I can finally record my host system's desktop, which I love, as I can show off some of the fun visual effects that I wasn't previously able to showcase in videos.

Immutable But Persistent Debian

The Debian Live project provides methods for enabling persistence within a live image, I'm not sure how widely used this feature actually is, but I am definitely one of the people in the world that has actively used it.

When I went homeless, I also needed to ensure that I can still save information and have a persistent desktop experience. I did this in a rather interesting and perhaps unorthodox way. At the time, I had several unused SD cards, and my Thinkpad had a barely used SD card slot. I thought this was perfect, as when an SD card is slipped into my Thinkpad, nothing is bulging out like a USB stick, making it much more manageable overall. As I wanted to explore more with this persistence, I configured several different SD Cards with varying configurations of persistence. This is done with Debian Live like so:

As you could imagine, there are many ways to configure your persistence, which is why I configured a few different SD Cards with different persistence.conf files. My main configuration was actually very different in that I had a single line /etc union, which I'm sure you can guess what that will do, it will persist my entire /etc file system. Why would I only persist this and nothing else? To enable proper LUKS support of course! I'm homeless on my only machine I have in various public locations, and have perhaps a lot of personal information that I wouldn't want otherwise stolen, so I used encryption in this way. It uses various systemd .mount files to mount various partitions from my internal hard disc into the Debian Live file system, so rather than say using /var/lib/libvirt or /var/lib/docker in my persistence.conf, these were instead actual partitions from my internal SSD. I also did this for /home as well of course, creating a semi-persistent system running from a Debian Live system. It works rather well, except for when it comes to updates...

Immutable Debian Updates

At first I had my Debian Live configuration, and it's cache all persistent on an external HDD, and just ran the lb build command every so often to generate a new immutable image which I'd then move into my boot partition. Over time, I wanted to better automate the creation of custom Debian Live images, namely for my Hacker's Edge project, which was a new immutable Linux distro I was once working one while I was homeless during this period in my life. A Live image created for public use would need to be very different than my own, but it would still share some similarities, such as software and some customizations. I didn't want to manage a dozen different Debian Live configuration directories, so I created a specialized tool, which I still use today, to generate the needed Debian Live directory structure allowing me to consistently build the same image with no errors.

However, since going homeless much more recently, and not entirely preparing, as my family mentioned I could stay with them, but then later lied, I thought I wouldn't need to worry about creating a different way to update... Least to say, I haven't performed a full update using my custom tools for several months now, but found a new way to patch in updates until I can one day built myself a new image. Why you ask? Well, when I build these Debian Live images, I always use a local mirror powered by Approx so I don't constantly hit the Debian mirrors taking up additional bandwidth. My entire build system is based on the fact that this local mirror exists, and while it's not too difficult to actually change, I don't currently have a place for this local mirror, and don't want to overly use the official mirrors.

Since I still do need to update my system, I had to do some research on how I could do this. At first, I began looking into the Debian Live, config-hooks system, where I can create some shell scripts on my boot partition which are then executed during the Live image boot to further customize it, such as updating packages. The huge downside of this, is that if I install packages in this fashion, the packages are installed into RAM, leading me to having less RAM available after boot for my applications... I needed to look into other ways, and then I realized a feature I only briefly read about, and accidentally used before, which is to merge immutable images together. Typically with Debian Live, you'd only have a single filesystem.squashfs on the live medium, however, if you place other file system images into the same live/ directory, they will be combined based on it's filename. In order to create a patch image, it would need to start with a letter after f. These file system images can be in various formats, such as .ext3. However, for initial testing, and simplicity, I choose .dir.

How does an update work using this method? Here's how it works. I boot into my immutable Debian as normal, then run the apt-get update && apt-get upgrade commands to update the packages on the Live system. Then here is where the fun part happens, I then need to create the patch .dir directory, and then copy the RAM disc image over to the boot partition, such as cp -av /run/live/overlay/rw/* /run/live/medium/live/updates.dir/ then reboot! This essentially copies the updates from RAM where they were actually installed into a persistent location, which has the ability to override any existing files, such as Firefox or Chrome. Upon rebooting, the system will then create combine both the filesystem.squashfs and updates.dir as a single unified root directory for the immutable system. This ensures that my system updates aren't taking up RAM, and are still immutable, but just layered on top of my existing system. This also allows me to easily revert my entire system back to the original if an update ever breaks anything, giving me a unbreakable system.

My Current System

Which finally brings up almost to the end of this post, the above explains how I currently manage my updates without a local mirror, but how does my current immutable Debian differ from how it was built before?

Well, I'm no longer using the Debian Live persistence options, but still have persistence! However, when I eventually can do a full rebuild, even this current workflow will change slightly due to this new knowledge of patching with a .dir directory. My current immutable Debian is purpose built, in that if you took my current filesystem.squash and tried to boot it on some other hardware, it will only get stuck in the boot process and never finish booting properly. The reason is that I baked the .mount units directly into my immutable Debian, so during the boot process, it will assume that my local partitions exist and will try to mount them, and the boot will fail if it cannot. However, after this new knowledge, my next iteration will instead place these .mount units into the boot partition instead, which can then allow for customized images to be layered if needed in the future, allowing this filesystem.squashfs to be bootable on any system I might own in the future with zero issues, and I'll have all my software ready to go in mere minutes, rather than having to go through a full operating system installation. I'd just buy a new laptop, partition it's EFI, copy on GRUB, and my filesystem.squashfs from a Live USB stick, a 5 minute process, then boot into the exact same immutable Debian with my full KDE Plasma desktop and all my software in mere minutes. And that is the actual advantage of this configuration, is that it can allow for such scenarios, a literal zero installation, with immediate results.

Okay, now with Debian, and how my immutability works, let's move on to the rest finally...

I mentioned above that my immutable Debian has firejail, and this is used in various places. The main ones are with my web browsers, so when I launch either Firefox or Google Chrome, they are always running within an isolated firejail. As I don't tend to do immediate updates on my system, this is very important, as it prevents any potential zero-day issues from deeply effecting my system, as it would be limited to the isolated environment I provide each of the browsers via firejail.

More recently, I began using some more advanced feature of firejail to isolate both LM Studio and Chatbox from the rest of the system, and network. Both of these applications run on an isolated network bridge without any external access, but can communicate with each other still as they are in the same network. As these LLM programs give the AI models the ability to use various tools and call out to online services, I wanted to ensure that I was in control of what the AI was able to do, rather than the AI being able to just do anything itself. This isolation prevents the AI from using tools which could otherwise cause damage to my system. Since I am a developer at heart, and am in the process of developing AI focused applications, this is very important for testing and the development of various tools.

Other than using firejail for process isolation, I also tend to use a healthy mix of local KVM and LXC instances for a variety of purposes. I decided that I wanted to place my access to my blog inside an LXC container running Firefox. The original idea was that I probably wasn't going to be-able to update my browsers without a local mirror easily, but still wanted to be-able to browse the Internet with an updated web browser, so the natural conclusion was to run Firefox from within an LXC container, which works rather well. I don't do any multimedia online, and the LXC container also has an instance of Privoxy as I'm using my mobile phone's hotspot as my primary Internet connection, and well, ads tend to take a lot of useless bandwidth, and this is a good solution to keep my bandwidth consumption low while still being able to browse the Internet. Performing updates in my LXC container, is just like a regular Debian system, allowing me to ensure that everything is always updated before going online.

Conclusion

In the modern era of computers there are a lot of risks online, and we need to ensure that our systems remain resilient to any potential issues. I have carefully crafted and iterated over time an immutable system which will always assure that I can boot into a working system. To further enhance my overall security online, I also use tools like firejail and LXC containers to further isolate programs which otherwise have access to the Internet to ensure that they just can't run rouge on my system. I'm sure that I will only continue to further refine this in the future.

#current #debian #immutable #linux #networking #projects #thought