Date:
Fresh OS installs are bliss. But the joy fades quickly as installing and
uninstalling programs leave behind a trail of digital debris. Even
configuration management and declarative systems like NixOS miss crucial
bits, like the contents of /var/lib or stray dotfiles. This
debris isn’t just unsightly. It can be load-bearing, crucial to the
functioning of your system, but outside of your control, and not preserved
on rebuilds. Full system backups merely preserve this chaos. I wanted a
clean slate, automatically, every boot.
“Erase your darlings”
inspired an idea in the NixOS community: allowlisting files and
directories that persist across reboots. Anything not on the list gets
wiped. The simplest implementation involves mounting / as a
tmpfs (i.e. in RAM), and then bind-mounting or symlinking the allowlisted
items to a disk-backed filesystem.
This has been implemented for NixOS in the impermanence project, and has some support in Guix too. In NixOS, declaring this allowlist looks like:
environment.persistence."/mnt/btrfs/persistent" = {
directories = [
"/srv/git"
"/var/lib/private/kea"
];
};After three years of running this setup across my NixOS desktop, laptop, home NAS, router, and VPS, I’m sharing my experience. Below is a consolidated view of everything I persist across my machines (each has its own unique allowlist).
/boot, needed for boot./etc/NetworkManager/system-connections, WiFi connection
data. I want my airport WiFi to work after a reboot, and managing this
declaratively is a pain.
/etc/ssh/ssh_host_*_key, SSH host keys./nix, the Nix store, containing all my binaries and
configurations.
/srv/git, my private Git repositories./srv/nas, my NAS: Big Buck Bunny in a variety of encodings.
/var/lib/NetworkManager/secret_key, for stable
RFC7217
addresses.
/var/lib/audiobookshelf, and various other directories for
stateful services.
/var/lib/private/kea, my router’s DHCPv4 leases./var/lib/systemd/timers, the last time systemd’s cronjobs
ran.
/var/lib/systemd/timesync, recent time, useful for machines
without an RTC.
~/.cache/restic, Restic backup cache.~/.config/Signal, Signal Desktop data.~/.emacs.d/init.el, my Emacs config, which lives outside of
Nix so I can easily use it from non-Nix machines.
~/.emacs.d/projects, Emacs stuff.~/.gnupg/private-keys-v1.d/, my GPG keys, used for Password
Store.
~/.password-store, my Password Store repo.~/.rustup, so people think I’m a pretty cool guy.~/.ssh/id_ed25519_sk, my SSH key (Yubikey handle).~/Downloads, browser downloads.~/persist, my personal files, source code, docs, etc.
Everything else, which includes things like:
/etc (mostly). NixOS manages parts of
/etc declaratively, but the rest is ephemeral.
/var/cache, caches./var/lib (most of it), ephemeral application state./var/log, my machine doesn’t persist logs between reboots.
This wasn’t a goal, I just haven’t needed to persist logs.
/var/nixos, NixOS-specific data for stable UID/GIDs.~/.config (mostly), etc. Managed dotfiles are handled by
Nix. Others are ephemeral.
~/.mozilla, my stateless Firefox profile (more on that in a
future post).
~ (mostly), I explicitly use ~/persist for
important data.
(Note: /usr is absent on NixOS systems.)
Peace of mind: My machines are defined by the tuple (nixpkgs commit, config commit, state). That “state” is now tiny, around 1MB, compared to the usual 20-30GB root filesystem. This eases anxiety about system drift. Bliss!
Fearless experimentation: Trying out new software or configurations no longer leaves ruins in its wake. If things get hairy, a quick reboot brings me back to a clean slate.
Easy reproducibility: When I encounter a bug, a reboot confirms whether it’s related to persistent state or a genuine issue. This makes bug reporting and collaboration easier.
Trivial backups: Backing up my machines (excluding the NAS) is quick, thanks to the minimal state.
Clear path to statelessness: The allowlist highlights opportunities to move even more state into declarative configuration. Resistance is futile!
Inconvenience: Persisting data requires conscious effort. It’s not always obvious what needs saving.
Installation complexity: Standard OS installs are simple: boot partition, a root partition, done! But my disk partitioning is now a bit more complicated. While disko helps, it adds to the learning curve.
Undefined behaviour: Applications aren’t designed for users to wipe half their state and though I haven’t run into problems, I expect things would break in unexpected ways.
Performance problems: I’ve noticed bindfs occasionally consuming lots of CPU. While some optimizations have been suggested, I suspect I’ve been living with slower I/O for a while now.
Declarative machine management and ephemeral state has banished my anxiety about stateful systems, a huge win. But this peace of mind comes at a cost: time. Ironically, despite aiming for less upkeep, I likely spend more time than users of conventional systems.
My aim is to simplify maintenance. While abandoning impermanence entirely (or switching to something like Fedora Silverblue) is an option, I’m taking a more gradual approach: expanding what I persist, starting with my Firefox profile. I’ll share my experience with ephemeral Firefox soon.