Soap boxing here but I feel these kinds of use cases is what NixOS is built for.
Declarative config to setup the system, users, and apps.
Declarative and customizable impermanence exactly how you want it.
I use Tails as well but NixOS is my daily driver. Anything not marked explicitly to persist is dropped each reboot. I’m the only user so I keep the last 30 days of non persisted data for like a power outage but that’s something I had to go out of my to setup for my use case.
This is why we trust but verify. Thanks mom for teaching me that cruel lesson of unplugging the phone cord to get me to bed (dial up days). It lasted about a week before I caught on you always came up from the basement before bed.
I’m so glad you never noticed I swapped my line with the guest bedroom. Also glad that ancient block in the basement could be hand wired.
For our lower environments we use rsync like the author but skip the pipeline altogether. The servers have a watch script to restart when files are rsynced. We then have a local watch script that rsyncs on file changes.
Relatively instant deploy (2-5s) whenever a file is saved.
Good call on a simulated failure. When I first set it up, it was LVM/BTRFS or ZFS as my top choices. It was a coin toss at the time because I hadn’t built this sort of setup before.
I use immutable nixos installs. Everything to redeploy my OS is tracked in git including most app configurations. The one exception are some GUI apps I’d have to do manually on reinstall.
I have a persistence volume for things like:
I have 30 days (or last 5 minimum) of system rollbacks using BTRFS volumes.
The personal files are backed up hourly to a local server which then backs up nightly to B2 Backblaze using rclone in an encrypted volume using my private keys. The local server has a mishmash of drives in a mirrored LVM setup. While it works well for having mixed drives, I’ll warn I haven’t had a drive failure yet so I’m not sure the difficulty of replacing a drive.
My phone uses the same flow with RoundSync (rclone + GUI).
Git repos are backed up in git.
Logs aren’t backed up. I just persist them for debugging and don’t want them lost after every reboot.
Caches/Games are persisted but not backed up. Nixos uses symlinks and BTRFS to be immutable. That paradigm doesn’t work well for this case. The one exception is a couple game folders are part of my personal files. WoW plugin folder, EvE online layouts, etc.
I used to use Dropbox (with rclone to encrypt). It was $20/mo for 2Tb. It is cheaper on paper. I don’t backup nearly that much. Backblaze started at $1/mo for what I use. I’m now up to $2/mo. It will be a few years before I need to clean up my backups for cost reasons.
The local server is a PC in a case with 8 drive bays plus some NVME drives for fast storage. It has a couple older drives and for the last couple years I typically buy a pair of drives on sale (black Friday, prime day, etc). I have a little over 30TB mirrored, so slightly over 60TB in total. NVME is not counted in that. One NVME is for the system, the others are a caching layer (monero node) or temp storage (transcoding as it also my media server).
I like the case, but if I were to do it again, I’d probably get a rack mountable case.
For the networking I found some repos with Nix and Gluetun (OCI containers). I don’t see them in my bookmarks, so it was probably a day project when I set up and didn’t keep the references.
That part is still in docker / podman. So any docker network guide just needs to be translated to nix.
Best resource I’ve found is searching GitHub.
My setup closely follows https://github.com/Misterio77/nix-config.
For servarr I just translated someone else’s docker compose setup to nix. There are some ready made nix ones you can look at like https://github.com/rasmus-kirk/nixarr/tree/main/nixarr.
The complex networking I just picked up over time once I knew my way around a little bit.
GitHub is your best resource. lang:nix search terms.
I wouldn’t run NixOS in a container. With native nix containers I’m pretty sure they share the store. For docker I’d use images built with nix (doesn’t run nix itself) or pull from docker hub.
OS: NixOS (high learning curve but its been worth it). Nix (the config language) is a functional programming language, so it can be difficult to grok. Documentation is shit as its evolved while maintaining backwards compatibility. If you use the new stuff (Nix Flakes) you have to figure what’s old and likely not applicable (channels or w/e).
BYOD: Just using LVM. All volumes are mirrored across several drives of different sizes. Some HDD volumes have an SSD cache layer on top (e.g., monero node). Some are just on an SSD (e.g., main system). No drive failures yet so can’t speak to how complex restoring is. All managed through NixOS with https://github.com/nix-community/disko.
I run stuff on a mix of OCI containers (podman or docker, default is podman which is what I use) and native NixOS containers which use systemd-nspawn.
The OS itself I don’t back up outside of mirroring. I run an immutable OS (every reboot is like a fresh install). I can redeploy from git so no need to backup. I have some persistent BTRFS volumes mounted where logs, caches, and state go. Don’t backup, but I swap the volume every boot and keep the last 30 days of volumes or a min of at least 10 for debugging.
I just use rclone for backups with some bash scripts. Devices back up to home lab which backs up to cloud (encrypted with my keys) all using rclone (RoundSync for phone).
Runs Arrs, Jellyfin, Monero node, Tor entry node, wireguard VPN (to get into network from remote), I2C, Mullvad VPN (default), Proton VPN (torrents with port forwarding use this), DNS (forced over VPN using DoT), PiHole in front of that, three of my WiFi vlans route through either Mulvad, I2C, or Tor. I’ll use TailsOS for anything sensitive. WiFi is just to get to I2C or Onion sites where I’m not worried about my device possibly leaking identity.
Its pretty low level. Everything is configured in NixOS. No GUIs. If its not configured in nix its wiped next reboot since the OS is immutable. All tracked in git including secrets using SOPS. Every device has its own master key setup on first install. I have a personal master key should I need to reinstall which is tracked outside of git in a password manager.
Took a solid month to get the initial setup done while learning NixOS. I had a very specific setup of LVM > LUKS encryption /w Secure Boot and Hardware Key > BTRFS. Overkill on security but I geek out on that stuff. Been stable but still tinkering with it a year later.
I’ve been screaming its just wage theft. My city provides tax breaks for occupancy (employees prop up the local economy buying lunch). They are making me pay for gas, time, and car maintenance (and lunch but fuck them, I’ll just not eat) for this tax break which goes to C-level bonuses/shareholders. Its just another way of skimming off the top of employee wages.
We worked fully remote for nearly 2 years and the hybrid policy just keeps getting worse and worse. Coupled with quarterly riffs, I also suspect this is to avoid severance pay/unemployment while accelerating the down sizing. Yet our CEO bonus keeps going up and up despite our stock plummeting since the end of COVID lock downs.
I have a few of those. During the reddit exodus when the API changes went live, I had to register on a few servers to finally get a working and stable account. Issues with email or setting up captchas to prevent bots resulted in a few of my accounts not going live until a week+ later on some instances.
https://feddit.org/post/3143093
Download 5 seasons of some show from multiple sources or some artist’s entire discography, and want to normalize all the file names? It is way easier in the terminal.
I’ll check this out, but I use https://github.com/stevearc/oil.nvim for such tasks as I have nvim’s full suite of editor commands to rename all the files way faster than I could in a GUI. I’m sure there are GUI apps to perform a similar task, but I already know how to use nvim.
Won’t auto update but you could add the upgrade command to a login script or something.
Won’t lie, nix has a high learning curve to get the most out of it, but installing a single app is pretty simple.
Most startups I’ve applied to are Linux friendly.
I currently work for a fortune 100 and managed to get a Linux machine purchased as a “lab” machine.
I’m fully in control. IT doesn’t even know it exists. I’m not allowed on the corporate network, but I managed to get some internal corporate access through another department’s lab network (IT sanctioned) that has a VPN with a few routes to things like ticketing, time cards, and our internal wiki. Most of the stuff I need to do my job is in AWS and we are allowed to add home IPs to the security groups.
IT still gives me a MacBook. I use it like once every 6 months.
nixos-unstable is the only thing I will use currently.
I’m running bleeding edge stuff like the latest kernel, Hyprland nightly, my own “shell” built from Gnome components and lots of custom stuff using GJS (Gnome JavaScript).
If you get one, and you are free to do whatever on it, encrypt your drives like your job depends on it. I have a memorized passphrase, pin protected hardware key, and a key in TPM. No biometrics.
As far as other nice things to have:
I work in software dev as FYI. For the few issues I have, my team has more issues getting stuff working consistently on macOS for our project. I used that as a justification when requesting the laptop: my dev environment should closely match our runtime environment. Most of that is moot now since we use Nix flakes in our repos for local dev envs.
Yeah I don’t want locally deleted media (to free up space) to sync those deletions to my remote.
My crypted remotes wrap a B2 Backblaze one which doesn’t delete, just hides. Periodically I go clean it up.
You are correct, fixed!
https://github.com/newhinton/Round-Sync. Not in any app store and have to download and install from GitHub.
It is an Android wrapper around rsync rclone.
Setup a remote, setup tasks, and setup triggers. Mine syncs every night. It supports encrypting with your own keys. Large number of remotes supported from self-hosted to cloud.
Immutable Nixos. My entire server deployment from partitioning to config is stored in git on all my machines.
Every time I boot all runtime changes are “wiped”, which is really just BTRFS subvolume swapping.
Persistence is possible, but I’m forced to deal with it otherwise it will get wiped on boot.
I use LVM for mirrored volumes for local redundancy.
My persisted volumes are backed up automatically to B2 Backblaze using rclone. I don’t backup everything. Stuff I can download again are skipped for example. I don’t have anything currently that requires putting a process in “maint mode” like a database getting corrupt if I backup while its being written to. When I did, I’d either script gracefully shutting down the process or use any export functionality if the process supported it.
No. Its all text file config. You wouldn’t use live CD mode. You define your own how you want it to work.
Its a steep learning curve so if looking for off the shelf solutions, don’t use nix. If you need something custom but through a single config paradigm, nix is awesome.