If you’re looking for a new daily driver, look at Fedora Silverblue. I also started on arch, and have been in nix for the last two years, and I’m planning to switch to Silverblue in the next year

For sure, love talking about my home lab! And it’s an easy way to help get a bit of content on Beehaw, albeit a bit scatterbrained haha

First, the OSI model! Yeah, this is a very “first you must invent the universe” moment, sorry (not sorry). Basically for you the important bits are layer 2 and layer 3 (though I HIGHLY recommend knowing the OSI model forwards and backwards in its entirety).

Layer 2, the datalink layer, uses MAC addresses and is how to computers on a LAN communicate directly with one an other. Layer 3, the IP layer, uses (you guessed it) IP addresses and is what allows you to communicate between networks (it’s where we get the term internet: inter-network). Layer 2 is where switching takes place, layer 3 is where routing takes place.

So, if two computers are in a single LAN (or vLAN), they will communicate through a switch only. If two computers are not in the same LAN/vLAN, they can only have connectivity through a router. It’s really important to understand a switch only sees MAC addresses, so you can’t firewall at the switch level.

If some of this is new to you, I highly recommend reading networking 101 and 201 from this site. It seemed like a good resource.

Okay, with that in mind, next you really need to understand bridges. Any *nix box can be act as a switch and/or router. By default when you install Proxmox it creates a bridge on the machine’s default network interface. This bridge interface is effectively a switch. Every VM then “plugs” into this switch, giving it access to your network’s router, and fetches its IP via DHCP. You can use ip link and brctl show to see it. Take a look at the related debian docs too.

Setting up OPN in Proxmox

So with OPNSense, you have a few ways of setting it up, but you always need at least two LANs. One which is shared by OPNSense and your edge router (the ActionTek), and the other is shared by OPNSense and the VMs running in Proxmox. I’d recommend just using two vLANs as the best solution for now. One vLAN will be for your edge router, and the other will be for your OPN router.

First I’d setup your new switch so that every Proxmox host is on a trunk port. This allows tagged traffic from the host, meaning you can specify a vLAN for each VM network interface. Make sure you get this working first, here’s a decent guide, and make sure to read the proxmox network docs too. Now everything should be on the native vLAN (special vlan 1, containing untagged traffic).

Next just add a second interface to your OPNSense with a different vLAN (maybe 100), this’ll be your OPN vLAN. Now you can move VMs to your OPN network by changing their vLAN to the OPN network’s vLAN in Proxmox. Easy peasy!

Finally you’ll want to setup proper connectivity between your routers. If you were running a cooler edge router you could setup BGP, but for now static routes will suffice. Setup a static route for your OPN network in your edge router, and point it to your OPN VM’s IP address (the IP the edge router assigned it, 192.168.1.X). Finally create the opposing route in your OPN router for the edge router network. This way you don’t need a double NAT.

Okay, really sorry if none of that made sense. I really did try to keep it concise :S

This sounds like a fun project! I recently just ripped out and redid the network segmentation on my 3-node proxmox cluster too.

Originally I had everything in a /16, but that was causing some routing problems because I actually needed to static route a /24 that was in the /16 to a VM for VPN. Anyways, I’m going to try to dig through your post and give some advice. This is all just personal opinion on how I’d set stuff up after over a decade of homelabbing/home infra, so ya know, take and leave what you want.

It sounds like you want to use one of your Proxmox nodes with a VM running OPNSense as your router? I’d highly discourage this. I know you call your setup a lab, but it’s running the *arrs and probably a streaming server; and there’s nothing worse than planning a movie night and having your networking be down. Also, it’ll make it easier to recover from a power outage or hardware failure, keep your network config much simpler, and provide physical boundaries between machines increasing security.

So, I’d say unless you’re fine with the possibility of extended outages, use dedicated hardware for network. I’m partial to pfsense’s netgate, it’s a good price and a lot of bang for your buck, and it comes from an awesome open source project. I use Unifi, though I wouldn’t necessarily recommend it due to some shady stuff the company has done/said over the last few years.

OPNSense looks neat, but the only reason I see to use it over pfsense is the integrated IDS/IPS, which is just a nice gui over suricata and a proofpoints subscription. Personally I’d run suricata in a VM and mirror WAN traffic to it via pfsense. This way a VM isn’t in your critical network path, but IDS is available and easy to manage.

Don’t forget, when you separate stuff into VLANs, it forces the traffic to go up to the router, and back down to the switch. This means any inter-VLAN traffic has a 1gb limit on it. So if you ever upgrade your servers with 10gb nics, if you setup VLANs incorrectly, you won’t get that performance. Or if you just have a lot of traffic, you’ll start getting TCP slow starts and retransmissions and it may play havoc on your network. That’s why many usually just don’t bother with VLANing, it gives you network isolation, but comes at the cost of increased routing.

As for routing, all VLANs will route between eachother automatically. As obvious as it is, just think of two VLANs as two separate physical switches plugged into the same router. By default those two switches will be able to communicate with each other through the router, but they can’t directly communicate with one an other (which would have higher throughput/bandwidth).

DMZ is interesting, in my mind it basically came from a time when networks had a hard shell and a soft interior (wrt security). I don’t DMZ because I have host level firewalls and network firewalls to do LAN segmentation. But, that isn’t to say it’s a bad idea if you’re up for it. Basically a DMZ (demilitarized zone) is a VLAN where you’d put stuff like a mail server, DNS servers, and maybe an HTTP server. Stuff you’d want to expose to the internet, as well as into your local network. The idea is that if one of those servers were compromised, you wouldn’t want it to have full access to your local network. So instead you split off a DMZ network so if a host in it gets compromised, it wouldn’t provide the attackers a good base to pivot into your local network from. I don’t expose any services to the internet, except a VPN, and that VPN definitionally needs a lot of access to my local network to be useful, so I don’t partition it off into its own DMZ. I’m not a network admin, so that’s just my interpretation of it.

As for structure, this is where I ended up:

• 10.0.0.0/24 - LAN management - Stuff like Unifi/Pfsense admin panels
• 10.0.1.0/23 - LAN - Where most of my normal stuff goes, desktops, laptops, phones, etc.
• 10.99.0.0/24 - OOB Administration - Things like IPMI and BMCs end up here
• 10.99.1.0/24 - Administration - Things like Proxmox hosts end up here
• 10.99.2.0/24 - Core network - Things like VPN, DNS, backups, basically important network services.
• 10.99.3.0/24 - Services - Things like *arr, etc. end up here. I actually run K8s via kubespray in Proxmox, so for me this is actually my metalLB service IP range.
• 10.99.100.0/24 - VPN IP pool - I static IP my VPN clients instead of masquerade, so they get an IP out of this pool, the VPN instance acts as a router, and I static route from my main router back to the VPN instance.

And I have a separate /24 for my 10gb network for Ceph.

So yeah! I don’t know if that helps at all, feel free to ask questions to clarify. If you still really want to install OPNSense in a VM I can give you some tips on that as well.