Understand that this is a rabbit hole where you’ll hear a million opinions. If you’re just starting out, I’d recommend keeping it simple. It’s been a while since I looked at what’s new, but I’d look at truenas scale or unraid first.
I personally use truenas core (based on freebsd) but truenas scale (based on debian) opens up more options like docker that might be useful in the future.
I recommend sticking to hardware you already own until you run into something that you can’t do. Then you’ll have a better idea of what to look for and how everything works.
I have a node 304 case with an itx board and I wish I’d have gone with something bigger with more expansion slots. Some sata ports died on me so I installed an hba card to give me more. That means I can’t add a 10g nic or a video card for hardware acceleration or…you get the idea. Ideally spend less money now and experiment, then in the future you’ll hopefully learn what suits your needs.