I’m trying to figure out how to host one myself. I’m trying to use barvarder and localai. But I am failing due to not enough knowledge and missing instructions. Any advice? did someone succeed with anything? I’d be happy to make other smaller steps at first as well. As long as I get somewhere.
If low on hw then look into petals or the kobold horde frameworks. Both share models in a p2p fashion afaik.
Petals at least, lets you create private networks, so you could host some of a model on your 24/7 server, some on your laptop CPU and the rest on your laptop GPU - as an example.
Haven’t tried tho, so good luck ;)