I’d like to self host a large language model, LLM.

I don’t mind if I need a GPU and all that, at least it will be running on my own hardware, and probably even cheaper than the $20 everyone is charging per month.

What LLMs are you self hosting? And what are you using to do it?

  • Scott@sh.itjust.worksEnglish
    21·
    1 month ago

    I got a home server with a Nvidia Tesla P4, not the most power or the most vram (8gb), but can be gotten for ~$100usd (it is a headless GPU so no video outputs)

    I’m using ollama with dolphin-mistral and recently deepseek coder