You can check to see if you can enable hardware transcoding. I find the delay is usually transcoding building up a buffer and if you have a good GPU/APU in your server it’s often a lot quicker.

Pretty sure on jellyfin by default that is off. Mainly because you need to install some packages to get the devices available under linux usually.