The enormous Virtualisation help thread

I haven't, but I've been casually following its progress. It's been around a while now (introduced in FreeBSD 10.0, with 10.3 and 11.1 currently the stable releases).

By all accounts it covers off all the basic things you'd want from a virtualisation system, and supports libvirt so it can be managed by all the standard open source tools.

BSD already has its excellent jail system, which should solve any need for BSD-on-BSD. But if you're wanting to virtualise other stuff on BSD and don't have crazy requirements, it seems bhyve is perfectly capable.

 

Can anyone report on their real world experience with GPU pass through under KVM, specifically stuff that is OpenGL capable with Windows guests?

 

GPU pass-through presents the GPU to the guest OS as if it were plugged into a physical computer (there is a bit happening in between but yeah)
All 3D OpenGL etc all works as it would.

You will need a dedicated graphics card per guest/s running at the same time to do it.

 

My real world experience is: "Your host does not support GPU passthrough".

 

You probably need VT-dx or whatever.

 

Thanks for your input - much appreciated, as always. To be perfectly honest, I could get away with running jails for 90% of the tasks I want to perform, but it would be nice to have the capability to spin up Linux VM's for whatever reason. I caught myself spinning up a FreeBSD VM on Bhyve for some tests the other day, which was a bit of a "WTF am I doing this for?" moment

I reckon I need to spend some more time with FreeBSD again before I can make a more informed decision. It was my preference over Linux for many years but I decided to move to CentOS and SL for a few different reasons which aren't so important these days, and aside from pfSense and FreeNAS I'm a bit out of the loop. I didn't even know about Bhyve until recently - I think the last FreeBSD installation I used was 6.4!

 

Nah, I need a host which doesn't have a discrete gpu.

 

Eh? Descrete gpus are exactly what you need.

You might have to run the host headless if you only have one though. Makes it harder.

If you have an igpu and descrete, that's the easiest. Run host on igpu, pass the hardware through. I got it working as a test with ovirt without even reinstalling the windows guest (ran directly from the hdd) .

Booted, ran. Then I broke the host config and couldn't be bothered any more.

If you do what I did, be prepared to fixmbr. That was annoying with only a shitty netbook to dl the iso.

 

From what I've read, nVidia optimus chips, like my 920M, don't work well with passthrough, not least of all because of drivers.

There's also the minor detail that the machine I would do it on, is one on which I don't want to exclude the "power" of the nVidia chip, as it gets used regularly.

 

Lappy?

Oh dear. Yeah, it'll be easier on a PC.

 

Yeah.

Eventually, if I ever end up moving back to Australia, or if I decide to get more stuff shipped over to PNG, I will likely get this happening on a dedicated rig. Unless, of course, they make a major breakthrough in passing through the GPU without having to blacklist it first.

 

So, whatever issue I had with the performance of VM's under Bhyve hasn't reared its head since I first noticed it. I don't know if the problem was disk IO, network IO, or what. Maybe I just hadn't allocated enough memory to the VM for the tasks it had to do when first set up. I haven't investigated too closely because the problem hasn't repeated.

Anyway, it looks like FreeNAS 11.1, which should be released late this month or early next month, will include the ability to attach a certain network interface to a VM. In my case, I want some VM's to have a tap interface for internal LAN use, and others to have a tap interface based on another interface so the VM's can have public IP's. It's doable now, bit awkward to set up. Not that I'm scared of the command line, but FreeNAS configuration works best from the GUI...

 

Hey, I'm trying to update ESXi 6.0.0 to 6.0 Upgrade 2 from CLI but when I attempt it I get the following error:

Failed to remediate the host: (None, "Failed to save Bootcfg to file /altbootbank/boot.cfg: [Errno 30] Read-only file system: '/altbootbank/boot.cfg.tmp'")

I have ESXi currently running from a USB, is this why its complaining about the read-only file system? I'm fairly new to upgrading ESXi so any assistance would be greatly appreciated.

 

Nevermind I sorted it, ended up grabbing the HP iso and upgraded.

 

I've noticed some interesting behaviour with Bhyve. I'm running Scientific Linux 7 on FreeNAS/Bhyve and also on CentOS/KVM. The two virtualization hosts are connected to the same switch on our network.

The SL7 instances on KVM fetch DHCP leases just fine and behave as normal. On Bhyve, the SL7 instances occasionally lose their DHCP lease, leaving the VM uncontactable and with high system loads. Logging in via VNC and ifup/ifdown fixes the problem, as does configuring static addressing.

I'm not sure if it's a problem with my pfSense DHCP server, Bhyve and the Virtio network device I've assigned to the VM, or a configuration issue with SL7. But it's interesting, and something I haven't encountered before.

 

Hmm, I think I'm in the correct thread... I'm setting up pfSense on my Hyper-V host, and I want to use SR-IOV. Yes, I know pfSense works fine with abstracted virtual switches, but this is 2017 and SR-IOV has been a standard for a pretty long time now - it is more efficient, and because pfSense is getting two dedicated hardware ports for it anyway, I figure giving it the lowest level access possible is for the best.

The problem is, I can't find anything, anywhere, that comes even remotely close to discussing this, let alone implementing it - neither pfSense nor even FreeBSD. Apparently FreeBSD had SR-IOV baked in years ago, but I can't find more than off-hand references - again, nothing I can actually work with.

So does anyone have any suggestions for how to go about this? Or am I just spitting in the wind here?

 

I've been running virtualised pfSense and FreeNAS for a few years with pci passthrough of dual network (pfSense) and LSI HBA (freenas), but not on Hyper-V; I use Xen on centos. My pfSense box uses a dual NIC so I can physically separate external from internal networks through the host.

In Xen, IOV is largely useless as you can't live migrate vm's on it, and the xen netback driver for virtual bridge interface between VM's allows >10gbps throughput. I get native performance from my ZFS VM to other VM's with volumes mounted as iSCSI (windows) or nfs (linux), although it does burn up a few CPU cores on the hypervisor.

Having said which I don't know much about Hyper-V. There is Xen pci backend support for vt-d devices baked into freeBSD 10 (i.e. current versions of pfSense / FreeNAS) so it all just works, which probably isn't the case for Hyper-V?

 

Good news! The most recent FreeNAS update to 11.1 brings the ability to assign specific network devices to Bhyve VM's. I now have a couple of VM's with public IP's. Easy as to configure. pfSense takes care of firewalling and intrusion detection.

I'm very happy with Bhyve on FreeNAS as a replacement for Ovirt and KVM. It's much more simple, which doesn't work for everyone but it's exactly what I needed in my case.

All I need to do now is find a new job for the Dell T710 which was running KVM VM's... and figure out why populating the final two memory slots on the Supermicro board in the FreeNAS box cause it to fail POST.

 

I went from Freenas (~ 3 years, no issues aside from one or two random lockups of unknown cause) with jails / virtualbox + phpvirtualbox to Ubuntu 16.04 LTS on bare-metal with ZFS, docker and full KVM. Not looking back.

Sure took a lot of fiddling to get it setup but a good upskilling experience and once I got it all setup it runs itself, just like Freenas, + I have full access to both containers and VMs, using a massively supported distro with all the nerd knobs natively exposed instead of jumping through appliance restrictions. Also took the opportunity to consolidate a bunch of lab management VMs running on a NUC under ESXi.

I must admit I don't know anything about Bhyve other than that its the FreeBSD virt thing.

I don't find KVM complex at all, at least running a few lab VMs on a single node on a very popular, very stable distro (not doing anything too complex or crazy). It was a bit of a mission setting up my crazy virtual networks I'll admit (learnt subinterfaces/bridging etc. + iptables NAT the hard way, without screwing my docker virtual networks as well lol), it was much easier on vswitches+ESXi+appliance virtual router but its much more elegant (and I save on an extra VM) running everything through linux native stack.

Then again if this thing breaks I am boned LOL - entire lab, all services + NAS

re: GUI for KVM, you can do without a second linux machine via wok+kimchi, its pretty basic though, works well for me though I did CLI and/or virt-manager some stuff like adding spice console, adding virtio devices etc, its more than fine for regular start/stop etc.

https://github.com/kimchi-project/kimchi

 

Which kernel? Are you using AUFS or overlay/overlay2 for docker storage?