Help moving from VMware to KVM

Discussion in 'Business & Enterprise Computing' started by 3t3rna1, Jan 21, 2011.

  1. 3t3rna1

    3t3rna1 Member

    Joined:
    Dec 24, 2001
    Messages:
    1,452
    Location:
    Perth
    Hi all,

    This morning my VMware cluster browned itself yet again, 3rd time in 2 weeks. I need to start getting a good nights sleep before I go postal so I'm changing hypervisor to KVM. I'm a bit sick of being stuck behind an unreponsive GUI when I'm rudely awoken by abusive phone calls so it has to change.

    The problem with VMware at the moment is a vendor driver. Issue is known but not being fixed and I can't fix it or find a workaround.

    At the moment I have 4 VMware hosts running -
    2x Xeon 5620
    48GB DDR3
    Single Port QDR Infiniband
    2x onboard intel nics
    USB boot

    Shared storage is all on a single OpenSolaris server running SRP (SCSI over RDMA). I know there should be redundancy here but the budget hasn't yet allowed for it.

    The plan so far is -
    • CentOS 5
    • Latest KVM
    • Multiple shared LUNs over SRP running GFS
    • Virt manager and command line for management

    Basically going to start with a fresh 2008R2 server on vmware and attempt to convert it across. Internal VM's will get converted next followed by client VM's.

    Getting it setup correctly shouldn't be a problem, I'm a bit worried about the conversion of VMDK's and virt-io support, seems to be a bit of a process. I have 52 VM's in total and all but 3 are Windows 2008.

    Does anyone have any advice or suggestions?

    Forget to mention I need to get this all done in under a week so I can move the servers to a new DC. Hooray :thumbdn:
     
  2. Gecko

    Gecko Member

    Joined:
    Jul 3, 2004
    Messages:
    2,715
    Location:
    Sydney
    I haven't tried this myself, but I'd give ShadowProtect a go to take an image of the VM from VMware, then a hardware-independent restore (works quite well when I've needed it) to restore it onto a KVM machine.
     
  3. Zer0

    Zer0 Member

    Joined:
    Jun 28, 2001
    Messages:
    907
    Location:
    Singapore/Brisbane
    how does KVM go with Windows guests? Do they have accelerated drivers for the guest similar to vmware and citrix etc?

    We're an all Citrix Xenserver shop and it has been rock solid using similar specs to you but we run storage over 10gbe rather than infiniband.

    Also looking at KVM as an alternative at the moment as it seems to have come a long way lately. Especially with Ubuntu Server having virtual machine host as a standard role now and management through the convirt2 web interface.
     
  4. OP
    OP
    3t3rna1

    3t3rna1 Member

    Joined:
    Dec 24, 2001
    Messages:
    1,452
    Location:
    Perth
    Bit of an update,

    First host is up and running with some caveats. So far I've done -
    • CentOS 5.5 install
    • Setup KVM
    • Setup OFED and SRP
    • Setup clustering (for a single node) and GFS2

    To do list looks like -
    • Update libvirt on Centos to match ubuntu desktop being used for management (virt-manager)
    • Fix issue with cluster services stopping before unmounting GFS2
    • Add second node and setup fencing using IPMI
    • Enable KSM (memory deduplication)

    The learning curve from Ubuntu to CentOS was a bit steep but I managed to get it sorted in the end. I even wrote a boot/shutdown script so I'm a little proud.

    I got one guest across using ShadowProtect (Cheers for the tip Gecko), under VMware I used a boot cd and dumped it to a Samba share on the KVM server. The booted up the CD in KVM and restored at a painfully slow rate (3MB/s). I think the network drivers were at fault due to using the Realtek driver, e1000 might be better.

    I wanted to try a VMDK conversion but vSphere throws an error half way through a transfer... awesome :thumbup:

    They have virt-io for paravirtualized network and disk IO. I didn't get a chance to update my test server but I'll do so tonight, I've read it makes a big difference.

    IIRC there is also a baloon driver similar to VMware's for memory management.
     
  5. joyufat

    joyufat Member

    Joined:
    Jun 27, 2001
    Messages:
    1,014
    Location:
    Moral High Ground
    What kernel are you running / do you plan to run on your Centos hosts? Also, I'm curious why you went with Centos at all, given your familiarity with Ubuntu.
     
  6. OP
    OP
    3t3rna1

    3t3rna1 Member

    Joined:
    Dec 24, 2001
    Messages:
    1,452
    Location:
    Perth
    A couple of reasons -
    Mellanox have an OFED stack for RHEL, same kernel as Centos
    It's more 'enterprisy' than Ubuntu
    It's the closest thing to RHEL - I'll hopefully get a subscription out of management soon. It will make the transition easier
    GFS is well supported under Centos - lots of good how-to's
    It's possible to get RedHat training for other staff
    I wanted a challenge

    Kernel is 2.6.18-194.32.1.el5
     
  7. ECHO

    ECHO Member

    Joined:
    Jun 17, 2002
    Messages:
    638
    Location:
    Canberra
    Yeah it works well. You can purchase an "IT Edition" (portable) for short periods of a time (1 week, 2 weeks, etc).
     
  8. Rampage101

    Rampage101 Member

    Joined:
    Jun 27, 2001
    Messages:
    2,034
    Location:
    Country NSW
    Just curious, but why the need/want to move from free to paid with RHEL? Do you need the support?
     
  9. OP
    OP
    3t3rna1

    3t3rna1 Member

    Joined:
    Dec 24, 2001
    Messages:
    1,452
    Location:
    Perth
    Well we don't 'have' to move, but it would be nice. I'm only 23 and the most senior technical person at my workplace so it would be great for someone to support me instead of me supporting them. Plus, when they drive me insane and I quit all they need to do is hire a RHEL guy, a Cisco guy and a web dev and they have me replaced.

    Most likely outcome is they don't see the point of paying for something they already get for free.
     
  10. Glide

    Glide Member

    Joined:
    Aug 22, 2002
    Messages:
    1,151
    Location:
    Was: Sydney Now: USA
    does KVM support .ova files? - you could use the vmware gui to export the vm's

    I thought all vendors supported this method now, could be wrong.
     
  11. passivekid

    passivekid Member

    Joined:
    Sep 10, 2003
    Messages:
    332
    Location:
    Perth, WA
    Very interested in your progress with this.

    Was looking at RHEV but can't justify $.
     
  12. OP
    OP
    3t3rna1

    3t3rna1 Member

    Joined:
    Dec 24, 2001
    Messages:
    1,452
    Location:
    Perth
    Got the second node up with IPMI fencing last night. Some dumbass (me) messed up the cluster.conf which was causing the nodes to constantly reboot each other. Fixed it eventually, live migration is working perfectly :thumbup:

    Only real problem I have at the moment is KSM (memory dedup) won't work. This kills my consolidation ratio a fair bit since everything is basically the same OS. CentOS 5.5 is using an older KSM module which can't even report shared pages, RHEL 6 has the newer version so I might just need to wait until CentOS 6 is released.

    I believe you can import VMDK's and OVF's, pretty sure you can even boot up a VMDK directly. It's just a bit of a process to download, convert, convert again, boot up, remove drivers, install virtio driver etc.

    Current proccess is -
    • Backup using shadowprotect trial or live cd to a SMB share (~15min)
    • Boot VM with emulated drivers and restore (~30min)
    • Reboot with Virtio NIC and temp Virtio HDD (~10min)
    • Reboot with all Virtio drivers (~5min)
    Seems to be working fine for now. I did read on the Proxmox website that there is a live cd which pipe the raw disk over SSH and create the VM for you.
     
  13. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    45,105
    Location:
    Brisbane
    KVM uses the open source libvirt and virt-io stack. You can download accelerated drivers for these here:
    http://www.linux-kvm.org/page/WindowsGuestDrivers/Download_Drivers

    If you pay for RHEL/RHEV, they will provide signed binary drivers for you. They're identical to the ones I link to above, if perhaps a few versions older due to their extra enterprise testing regime.

    I'm 100% against the idea of running VMs as files on a file system. The extra overhead is not worth it, in my opinion.

    I always set up my storage system to provide a LUN per physical disk I want to present to my VMs. I can always increase that later by presenting another LUN and using LVM or equivalent disk spanning systems to span the disks (most SANs will support dynamic LUN resizing without data loss too).

    The standard argument I hear against that is "it's too much effort". I personally don't see the effort. You create the LUN, present it as the same LUN ID to all hosts in the cluster, and you're done.

    Linux (and thus KVM) has a very neat way of identifying disks. Under /dev/disk you'll see three directories:

    by-id
    by-path
    bu-uuid

    For FC and iSCSI, I use the /dev/disk/by-id/blah reference. No matter which host the VM is on, that identifier will always stay the same (again, if you ensure the LUN number is identical on all hosts, which you should anyway for sanity's sake). So dealing with direct storage/LUN mappings and clustered virtualisation hosts (and all the usual migration stuff) is quite simple (no more difficult than pointing to a file on a file system).

    Dealing with GFS fencing and STONITH is a pain in the arse. The overhead of clustered file systems is also high, and then on top of that you're adding file system and locking overhead of having your VM's file system inside a virtual disk inside a file inside a file system inside a cluster. Watch your I/O performance vanish!

    I know it's the "VMWare way" to use VMFS, but it's not correct or intelligent. Even VMWare themselves gave in when I argued the point with one of their top sales guys and accompanying tech when they presented to our company. They could not truthfully argue that VMFS, no matter how well written they think it is, is better for high I/O systems than a direct path to storage.

    I'm in charge of both our company's KVM cluster, and our HP-UX Integrity Virtual Machine cluster. Both of these use straight-to-storage setups (KVM via iSCSI, HP-UX via FC), and both blitz the I/O numbers of both Linux and Windows hosts running off VMWare/VMFS attached to our fastest SAN.

    I've done similar tests on RHEL/GFS2, and given up after the I/O performance didn't reach my expectations. Likewise I've used GlusterFS (far simpler than GFS, and much faster), but it too didn't cut the mustard for the performance I required.


    KVM doesn't support this directly, but there are tools to convert various containers to the native qcow format used by KVM/qemu (and they're all free).

    But again, I advise against this. When migrating from VMWare -> KVM, I use a LiveCD to boot each VM, and run dd over an SSH tunnel.

    dd if=/dev/sda | bzip2 | ssh root@otherserver "cat | bunzip2 > /dev/sda"

    Once done, fire up the new server on the KVM box, install the virt-io drivers, and go nuts.


    I agree with the choice of CentOS. It's a trivial migration to RHEL from there should you want support in the future. Plus the support scheme is good.

    My only advice is to got for CentOS 6**. RHEL6 is the base for the new RHEV2.2.

    Don't worry so much about the kernel revision. RedHat own KVM, and are very adept at backporting all the enterprise features into their production kernels.

    With that said, things like memory ballooning, KSM (Kernel Shared Memory) and a few other features do require CentOS6**. So again, I recommend that version.

    ** I realise CentOS6 is not out yet, but it is due soon (RHEL6 is in out as of November 2010). It's easy enough to upgrade to it - migrate your VMs off a host, upgrade the host via a clean install and move the VMs back. Continue until the entire cluster is upgraded, and enjoy the new features.
     
    Last edited: Jan 28, 2011
  14. OP
    OP
    3t3rna1

    3t3rna1 Member

    Joined:
    Dec 24, 2001
    Messages:
    1,452
    Location:
    Perth
    I haven't really thought about this before... but its a great idea :thumbup:

    Most are low IO terminal servers so I can't really justify a raw disk per VM. I have a few medium IO file and mail servers which I can justify giving thier own LUN, but the rest is performing just fine on GFS. I'm not running databases that need 10,000 IOPS per server... these terminals barely hit 150 IOPS each in peak times.

    GFS is running on shared 1GB ethernet at the moment, I'm going to switch it over to IPoIB which will give it ~15G of bandwidth.... should be plenty of room for overhead. Fencing was also very easy with IPMI.

    The plan is to switch to Gluster when I'm given the monies for another storage server. I can take advantage of the 40G RDMA to make sure performance is up to scratch and to make it scale like a mofo.

    Really appeciate your input Elvis, experience is never overated
     
  15. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    45,105
    Location:
    Brisbane
    I think you'll find with GFS an upper limit of performance for the file system.

    GlusterFS can scale a lot higher, but only if you stripe blocks across hosts. You can use a combination of stripe+mirror to achieve redundancy and speed. I'm using GlusterFS for a site that does a lot of shared storage for rendered frames (average 300MB per frame, hundreds of thousands of frames, dozens of render nodes and user workstations), which it's great for. I've considered it for VM storage, but again I really don't like the file system overhead problem.

    My motto is stick as close to bare metal as possible. If I can present raw storage and not add substantial management overhead, I'll stick with it.

    [edit] And *YAY* for a VMWare->KVM migration thread. You won't regret it. Wait until you have to migrate a few hundred guests, and it goes down from hours to minutes. I've had VMWare guys tell me I've rigged my KVM system to give false benchmarks when they see how fast it is. I just stand there laughing. :D
     
    Last edited: Jan 28, 2011
  16. Primüs

    Primüs Member

    Joined:
    Apr 1, 2003
    Messages:
    3,458
    Location:
    CFS
    Such the activist :lol: either way you've convinced me to ditch VB or VMWare or anything, and use KVM on CentOS (which i'll wait for 6 now). So congrats :p.
     
  17. OP
    OP
    3t3rna1

    3t3rna1 Member

    Joined:
    Dec 24, 2001
    Messages:
    1,452
    Location:
    Perth
    More of an update,

    Moved everything to the new datacentre on Saturday, finally out of that shitty, half airconditioned, carpetted, no-UPS'd server room. Everything came up perfectly... except VMware :mad:

    Something about those OFED drivers in VMware was just screwy, it would boot 1 VM and drop everything when booting the second. I changed it all over to iSCSI (thank you COMSTAR) so it's limping along now. KVM is still using SRP/Infiniband and never had a problem.

    I managed to figure out the KSM module, I think. You have to run the command every time you want to dedup memory so I'll probably make a cron job to do it every 10 minutes. It takes a whole core whilst it does the scan and I don't there there is a way to change that. Results are impressive however, memory usage without KSM is 28.5GB whilst afterwards its 20GB. Not a bad effort :thumbup:
     
  18. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    45,105
    Location:
    Brisbane
    Knock yourself up a little script that checks:

    1) Is dedup already running?
    2) Is the overall system load of the host at a stupidly unreasonable amount (left up to you to decide what that is based on your requirements)?

    If not, then go go dedup.

    Converting the world, one person at a time. :)

    PS: Hi Red Hat staff reading this (I know you do, because you email me at work and tell me :)). I love you guys. I just wish you had RHEV consulting jobs up in Queensland!
     
    Last edited: Jan 31, 2011
  19. wellonchompy

    wellonchompy Member

    Joined:
    May 29, 2002
    Messages:
    747
    Location:
    Sydney
    If you can't justify 1 LUN per guest, there is a much better method than using a clustered filesystem: LVM logical volumes. You present 1 LUN to all guests, and then set it up in CLVM. You then slice it up into LVs, and use each one as a "raw" disk in libvirt. If you do get your hands on a RHEL subscription (or can wait until CentOS recompiles it), libvirt in RHEL6 will manage logical volumes for you. You give it a volume group, and it manages the LVs much the same way it would use files, except with minimal overhead. Guest images residing on a clustered filesystem is one of VMware's weak points, and KVM makes it trivial to do better.
     
  20. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    45,105
    Location:
    Brisbane
    CLVM still requires the same sort of setup as GFS, with fencing and whatnot.

    It's definitely better from a performance point of view. But it does have the management overhead of GFS and similar clustered systems.

    I actually have one system I manage that is "beige box" storage (8x 1.5TB disks in software RAID6), sliced up by LVM, and each LV presented via iSCSI as a LUN back to a group of hosts.

    Performance is lightning quick, considering the "low end" nature of the whole setup.
     

Share This Page

Advertisement: