Let's talk about XenServer

[ewok85]

I've been rather sad at the lack of XenServer stuff on here, so I thought I'd create my own thread.

I've been stuck at work with not being limited in my choices for budgeting - I'd love a better version of VMware but instead to get the features I want (mainly live VM migration) I've turned to XenServer.

To get things started I built a storage box just using FreeNAS - quick and simple to do, just running off a USB stick. Got about a dozen 1TB drives mirrored in one big zpool, with two large zvols. Box has 6 gigabit connections to a switch dedicated to iSCSI usage.

Next step was to install XenServer - installing from USB is simple and quick enough. Then installed XenCenter on a Windows machine. Interface is nice and simple, seems easy enough to use without having to read endless documentation.

First hiccup was that I cannot use the nifty VMware->XenServer conversion tool, as I have a "free" edition of ESXi. Bummer Alternative is to either use their P2V tool, copy the VMDK files, or "Export" to OVF.

Spent my Saturday creating a few Windows VM's, copying files around and testing general performance - in general very good, no lag while using remote desktop and moving multiple files at 90mb/s.

Sunday was VM migration day - moved some Linux VM's to XenServer (voip, web test, web production, nagios/rt/sugarcrm) - took a little while but everything went smoothly. Once the old ESXi host was empty I shut it down and installed XenServer so I would have 2 hosts.

Then came my next problem - the old ESXi box only had dual onboard NICs - crack it open and its PCIX... off to ebay for some cards. Once they arrive (probably Wednesday) I can migrate the VMs, shut down the box, and install the card. I'll probably put one of my spare Intel dual-gig cards in the other box as well.

The real fun begins when I migrate my two best ESXi hosts which run mainly Windows guests - the domain controllers will be re-created from scratch, mainly because they are still 2003, but I'm not sure how to handle our SQL server... I'm hoping the P2V works!


Anyone else out there using XenServer? What do you like, what do you dislike? Any gotchas or issues I need to watch out for?

[NSanity]

Why can't you use Hyper-V out of curiosity? Linux Guest support problems?

[ewok85]

Originally Posted by NSanity

Why can't you use Hyper-V out of curiosity? Linux Guest support problems?

No reason why, it just wasn't first on my list of things to try. I'm keen to have a look at that as well.

[bsbozzy]

I manage a few XenServer boxes.

Pros:
-Central Management - Free, no Vcenter or SCVMM needed.
-Free live migration (you pay for auto migration)
-It is simple to use
-CLI access with most standard linux tools - good for troubleshooting
-General Performance is as good if not better than the others

Cons:
-Lack of multipath for Dell Equallogic SAN (i believe the HP lefthand stuff is the same, and a few others), ESXi does it out of the box. I have to rely on teamed nics for failover, and then use in guest ISCSi for performance (which is how I would do it anyway, but I would like multipathing for OS access). What I would like is proper multipathing support for more vendors (or just fix the stupidity and allow 3rd party driver installs, it is centos ffs...)
-Takes a version update to include support for newer OS
-Last few version updates have had issues (patches released fixed them)
-Alot of stuff has to be done via CLI and cannot be done in the gui, e.g. 4 -port nic bonds, LACP bonds are unsupported with version prior to 6.1

I could go on, dont have the time....


My customer who has all these boxes might yet be migrated to Win 2012 with Hyper V - All MS shop, will save them thousands on their SA renewals.

[weznagwama]

Used XenServer once (2 x host connected to openfiler NAS) and had continual issues with nic-bonding.

The free features were nice (especially the "increase xenapp speed" tickbox LOL) but for stability, couldnt go past ESXi. Just to many stability issues with it.

That was 5.6(?), so has probably increased alot since then.

Originally Posted by bsbozzy

I manage a few XenServer boxes.
...migrated to Win 2012 with Hyper V - All MS shop, will save them thousands on their SA renewals.

Sorry for off-topic, can you elaborate on that?

[NSanity]

Originally Posted by weznagwama

Sorry for off-topic, can you elaborate on that?

2012 standard includes 2 guest licences for 2 sockets. 2012 Datacenter includes unlimited guest licenses for 2 sockets.

[Herballizard]

Xenserver here as well, using solaris express + napp-it

NFS and CIFS

Running openxenmanger via macports on osx

[mjunek]

We are using ESXi internally for alot of things, however the new licensing model introduced with V5 and removed with V5.1 really threw me off. So sticking with ESXi 4.1 for the moment.

Currently looking and testing XenServer, mostly due to cost.

Looking at purchasing 4 servers & 2x FC SANs, a pair+san for Melb and a pair+san for Sydney. Licensing on VMware for the lot would cost me around 3k/server + 8k for vCenter server- = 20k software, for 29k worth of kit! So that's a big increase.
XenServer, on the otherhand, is just under 1k per box, and includes the feature set I need. Can't argue with the cost.

As I'm the only one in the company who really looks after our VM environment, switching to another product wouldnt be an issue from a staff perspective, as I will need to train them on either one.

The management interface feels very "Linuxy" whereas ESXi feels very "Windowsy" - by that, there are lots of little options in Xen, the interface doesn't feel as clean as VMware. All tasks can be performed without issue however.

I think for my purposes it's neither here nor there. I like the idea of ext3 backend filesystem, would make failure recovery alot easier than VMFS.

That's my ramble of thoughts for the moment.

[username_taken]

Originally Posted by mjunek

Looking at purchasing 4 servers & 2x FC SANs, a pair+san for Melb and a pair+san for Sydney. Licensing on VMware for the lot would cost me around 3k/server + 8k for vCenter server- = 20k software, for 29k worth of kit! So that's a big increase.
XenServer, on the otherhand, is just under 1k per box, and includes the feature set I need. Can't argue with the cost.

what's your reason for going FC? In my experience with good 10gbps switching gear there's no reason you couldn't run the whole stack on iSCSI or NFS and skip the hardware/management overhead of fiber channel... and this is coming from a guy who used to be all about the FC.

[mjunek]

Originally Posted by username_taken

what's your reason for going FC? In my experience with good 10gbps switching gear there's no reason you couldn't run the whole stack on iSCSI or NFS and skip the hardware/management overhead of fiber channel... and this is coming from a guy who used to be all about the FC.

Mainly because I'd have to put in the 10Gb infrastructure to support it. So the hardware / management side still comes back to me, just in a different form.

[pH@tTm@N]

MS 2012 Hyper-V looks like buckets of win. Keen to take a look at it in anger and save heaps compares to vmware.

[albeeny]

Has anyone had good results running XenServer over 10Gbps iSCSI? I'd be interested to here your story.

We've been running XenServer in production for a few years now (5.5,5.6 and 6.0). 2 pools with 4 nodes each. SAN devices are lowend Dell MD3000i and MD3200i. 4x 1GB iSCSI multipath'ed. Can't complain too much. Very few issues. Software iSCSI Multipath set-ups can be tedious, but once it's in place it's very stable. Our IO loads aren't massive, mostly Windows Application servers, DCs, Exch2003, a couple of SQL2008 nodes...
I've found the Citrix support to be very very good. The support chaps I've spoken to are from the Sydney office and they'll do everything in their power to get best results for your solution.

I like:
- The price
- The Support
- The new StorageMotion and LACP bonding for VM traffic in the latest version 6.1
- The XenConsole is easy as, once you let go of the vSphere Client way of thinking.

I don't like:
- Hardware compatibility list (last time I looked there was bugger all SAN options on the XenServer Compatible hardware list)
- version 6.0 did terrible things to software iSCSI performance.
- lack of integration features with SANS, vmware gets all the attention here
- When you call a Server or San manufacturer (*cough* de11 *cough* 3mc) looking for support information about XenServer, they have know idea what you're talking about.

[Daemon]

Originally Posted by bsbozzy

-Lack of multipath for Dell Equallogic SAN (i believe the HP lefthand stuff is the same, and a few others), ESXi does it out of the box. I have to rely on teamed nics for failover, and then use in guest ISCSi for performance (which is how I would do it anyway, but I would like multipathing for OS access). What I would like is proper multipathing support for more vendors (or just fix the stupidity and allow 3rd party driver installs, it is centos ffs...)

Agree, the downsites to XenServer is the support for the Equallogic SAN (which is a great product) and the driver support (quick fix = don't use Broadcom).

Originally Posted by mjunek

Mainly because I'd have to put in the 10Gb infrastructure to support it. So the hardware / management side still comes back to me, just in a different form.

Do you have existing FC infrastructure? Also, don't discount plain old 1Gb just because it doesn't sound as good. As anyone in the storage world knows, latency is the critical factor. I manage an Equallogic SAN with some dedicated and Xen nodes connected via a dedicated 1Gb network. The only time we max the connection out is during backups and reboots yet the system is still very quick as all the access requests are sub 5ms (99.9% of the time). That said, the cost of 10Gb gear has dropped dramatically in the last 2 years and will continue to do so.

Hyper-V 2012 looks interesting but it's still quite costly if you're not 100% MS shop. SPLA licensing for a dual processor box will still cost roughly 3k per year to enable unlimited virtuals. Add in System Center to provide the centralised management and it's quite easy to see why Hyper-V is "free".

[bsbozzy]

Originally Posted by albeeny

Has anyone had good results running XenServer over 10Gbps iSCSI? I'd be interested to here your story.

We've been running XenServer in production for a few years now (5.5,5.6 and 6.0). 2 pools with 4 nodes each. SAN devices are lowend Dell MD3000i and MD3200i. 4x 1GB iSCSI multipath'ed. Can't complain too much. Very few issues. Software iSCSI Multipath set-ups can be tedious, but once it's in place it's very stable. Our IO loads aren't massive, mostly Windows Application servers, DCs, Exch2003, a couple of SQL2008 nodes...
I've found the Citrix support to be very very good. The support chaps I've spoken to are from the Sydney office and they'll do everything in their power to get best results for your solution.

I like:
- The price
- The Support
- The new StorageMotion and LACP bonding for VM traffic in the latest version 6.1
- The XenConsole is easy as, once you let go of the vSphere Client way of thinking.

I don't like:
- Hardware compatibility list (last time I looked there was bugger all SAN options on the XenServer Compatible hardware list)
- version 6.0 did terrible things to software iSCSI performance.
- lack of integration features with SANS, vmware gets all the attention here
- When you call a Server or San manufacturer (*cough* de11 *cough* 3mc) looking for support information about XenServer, they have know idea what you're talking about.

I installed a 10Gb Equallogic SAN this year, SSD/HDD hybrid that a Xen pool connects to. What would you like to know?

[albeeny]

Originally Posted by bsbozzy

I installed a 10Gb Equallogic SAN this year... What would you like to know?

I've read lots of forum threads where people were disappointed with the speed they were getting from 10g iscsi on Xenserver.
I'd like to know
- what switches you use
- what NICs you use
- What version of Xen
- Did you use Software iscsi for VM disk storage?
- Multipath on?
- Jumbo Frames on?
- Switch port Flow control on?
- What you used to benchmark your results
- ...Benchmarck results
- Did you align your NTFS sectors and stripes???
- did you notice a distinct lack of MB/s when writing large sequential files?

*gasp*