1. OCAU Merchandise now available! Check out our 20th Anniversary Mugs, Classic Logo Shirts and much more! Discussion here.
    Dismiss Notice

My VmWare nightmare, a warning ...

Discussion in 'Business & Enterprise Computing' started by Nyarghnia, Oct 14, 2010.

  1. Jimoin

    Jimoin Member

    Joined:
    Jul 26, 2002
    Messages:
    579
    Location:
    Melbourne
    IAC/Elvis and a few others here may see sizes that vmWare just isn't capable of, the reason I chimed in with my experience with a small environment is because Ny isn't any bigger and is still seeing big issues when he shouldn't be.

    Something is wrong seeing issues like that in such a small environment.
     
  2. The_Derro

    The_Derro Member

    Joined:
    Jul 15, 2001
    Messages:
    547
    Location:
    Sydney
    I just got off a call with a customer who is running a production environment with literally double the hosts and >2000 VM's.

    They're running perfectly fine, have gone from 3.5 > 4.0 and are in the process of rolling out 4.1 in some clusters.

    So yes, it CAN be done - does this help you.. nope :) But it can be achieved. :D

    Yes, they were using totally different hardware (no prizes for guessing what servers :D), using EMC V-MAX SAN. No idea who designed it all - I was just troubleshooting an issue with their Nexus 1k configuration.
     
  3. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    41,746
    Location:
    Brisbane
    If you've got any sort of workload that requires half decent I/O, regardless of size VMWare is going to shit itself.

    To rule out scalability, I've tested single host single guest setups with VMWare, HP IVM, Xen and KVM. HP IVM, Xen and KVM generally see a 1-4% degradation from bare metal disk speeds. VMWare is as bad as 40% degradation. Utterly appalling.

    Do you know what their workloads are like? Are they machines that sit idle for large portions of the day? Is it high CPU load? High I/O load?

    I certainly wouldn't call our x86 workloads huge (the real heavy lifting is done on UNIX boxes). But it's not tiny either. I've seen places that virtualise a lot of machines that sit idle for 90% of the day, which ours certainly do not.
     
    Last edited: Oct 14, 2010
  4. BADBOYBUBBY

    BADBOYBUBBY Member

    Joined:
    Dec 6, 2004
    Messages:
    947
    I went from 3.5 to 4.0 having the migration done by "some company" ley left and it ran like shit ..it was a user whinge fest with me caught in the middle

    I asked them why and they blamed my antivirus ,I asked why each server only had one CPU they said it didnt matter it only needed one

    I thought fuckit what do I have to lose ,went and threw 4 cores into one of our citrix servers and guess what it ran like a fuckin dream so I put 2 in the rest everyones happy.

    Then on the final invoice there's a cost for adding cpu's to the servers the fucking hide of some people.
     
  5. encode

    encode Member

    Joined:
    May 21, 2003
    Messages:
    494
    Location:
    Sydney
    I fit into a third category, working for a large (three letter acronym) company that has quite a number of large deployments, all working reasonably well.

    Unfortunately the largest of the environments that I've personally been involved in only run to the 16-20 hosts, 300-400 VMs sort of scale (although there are other environments more along the size you describe), but that workload includes large database servers (Oracle and Domino), SAP, massive file servers, application servers, etc. All of which generate large amounts of I/O.

    None of the environments run 4.1 yet though. I'm also not sure how well your HP kit compares to the kit in those environments, but our kit is about as high end as it gets, with large expensive SANs and associated fabric to match.

    It sounds to me like you've had a bad run of crappy consultants who take one look at your complex interdependent setup and choke. There are so many different layers that can be tuned with a large vmware environment, and it sounds like you've built a whole environment from scratch, so it's hard to pin down where the problem is.

    You have my sympathy.
     
  6. The_Derro

    The_Derro Member

    Joined:
    Jul 15, 2001
    Messages:
    547
    Location:
    Sydney
    The couple of hosts that I looked at were at ~60% CPU (dual socket westmeres Xeons @ 2.93) and ~160gb memory used - so yeah, those boxes were at pretty high utilisation, but I've got no idea if that was a 'typical' one.

    I've seen several sites that are around the sizes you are talking - granted, I have no idea what type of IO etc they are doing, but I've certainly seen things of the same scale you are talking about.

    Again, not trying to say "your setup must be crap" or anything - just saying that I have indeed seen very very large setups working well (would they run better with KVM or something - possibly :) )

    I'd suggest locking all the vendors in a room and tell them all to fight it out Thunderdome style.
     
  7. Oxley

    Oxley Member

    Joined:
    May 31, 2006
    Messages:
    484
    Location:
    Old Bar NSW
    Thats why we now steer clear of “consultants” and the like, we have been burnt so many times on critical systems, now if it can't be supported in house from the get go, its not going in.

    We have a new system going in and the developers and support are whining non stop about me asking for documentation on how to set-up and configure, rather than paying them to it.

    We ended up learning so much about our current production system (initials CC) that we have other abattoirs contacting us first for support, especially after the vendor starting asking for more money, but was quite clear that we would get the same shite service and no support.
     
    Last edited: Oct 14, 2010
  8. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    41,746
    Location:
    Brisbane
    Oh don't worry, I'm 100% certain it's crap. :)

    That would be a dream come true. The company I work for has a reputation for letting vendors walk all over them. I've never seen a bunch of people so afraid to tell a vendor to pull their finger out and earn their money.
     
  9. Lethal_Lynx

    Lethal_Lynx Member

    Joined:
    Sep 8, 2005
    Messages:
    266
    Location:
    Brisbane
    We have been running vmware for the past year and we have had no issues.

    We use Veeam for our backups and replication and we have NEVER had a problem with it. It always just works. So I wonder why you are having problems.
     
  10. CordlezToaster

    CordlezToaster Member

    Joined:
    Nov 3, 2006
    Messages:
    4,078
    Location:
    Melbourne
    Another positive Vmware experience.

    Using dell equipment.
    6 equal logic sans.
    5 esx hosts.
    10gb lan.
    750+ users.
     
  11. MrvNDMrtN

    MrvNDMrtN Member

    Joined:
    Dec 24, 2001
    Messages:
    1,358
    Location:
    SW Syd
    Our environment is not that intensive in cpu/mem but certainly disk iops is critical.

    We run about a dozen HP G6 hosts and several hundred VMs mixed with 3.5 and 4.

    Hosts are talking to the NetApp SAN via NFS.

    Our only issue is that the SAN cannot keep up unless we buy better controllers.

    All in all our company find VMware decent. We also use XEN for VDI infrastructure and no issues there either.
     
  12. roweysvn

    roweysvn Member

    Joined:
    Dec 22, 2005
    Messages:
    455
    Location:
    Gold Coast / Brisbane
    Ahh VM's! I'm supposed to be undertaking a VMware course next month, I know very little on how to look after the ones we have (Currently have 7) so it was offered that I could take a course. I'm hoping I can get some insight on how to use this stuff properly.

    Currently running 3.5.
     
  13. lavi

    lavi Member

    Joined:
    Dec 20, 2002
    Messages:
    4,004
    Location:
    Brisbane
    i would like to have a crack at sorting your virtual environment :) i mean i can fix it, i'm just too far.

    to fix your backups and stuff ... can you post the hosts file from the servers? and can you make sure the file from one host match the rest of them? for some reasons they don't get generated corectly and it breaks backups
     
  14. narkotix

    narkotix Member

    Joined:
    Oct 8, 2003
    Messages:
    518
    pay me 500k and ill fix it....i swear....

    anyway serious stuff - OP u always seem to have the worst luck....dunno y...maybe ur cursed or something?
     
  15. RavenKittie

    RavenKittie Member

    Joined:
    Dec 12, 2002
    Messages:
    1,602
    Location:
    Sydney
    Sorry to hear this Nyarghnia, I really feel for you. In fact I was a little worried it would happen to us after our virtualisation project kicked off but fortunately we are part of the VMware success camp (so far fingers crossed).

    Having said that - we are a small shop compared to most in the thread. (2 hosts with a third en route, 25 current vm guests with another 19 projected)

    Also roweysvn are you based in NSW? If so where are you doing the training course? I'm just wondering if you end up at the same place I did mine at :)

    Nyarghnia I have some questions for you, what version of vRanger are you using? Are you using FC or iSCSI storage for your vmdks? How many hosts and what specs?

    Genuinely curious, sad to hear your situation though. Hopefully the collaborative brains of OCAU can give some help. Sometimes reading your posts in this forum section I used to think you were secretly my boss (we seemed to face similar issues alot of the time)

    Also I love virtualised infrastructure except for the part where every man and his dog wants a VM :-(
     
  16. roweysvn

    roweysvn Member

    Joined:
    Dec 22, 2005
    Messages:
    455
    Location:
    Gold Coast / Brisbane
    Nah I'm living on the Gold Coast but working in Brisbane, so the course would be in Brissy! I'm hoping it all goes ahead, my boss has been 'seeking approval' for a couple of weeks now and the lady from the training place is on my back every other day wanting to lock me in.
     
  17. Kizuka

    Kizuka Member

    Joined:
    Sep 22, 2008
    Messages:
    249
    Location:
    SETI ALPHA FIVE!
    Sounds like the light at the end of your tunnel...
     
  18. OP
    OP
    Nyarghnia

    Nyarghnia (Taking a Break)

    Joined:
    Aug 5, 2008
    Messages:
    1,274
    The VMware stuff itself seems to be 'fine', but I can see that I can't expect apps to perform anywhere near as well if they are heavy lifters in terms of I/O. At present my core systems run on HP UX, I was thinking of evaluating a Linux based migration under VMWARE.

    Now i'm thinking of a Linux based migration on bare metal or just sucking up the costs and staying on trusty HP UX.

    We tried the latest versionf of VRanger Pro you can download and VEEAM that you can also download, both apps failed dismally to back up our environment.

    It turns out.. that it seems that there's a real issue with VmWare tools and interaction with Windows Server 2008 R2, my 2k3 VM's from P2V would work ok, but not my new stuff, oh my Exchange box would never back up either.
    Looks like both of these apps use the VmWare API's to snapshop, quiesse VM guest OS's for backups and there's something busted there.

    Oh yeah, if you deploy VCentre 4.1 into a Windows 2k8 R2 VM (which is fully supported btw) don't expect any snapshot based backup tool to back it up, we threw everything on the market at it, everything.

    I've cooked up a McGuiver backup solution that bloody works, i'll write it up and post it here in next few days, i've tested it and it WORKS and doesnt' need you to fuck about with Vmware API's to back things up and gives you complete granularity AND bare-metal/VM recovery options... there's one vendor out there who's praises i'm more than prepared to sign, but just let me do some more testing first:

    StorageCraft... ShadowProtect.. that's right, you can use it to restore INTO VM's and ONTO bare metal from within VMWARE, even on VM's that have never ever seen physical metal before, done it here, tested it and it bloody works, there's a few steps you need to follow and I'll post those up here soon. Compared to all these other backup tools, turns out to be cheaper and .. hey it works!! and it doesn't use VMware's API's either.. hows THAT for bang for buck??

    Use it inside your VM's, taking VSS snapshots which don't shit themselves, use ImageManager to manage the backup images and to it's replication thing, and if you need to restore a VM... you can...easily. Sure. it's got a few more steps than just right clicking on something in a store and selecting 'restore'. Hell, if your VMware envionment is a burning pile of slag due to server room fire.. no problem! If you got some grunty boxes sitting about.. you can do a Hardware Independant restore! Assuming that the hardware can handle the OS.

    I hope so too, but I just wanted to put out a message that 'look before you leap' applies today even more than it did years ago, even if that makes me look stupid doing so.

    You're lucky i'm not your boss, i'm a cranky, rude bastard these days, I'm usually swearing and cursing at vendors/suppliers and people by 8:00am and i'm seriously thinking of taking up whisky and chain-smoking.

    Oh, I just found a real doozey with HP's storage agents and LTO-4 tape drives. will post details in anohter thread, it's a shocker as it affects LTO-5's as well...

    -NyarghNia
     
    Last edited: Oct 15, 2010
  19. Skitza

    Skitza Member

    Joined:
    Jun 28, 2001
    Messages:
    3,764
    Location:
    In your street
    Have you tried Acronis ABR10 Virtual Edition or Backup Exec 2010? ABR10 snapshots 2008R2 VM's no problems whatsoever :) Hell I use it to backup the physical host as well, no issues.
     
  20. OP
    OP
    Nyarghnia

    Nyarghnia (Taking a Break)

    Joined:
    Aug 5, 2008
    Messages:
    1,274
    Hi,

    Didn't get around that that version of Acronis, but Backup Excec 2010 R2 was also unable to get a snapshot of our VCenter VM (even with the agents all installed and inegrated into VmWare).

    -NyarghNia
     

Share This Page

Advertisement: