OCAU VMware Virtualisation Group!

Discussion in 'Business & Enterprise Computing' started by NIP007, Apr 16, 2008.

  1. one4spl

    one4spl Member

    Joined:
    Dec 9, 2005
    Messages:
    427
    Location:
    Jamboree Hts, Brisbane
    I'll outline the last VMware system I built for my former employer... working as a contractor now and about to quote on a similar rig, but probably with IBM servers and NetApp storage.


    - Number of users:
    ~300
    - Number of offices and location:
    Offices in SYD, MEL and an office and two data centres in BNE
    - Number of physical servers before virtualisation:
    30ish, had about 80 VMs when I left.
    - Major applications/systems:
    Mostly MS stuff, trading systems, SQL, Web, F&P, D&RMS, Citrix. Only stuff not virtual is a Domain Controller in each DC, VCB and the Backup Server attached to the autoloader
    - Reason for virtualisation:
    Several - DR, ease of management, limited rack space in hosting
    - Outline of Server infrastructure:
    Two DCs and head office connected via a triangle of dark fibre for FC and Ethernet.

    Each DC with-

    EMC CX3-20 array with three shelves of FC drives and a shelf of SATA for backup to disk, etc
    4x Dell 2950s - 2x73GB boot, 32GB RAM, dual port FC HBAs, Quad Intel 1GB NICs
    2x McData 4GB FC switches
    Cisco 3750G stacks
    links to ISP, etc, symmetrical across both DCs with a /24 from APNIC
    VRRP'd internet routers and Checkpoint on Crossbeam C6's clustered across DCs

    - Outline of Network infrastructure:

    Both DCs run 'hot' with the load split evenly across both. LUNs for the VMs nominaly hosted at each DC replicated to the other DC. VLANs trunked across both DCs with VRRP so that the switch in the remaining DC would take over as the default GW for each subnet in a failure.

    In a 'whole DC' failure all that needs to be done is mount the replica LUNs on the cluster at the other DC, fiddle with "DisallowSnapshotLUN" and boot the VMs.

    I implemented VCB 1.0 on this, so one DC has a dedicated VCB box with internal storage to do snapshot backups into Legato. VCB 1.1 would have made this a lot easier.

    Each DC also has a physical domain controller to look after basic stuff like DHCP, time sync, DNS and logons. Virtualising this stuff causes lots of chicken and egg problems that I'd rather not deal with in a disaster.

    Suffice to say that I think it’s a damn sweet setup.

    I <3 VMware, EMC Clariion, and 3750s.
     
  2. Kodaz

    Kodaz Member

    Joined:
    Apr 16, 2004
    Messages:
    938
    Location:
    Brisbane 4124
    Any reason you went for the CX3-20 with only 4 shelves? and 4 hosts?
     
  3. one4spl

    one4spl Member

    Joined:
    Dec 9, 2005
    Messages:
    427
    Location:
    Jamboree Hts, Brisbane
    Umm, good question, but I don't think the CX3-10 supported Mirrorview, which is used to replicate data from one DC to another.

    I see that it does now... *shrug* this was before the iSCSI versions existed.
     
  4. Kodaz

    Kodaz Member

    Joined:
    Apr 16, 2004
    Messages:
    938
    Location:
    Brisbane 4124
    Yeah, it looked like you may have needed it to go to another shelf for growth. Just seemed odd since the rest looked well thought out.
     
  5. lavi

    lavi Member

    Joined:
    Dec 20, 2002
    Messages:
    4,004
    Location:
    Brisbane
    i would also gone for dual HBA's not dual port HBA as well as DUAL switches for iSCSI traffice, ie. each SP has 2 ports so one connection to each switch and from the host server one HBA per switch but that's just me

    newer CX have more then 2 ports per SP so you have more bandwidth to play with
     
  6. one4spl

    one4spl Member

    Joined:
    Dec 9, 2005
    Messages:
    427
    Location:
    Jamboree Hts, Brisbane
    Dual HBAs over Dual port HBAs is a bit of a religious debate. This is all obviously IMHO-

    The PCIe bus has much more bandwidth per slot than even 4GB FC, so putting two ports in one slot isn't a bandwidth issue, and the card looks like two cards to the OS so multi-pathing works the same. (Not that ESX can do multi-pathing on an array like the CX3, anyways)

    So the only reason to drop another $1000 on another card is to get more redundancy. And I, humbly, don't think you're getting any more *pratical* redundancy at all. The unreliable bits of anything electronic are the connectors, and in particular the connectors that people can touch, and moving parts like fans and disk drives. The dual port card has two, seperate connectors, and two cables going to seperate switches.

    In this case they were 2RU servers anyways, but with dual port HBAs and one Quad NIC then they could have just as easily been 1RU 1950s with the same spec. In this case they cost about the same and we would have over-run the rack densiity of the data center if we filled the rest of the rack with 1RU boxes, so we went 2RU 2950s.

    So, if you were so worried about the reliability of individual machines- what do you plan to do about the *massive* single point of failure that is the motherboard?

    Using VMware to treat the entire servers as redundant components kind of makes worrying about one lump of silicon on one card a bit trivial to me.


    CX500s were the same. I think only the cheap stuff where you could option out of one of the SPs are non-redundant.
     
  7. Kodaz

    Kodaz Member

    Joined:
    Apr 16, 2004
    Messages:
    938
    Location:
    Brisbane 4124

    Not true, the CX3-10 is limited to 64 hosts and the 20 opens this up to 128. This may be a licensing thing but the 20 does have a few more options which is why I asked as his decision wasn't based on hosts or disks.

    http://www.emc.com/collateral/hardware/comparison/emc-clariion.htm
     
  8. lavi

    lavi Member

    Joined:
    Dec 20, 2002
    Messages:
    4,004
    Location:
    Brisbane
    esx multipathing works well with CX series, i've tried it and the only problem is the tresspassing don't revert to previous state, you have to tresspass them back to the SP you want etc.

    I had HBA's die on me before and what's $1000 anyway? peanuts! and if you're paying 1k for a QLA iSCSI HBA you're gettin ripped of a bit, i think i get them for around 800 or so

    the chance of a mainboard failing is higher then a HBA failing but that depends on the mainboard, touch wood we never had a DL380 fail yet nor a DL360. we are a one stop HP shop for the simple reason, reliability and service, we're not locked into HP or any crap like that we just found their systems to just work

    i'm yet to try changing HBA's on the DL 5 series while the server is running, it works with win2k3 but haven't tried with ESX
     
  9. Rubberband

    Rubberband Member

    Joined:
    Jun 27, 2001
    Messages:
    6,750
    Location:
    Doreen, 3754
    - Number of users: 400+ Staff
    - Number of offices and location: 1
    - Number of physical servers before virtualisation: 10
    - Major applications/systems: TS, SQL, Print, FS cluster, Web server
    - Reason for virtualisation: server consolidation, ease of management, quicker response times for setting up test environments etc, high availability, cost savings, plus it all ties into our disaster recovery plan
    - Outline of infrastructure:

    - 5 x IBM HS21's, 10GB RAM, 2 x 73GB SAS drives in RAID 1 with one spare blade.
    - Fibre connected to CX300 with DAE for total of 6TB
    - Commvault to manage backup to disk to tape
    - Dell 2850 with MD1000 DAS (6TB SATA Raid 5) for backups and management
    - Running ESX with VI 3.5
    - No single points of failure

    Bought via preferred supplier at Uni and being installed by them.

    Will be moving the 2850 and MD1000 off-site for increased DR then by 2010 a second Blade cluster for total fail-over Dr.
     
    Last edited: Apr 19, 2008
  10. Mr_PudS

    Mr_PudS Member

    Joined:
    Jul 19, 2001
    Messages:
    242
    Hi guys

    Just wonder what software do you guys use for monitoring VM Disk usage?

    I'm trying to find something that can hopefully be used to display disk usage (and any other useful info) to a webpage.

    We already run HP SIM to monitor server health. Is anyone using SIM and able to monitor disk usage on VMs with it?

    Hoping you guys have some suggestions for me.
     
  11. Rubberband

    Rubberband Member

    Joined:
    Jun 27, 2001
    Messages:
    6,750
    Location:
    Doreen, 3754
    I use IPMonitor to monitor all my servers. It uses WMI and SNMP for most server queries.
     
  12. stalin

    stalin (Taking a Break)

    Joined:
    Jun 26, 2001
    Messages:
    4,581
    Location:
    On the move
    Nagios, if it can't do it, you probably don't need to.
     
  13. yanman

    yanman Member

    Joined:
    Jun 4, 2002
    Messages:
    6,587
    Location:
    Hobart
    We've just configured a WhatsUpGold server... working really well so far. We have a few big screens in our area displaying all the important links and alerts for servers and network
     
  14. one4spl

    one4spl Member

    Joined:
    Dec 9, 2005
    Messages:
    427
    Location:
    Jamboree Hts, Brisbane
    Using Active-Active and Fixed paths on a CX isn't supported and you will get LUN thrashing where the LUN config is copied from one SP to the other as each request is made.

    http://www.vmware.com/pdf/vi3_san_guide.pdf - Page 67

    The multi-pathing on ESX is on a per-LUN basis, so all the requests for a particular LUN/Host combination go over the one path. Your only load balancing technique is to make the various LUNs use unique paths, and on a SAN like the EMC CXs or HP MSAs you can't even make that choice to any practical extent. You don't get PowerPath style multi-pathing where requests are load-balanced over multiple paths for each LUN.

    Yeah, its not much cash for the cards themselves, but if you were in the position to need to buy another rack for 2RU boxes then that’s another story.

    Right click on the server, select Enter Maintainence Mode, then Shutdown... and not sweat bullets :cool:
     
    Last edited: Apr 20, 2008
  15. yanman

    yanman Member

    Joined:
    Jun 4, 2002
    Messages:
    6,587
    Location:
    Hobart
    The question I'd like to ask is: When will the management software (vmware, SAN control, fail-over mechanisms) be powerful enough that we can just go fill up our racks with white-box non-redundant servers and sleep at night? :)

    Or is it already at this point, just that it's too much effort to DIY servers?
     
  16. one4spl

    one4spl Member

    Joined:
    Dec 9, 2005
    Messages:
    427
    Location:
    Jamboree Hts, Brisbane
    I'd bet decent money that is what ESX 4.0 will bring.... but as always theres a trade off.

    It will need to run two copies of your virtual machine, so you will need twice the resources, and the two machines will need to talk to each other at close to RAM access speeds... so you will need something like infiniband.
     
  17. lavi

    lavi Member

    Joined:
    Dec 20, 2002
    Messages:
    4,004
    Location:
    Brisbane

    with ESX and CX each path is to a LUN but on the CX you can tresspass the LUN to any SP you want hence if you have a few LUNS and pending how you set them up on/in what raid group/drives you can get a load balance effect by say tresspassing xyz LUN on SPA and abc LUN on SPB

    With ESX this way there is no "trashing", ESX can't do load balance by default like I said before, where you will get trashing is when you use PowerPath and i seen it done.
     
  18. BBB

    BBB Member

    Joined:
    Jun 27, 2001
    Messages:
    440
    Location:
    Up the road apiece...
    G'day troops!

    I'm very new to virtualisation so take my question in that context, but what are the downsides to the concept? There has to be something that would stop it dominating everything.

    BBB
     
  19. lavi

    lavi Member

    Joined:
    Dec 20, 2002
    Messages:
    4,004
    Location:
    Brisbane
    nothing so far except EMC/VMware upping the license fees per core but then there is allways xen/citrix

    also you need to know what you're doing or else down the drain goes 200k
     
  20. sponks

    sponks Member

    Joined:
    Jan 24, 2005
    Messages:
    56
    hey all. Thank god for VMware. I am a former Network Admin turned consultant mainly doing VMware work. There are some VMware User Groups for each major city if people are interested too, just google it. Getting my VCP was the smartest thing I ever did.

    I have used some very dodgy software in my time, but I rate ESX as probably the best system I have ever used. I can see that integrated hypervisor stuff is probably going to change alot of the costings of this sort of stuff, which makes it kind of dangerous in my opinion, but we'll see.
     

Share This Page

Advertisement: