1. OCAU Merchandise is available! Check out our 20th Anniversary Mugs, Classic Logo Shirts and much more! Discussion in this thread.
    Dismiss Notice

6Gb/s SAS Expander Cards

Discussion in 'Storage & Backup' started by snick, May 18, 2011.

  1. OP
    OP
    snick

    snick Member

    Joined:
    Sep 8, 2001
    Messages:
    114
    Getting supermicro boards in Australia is a real pain.. they are either way overpriced, or impossible to find. Unfortunately. I guess you can get one from the US, ut after shipping, it ends up being overpriced, once again.


    The only reason I use a sas expander is the amount of drives. I have 2 HBA's, with the ability to go to 3 hbas currently.. however, the limitation is the cases.. 24 drives in a norco.. that is kinda limited ;) So, for me, I use 24 drives in the norco on a sas expander.. and 6 drives on the HBA. If i want to add another 10 drive vdev to my zpool, or another 20 drives, for that matter, then all i'd need to do, is another case, another expander and I am set to go.

    With the 3gb expander.. I noticed that it really didnt have the bandwidth to handle zfs properly at all.. (20 drives in 2x 10 drive vdevs only getting 700MB/s in a scrub). replaced with a 6gb sas expander.. and 1.28GB/s in a scrub.. same data, same pool, same hdds.

    I friend of mine is running a similar setup to mine, however, only using HBA's and he does get faster than I get on a scrub..

    so yes, I agree, expanders arnt the most ideal setup for zfs, but they certainly can be used if needed
     
  2. OP
    OP
    snick

    snick Member

    Joined:
    Sep 8, 2001
    Messages:
    114
    incidently, anyone know much about dual link? :) still havent found anything of consequence about it
     
  3. cynix

    cynix Member

    Joined:
    Jun 28, 2001
    Messages:
    6
    Thanks for the explanation. I ended up getting a 9201-16i, plus the built-in SAS2008 on my X8SI6-F, so I have 24 real ports.

    My concern is, however, that the 9201-16i is only using PCI-E x8, same as the SAS2008 chip. Doesn't this mean it will only offer half the bandwidth per drive compared to the 8-port SAS2008? How should I connect the HDDs to work around this?
     
  4. goller

    goller Member

    Joined:
    Aug 27, 2003
    Messages:
    22
    Hey guys,

    I am running a CK23601 with a Supermicro USAS2 L8-e and can't seem to get any attached hard drives to display in the operating system (I have tried Windows 7, WHS 2011 and about to try Server 2008).

    Connecting the drives directly to the L8-e enables the drives in the operating systems, so the HBA card isn't the issue. I can also see attached drive on the CK23601 from within the Megaraid application however they appear as unconfigured.

    Any ideas on what might be wrong would be appreciated.

    *EDIT* Seem to have spontaneously (*read* hours or screwing around) fixed itself. I think it may have been a loose cable from the backplate, althought not the backplate being used by the drives. Hope this can help someone else in the future.
     
    Last edited: Dec 24, 2011
  5. Filthyfil

    Filthyfil Member

    Joined:
    Oct 29, 2011
    Messages:
    22
    Just out of interest. which OS did u end up using?
     
  6. goller

    goller Member

    Joined:
    Aug 27, 2003
    Messages:
    22
    Ended up with WHS 2011 but have also had it running on Server 2008 R2. Since I got it up and running it has been pretty stable, minus some problems with heat on the L8-e which were fixed with a fan.
     
  7. Clone

    Clone Member

    Joined:
    Jan 18, 2007
    Messages:
    133
    Altech (Another Disty) has them currently in stock. They also have the Universal Expander Kit . I was looking at these for a rackable storage add-on from a target server with external Mini-SAS.
     
  8. DrFrag

    DrFrag Member

    Joined:
    Aug 26, 2003
    Messages:
    1,311
    Location:
    South Australia
    Does this only impact speed, or reliability?

    I can handle a speed hit on a domestic media server, but I'm not so keen on drives dropping out.
     
  9. flain

    flain Member

    Joined:
    Oct 5, 2005
    Messages:
    3,063
    Location:
    Sydney
    I had reliability issues which in turn caused speed issues. I couldn't get WD Green drives to work properly on ZFS through a Chenbro expander. Other brand drives worked ok (hitachi was fine, samsung was fine). Out of my 20 drives I had 10 WD drives, so ended up solving my issues by getting a HBA with more ports to connect the drives directly.

    The symptons i would get was crazy amounts of hw and transport errors on the iostat -en output for any WD drive going through the expander, leading to much slower performance overall.
     
  10. DrFrag

    DrFrag Member

    Joined:
    Aug 26, 2003
    Messages:
    1,311
    Location:
    South Australia
    I'm running WD Green EADS drives through my Chenbro. Hitachi 5K3000 and Samsung F4EG too.

    But I also don't really know what I'm doing, so they could be running at half speed and I wouldn't know.
     
  11. flain

    flain Member

    Joined:
    Oct 5, 2005
    Messages:
    3,063
    Location:
    Sydney
    The EADS drives i had weren't as bad as the EARS ones, but would still log errors from time to time. The EARS ones were horrendous though, error counters would climb every minute.
     
  12. Schnooper

    Schnooper Member

    Joined:
    Apr 20, 2007
    Messages:
    796
    Interesting, I have had 7x WD20EARS running on a IBM M5015 (rebadged LSI 9260-8i) in RAID5 for ages and it appears to be running really well. No errors on the drives. I think the trick is to switch off the power management, so the drives dont power down.
     
  13. sreg0r

    sreg0r Member

    Joined:
    Jul 9, 2001
    Messages:
    1,146
    Location:
    Melbourne
    I'm running 4 x WD20EARS on ZFS through a BR10i and I haven't had any read/write errors after ~1 year.
     
  14. flain

    flain Member

    Joined:
    Oct 5, 2005
    Messages:
    3,063
    Location:
    Sydney
    Its specifically an issue of them running through a Chenbro SAS expander on ZFS - direct to SAS card had NO problems

    If you are running ZFS on solaris, do an iostat -en

    If it's all 0's then you are good.

    Code:
    
    flain@monkey:~# iostat -en
      ---- errors ---
      s/w h/w trn tot device
        0   0   0   0 c4d1
        0   0   0   0 fd0
        0   0   0   0 c11t0d0
        0   0   0   0 c11t1d0
        0   0   0   0 c11t2d0
        0   0   0   0 c11t3d0
        0   0   0   0 c5t5000CCA228C0FB76d0
        0   0   0   0 c5t5000CCA228C0EAC2d0
        0   0   0   0 c5t5000CCA228C0ED55d0
        0   0   0   0 c5t5000CCA228C09116d0
        0   0   0   0 c5t50014EE2029CBC19d0
        0   0   0   0 c5t50014EE2AEDD1D61d0
        0   0   0   0 c5t50014EE203AF718Fd0
        0   0   0   0 c5t50014EE2AD9F5D65d0
        0   0   0   0 c5t5000CCA228C0E4E0d0
        0   0   0   0 c5t5000CCA228C0ED61d0
        0   0   0   0 c5t50014EE2AE46C274d0
        0   0   0   0 c5t5000CCA221E08EF4d0
        0   0   0   0 c5t5000CCA228C06632d0
        0   2   1   3 c5t5000CCA228C08502d0
        0   0   0   0 c5t5000CCA228C091FAd0
        0   0   0   0 c5t5000CCA228C08FFFd0
    flain@monkey:~# uptime
      3:50pm  up 68 day(s),  4:57,  1 user,  load average: 0.18, 0.18, 0.19
    flain@monkey:~#
    
    I'm not concerned about those 3 errors on that one drive, when i had issues i had hundreds and sometimes thousands within a few days.
     
    Last edited: Jan 16, 2012
  15. Stanza

    Stanza Member

    Joined:
    Jun 27, 2001
    Messages:
    2,901
    Location:
    Adelaide

Share This Page

Advertisement: