New server build /w LSI 9265 RAID5 benchmark

Discussion in 'Storage & Backup' started by Danthemanz, Jan 21, 2017.

  1. Danthemanz

    Danthemanz Old School Admin

    Joined:
    Jun 27, 2001
    Messages:
    2,421
    Location:
    Sydney
    Thought i would share some info and a really good lead on some cheap disks. Happy to take questions and run any benchmarks anyone wants to see.

    Im one to store alot of data at home on various servers.
    Ive never used RAID 5 at home as ive always micro managed space and multiple copies.
    Im an IT consultant with a background in server admin who uses RAID everyday, so i finally decided to build something to last many years now that i have a baby due in April. Also needed a decent server 2016 box for training.
    Data is setup for one direction replication nightly to my HTPC with Seagate 8TB Archive disks.

    Server specs:
    Supermicro X11SSH-LN4F Motherboard
    Xeon E3-1230 v5
    64GB ECC DDR4
    Nvidia Quadro K4200
    Silverstone TJ08-E uATX case
    Intel X520-DA2 (10Gb dual port SFP+ NIC (on the way)

    4 x 2.5" hotswap bay for additional disks
    Samsung 512GB 950 Pro (oem) (for VMs)
    Samsung 256GB 850 Pro (For boot)
    2 x Seagate 4TB 2.5" SMR HDDs
    5 x OEM Toshiba 6TB 7200rpm Enterprise HDDs, 512e
    LSI 9265-8i 1GB DDR3 with capacitor (M5016)

    The Toshiba disks on their own are good for about 200MB/s reads and 180MB/s writes. Disks are an absolute BARGAIN at $225 + delivery. I owned 2 for the past 6 months then bought another 3 cause they are so fast and cheap. They are a little noisy and hot though as they are datacentre disks. Looking to get another for a cold spare.

    Originally built with an LSI 9260 (m5015) which was fast (link below), but i wanted something better and newer that i can trust for many years.

    Now i understand that most of these numbers are from the 1GB cache, but i still wasnt expecting ~3.5GB/s of writes. Makes me wonder what the newer PCI-e 3.0 controllers can do.
    This controller is FAST. Ive no doubt used it before in client servers with SAS disks but i never properly tested it. I only paid about $270 for a new controller with a 2nd hand capacitor, all from china.

    Im happy to run any other benchmarks anyone is interested in seeing. Math would suggest the disks themselves should be able to do close to 1GB/s Reads and maybe ~700MB/s writes.
    Card is running from a PCI-e 3.0 8x slot off the CPU but running in PCI-e 2.0 mode.

    [​IMG]

    http://imgur.com/a/kqj97

    Comparison to the same array on an LSI 9260 512MB (M5015).
    http://imgur.com/a/IB5aH

    .
     
  2. hairy

    hairy Member

    Joined:
    Dec 11, 2010
    Messages:
    430
    can you do a test with over 1gig so we can see the non cache numbers ?
    what o.s ?
    good to see someone building a server for a change, most people just stream now days.
    I still horde a lot of media because my net sucks.
     
  3. Danthemanz

    Danthemanz Old School Admin

    Joined:
    Jun 27, 2001
    Messages:
    2,421
    Location:
    Sydney
    OS is Windows server 2016, using hyper-v for additional VMS but the benchmarks were on the host OS.

    I can run some more benchmarks tomorrow, in the middle of a huge data sync over the network and i wouldnt be fair to benchmark while in use.
     
  4. bcann

    bcann Member

    Joined:
    Feb 26, 2006
    Messages:
    4,251
    Location:
    NSW
    ahh looking at the sticker on that drive and one from say newegg, there is a vast difference with the info and look on them. I'm also guessing that these will be either manufacturer refurb or something else entirely.

    Have you done a warranty check on them on toshiba's website and if so can you post that back on here. Otherwise they are an amazing buy @ that price.
     
  5. Gaidin

    Gaidin Member

    Joined:
    Jun 27, 2001
    Messages:
    1,952
    Location:
    Vic
    Would love some more info on those Toshibas as well. I can probably reduce my 8 disk 4Tb RAID5 array down to a 6 disk 6Tb one :thumbup:
     
  6. Gaidin

    Gaidin Member

    Joined:
    Jun 27, 2001
    Messages:
    1,952
    Location:
    Vic
    VERY sceptical about storing anything on these drives until we can get some more info. The ebay listing is pretty useless in detail :)
     
  7. ae00711

    ae00711 Member

    Joined:
    Apr 9, 2013
    Messages:
    1,069
    :confused: why? these are toshibas, toshys have a good rep.
     
  8. Danthemanz

    Danthemanz Old School Admin

    Joined:
    Jun 27, 2001
    Messages:
    2,421
    Location:
    Sydney
    The drives are very much oem disks.
    These would probably be disks that are sold to OEM's such as HP/IBM for rebaging.
    No model numbers on them at all, not in firmware either.
    That said, 12 months warranty from supplier for early failures, they are really high spec enterprise drives so I'm expecting a good life from them.
    Even if I had to replace each one in a 5 year period I would be ahead.
    I'm going to keep a cold spare for my raid 5. I pulled a disk for a rebuild test and it rebuilt in 8 hours which is quite fast for this much data.
    They are brand new in sealed bags with zero hours on the smart info.

    They are loud like a sas disk but my god they are fast, fastest sata disk ive ever used.
    I'm everything i can gather these disks would be Toshiba MG04ACA600E.

    I'm certainly not concerned about storing data on them, I trust them more than some bloody slow desktop disks. Why the great concern Gaidin?
     
    Last edited: Jan 29, 2017
  9. StratosFear

    StratosFear Member

    Joined:
    Jun 27, 2001
    Messages:
    7,956
    Location:
    Melbourne, Australia
  10. Gaidin

    Gaidin Member

    Joined:
    Jun 27, 2001
    Messages:
    1,952
    Location:
    Vic
    Now the OP has checked back into the thread and provided some more detail then they sound fine, might be time to invest and roll the 2012 R2 server over to 2016 too :thumbup:
     
  11. Danthemanz

    Danthemanz Old School Admin

    Joined:
    Jun 27, 2001
    Messages:
    2,421
    Location:
    Sydney
    Sorry for the delay, my house was flooded and i had to move out.
    The server was ok though so not all was lost ;)
     
  12. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    16,185
    Location:
    Canberra
    ya'll crazy wasting ~$1600 on a Raid card in 2017 (which is the approx retail for a SAS2 Hardware Raid w/ CacheVault and Cachecade).

    Just use a HBA and Storage Spaces already (or BTRFS or ZFS).

    Cross-Flashing cards is not my cup of tea when you're talking about $1000's in storage.
     
    Last edited: Jan 29, 2017
  13. ae00711

    ae00711 Member

    Joined:
    Apr 9, 2013
    Messages:
    1,069
    *shrug* :pirate:
     
  14. Danthemanz

    Danthemanz Old School Admin

    Joined:
    Jun 27, 2001
    Messages:
    2,421
    Location:
    Sydney
    Paid $250 for the card after trying storage spaces and getting terrible performance. I didn't flash the card either.
    Have you used parity with storage spaces? Write speed was about 1/20th of the LSI controller, that was after tweaking.
    Windows was a requirement for the server so ZFS is out...

    I was very excited for storage pools under 2016 but after days of playing it didn't suit what I needed.
    Striped and mirror was still good though. Pity you have to match the "raid" type for tiered storage, i wanted to use a 950 pro for caching but I wasn't going to buy 3 of them..
     
    Last edited: Jan 29, 2017
  15. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    16,185
    Location:
    Canberra
    So - yeah.

    First thing is first though. Parity spaces (raid card or otherwise) suck at IO - so hosting VM's on them is always going to end in tears. And sure, for 1-2 vm's it will be ok, but the second you get a bit of an io blender, it falls apart.

    Second thing - why in god's name are you building a Raid5 array with 6TB members? You're looking at about a week+ to rebuild the array in the event of a failure. Thats a *long* time.

    But in terms of building a large parity space for typical File Serving - no worries flatlining 1GbE with 5 drives (roughly about 550MB/sec or so - but it depends *entirely* on your file mix). You need to enable the power protection flag and a WBC / Tiering helps *immensely*.

    For your specific use case mentioned (vm labs/training), I would be just grabbing a 1-2TB SSD and never thinking about it again.

    Ehhhh, You can get around it (e.g KVM/ESXi serving up from NFS - but even i'll admit its a stretch).

    There is really bugger all info out there regarding Storage Spaces design - definitely more in 2016 than 2012r2 (which was *entirely* Fujitsu's whitepapers) though.

    For "budgetish" builds, try;

    https://blogs.technet.microsoft.com/filecab/2016/10/14/kepler-47/
    https://www.tech-coffee.net/2-node-hyperconverged-cluster-with-windows-server-2016/
     
  16. Danthemanz

    Danthemanz Old School Admin

    Joined:
    Jun 27, 2001
    Messages:
    2,421
    Location:
    Sydney
    There are many purposes for this server and the large data storage is not for vms but storing personal high res photos and video as well as a high speed storage server for working off them, in addition to long term storage of other video and software type files.
    VMs are running on a Samsung 950 on its own so no problem there.

    As I said in a previous post, I did a disk failure test. Array was half filled and i pulled and rebuilt a disk in the array.
    It was quoting 9 hours but i just checked the logs and it was 10 hours, not too slow at all! This card is pretty dam fast, as are the disks.

    Yes i was using the power protection flag on the storage spaces, writes went from 30MBs to about 60... Not quite what I was looking for with the 10gb network I rolled out...
     
  17. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    16,185
    Location:
    Canberra
    You're never ever going to get anywhere near 10G with so few spindles and real world use.
     
  18. Danthemanz

    Danthemanz Old School Admin

    Joined:
    Jun 27, 2001
    Messages:
    2,421
    Location:
    Sydney
    ~600MB/s is actually a lot faster than the 112MB/s I was getting before on 1Gb...
    Way better than 60MB/s the storage spaces pool was going to offer.
    We will see after my house is repaired but im incredibly happy with the performance for what I'm using it for, its not like im running a corporate Sql database on it, its made for large file transfers.
     
  19. Gaidin

    Gaidin Member

    Joined:
    Jun 27, 2001
    Messages:
    1,952
    Location:
    Vic
    All sold now it seems :)
     
  20. encode

    encode Member

    Joined:
    May 21, 2003
    Messages:
    464
    Location:
    Sydney
    Out of curiosity, what were you using the house the 3.5" disks? The case apparently only has space for 4 disks.
     

Share This Page