Overclockers Australia Forums

OCAU News - Wiki - QuickLinks - Pix - Sponsors  

Go Back   Overclockers Australia Forums > Specific Hardware Topics > Storage & Backup

Notices


Sign up for a free OCAU account and this ad will go away!
Search our forums with Google:
Reply
 
Thread Tools
Old 21st January 2017, 8:57 AM   #1
Danthemanz Thread Starter
Old School Admin
 
Danthemanz's Avatar
 
Join Date: Jun 2001
Location: Sydney
Posts: 2,421
Exclamation New server build /w LSI 9265 RAID5 benchmark

Thought i would share some info and a really good lead on some cheap disks. Happy to take questions and run any benchmarks anyone wants to see.

Im one to store alot of data at home on various servers.
Ive never used RAID 5 at home as ive always micro managed space and multiple copies.
Im an IT consultant with a background in server admin who uses RAID everyday, so i finally decided to build something to last many years now that i have a baby due in April. Also needed a decent server 2016 box for training.
Data is setup for one direction replication nightly to my HTPC with Seagate 8TB Archive disks.

Server specs:
Supermicro X11SSH-LN4F Motherboard
Xeon E3-1230 v5
64GB ECC DDR4
Nvidia Quadro K4200
Silverstone TJ08-E uATX case
Intel X520-DA2 (10Gb dual port SFP+ NIC (on the way)

4 x 2.5" hotswap bay for additional disks
Samsung 512GB 950 Pro (oem) (for VMs)
Samsung 256GB 850 Pro (For boot)
2 x Seagate 4TB 2.5" SMR HDDs
5 x OEM Toshiba 6TB 7200rpm Enterprise HDDs, 512e
LSI 9265-8i 1GB DDR3 with capacitor (M5016)

The Toshiba disks on their own are good for about 200MB/s reads and 180MB/s writes. Disks are an absolute BARGAIN at $225 + delivery. I owned 2 for the past 6 months then bought another 3 cause they are so fast and cheap. They are a little noisy and hot though as they are datacentre disks. Looking to get another for a cold spare.

Originally built with an LSI 9260 (m5015) which was fast (link below), but i wanted something better and newer that i can trust for many years.

Now i understand that most of these numbers are from the 1GB cache, but i still wasnt expecting ~3.5GB/s of writes. Makes me wonder what the newer PCI-e 3.0 controllers can do.
This controller is FAST. Ive no doubt used it before in client servers with SAS disks but i never properly tested it. I only paid about $270 for a new controller with a 2nd hand capacitor, all from china.

Im happy to run any other benchmarks anyone is interested in seeing. Math would suggest the disks themselves should be able to do close to 1GB/s Reads and maybe ~700MB/s writes.
Card is running from a PCI-e 3.0 8x slot off the CPU but running in PCI-e 2.0 mode.



http://imgur.com/a/kqj97

Comparison to the same array on an LSI 9260 512MB (M5015).
http://imgur.com/a/IB5aH

.
__________________
At least 15 years and $30k worth of trades...
Danthemanz is offline   Reply With Quote

Join OCAU to remove this ad!
Old 21st January 2017, 10:57 AM   #2
hairy
Member
 
Join Date: Dec 2010
Posts: 389
Default

can you do a test with over 1gig so we can see the non cache numbers ?
what o.s ?
good to see someone building a server for a change, most people just stream now days.
I still horde a lot of media because my net sucks.
__________________
antec skeleton, msi gaming m5, 6700k, 32 gig 3200 ram, gtx 1080ti sli, hb bridge, cm 280 aio, 500gig 950pro, 750gb crucial ssd, 43inch phillips and 24 inch aoc monitors. server dual xeon quads, 48gig ram, 5x3tb and 2x1tb, hx850, lianli case, freenas.
hairy is offline   Reply With Quote
Old 21st January 2017, 11:11 AM   #3
Danthemanz Thread Starter
Old School Admin
 
Danthemanz's Avatar
 
Join Date: Jun 2001
Location: Sydney
Posts: 2,421
Default

OS is Windows server 2016, using hyper-v for additional VMS but the benchmarks were on the host OS.

I can run some more benchmarks tomorrow, in the middle of a huge data sync over the network and i wouldnt be fair to benchmark while in use.
__________________
At least 15 years and $30k worth of trades...
Danthemanz is offline   Reply With Quote
Old 24th January 2017, 2:07 PM   #4
bcann
Member
 
Join Date: Feb 2006
Location: NSW
Posts: 4,185
Default

Quote:
Originally Posted by Danthemanz View Post
OS is Windows server 2016, using hyper-v for additional VMS but the benchmarks were on the host OS.

I can run some more benchmarks tomorrow, in the middle of a huge data sync over the network and i wouldnt be fair to benchmark while in use.
ahh looking at the sticker on that drive and one from say newegg, there is a vast difference with the info and look on them. I'm also guessing that these will be either manufacturer refurb or something else entirely.

Have you done a warranty check on them on toshiba's website and if so can you post that back on here. Otherwise they are an amazing buy @ that price.
__________________
Quote:
Originally Posted by elvis View Post
All I do is hand folks the working gun. Up to them as to whether or not they go hunting to bring home the bacon, or shoot themselves in the foot. I am merely the lowly gunsmith, and nothing more.
bcann is offline   Reply With Quote
Old 25th January 2017, 12:45 PM   #5
Gaidin
Member
 
Gaidin's Avatar
 
Join Date: Jun 2001
Location: Vic
Posts: 1,952
Default

Would love some more info on those Toshibas as well. I can probably reduce my 8 disk 4Tb RAID5 array down to a 6 disk 6Tb one
Gaidin is offline   Reply With Quote
Old 29th January 2017, 12:07 PM   #6
Gaidin
Member
 
Gaidin's Avatar
 
Join Date: Jun 2001
Location: Vic
Posts: 1,952
Default

VERY sceptical about storing anything on these drives until we can get some more info. The ebay listing is pretty useless in detail
Gaidin is offline   Reply With Quote
Old 29th January 2017, 12:45 PM   #7
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 1,032
Default

Quote:
Originally Posted by Gaidin View Post
VERY sceptical about storing anything on these drives until we can get some more info. The ebay listing is pretty useless in detail
why? these are toshibas, toshys have a good rep.
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote
Old 29th January 2017, 2:42 PM   #8
Danthemanz Thread Starter
Old School Admin
 
Danthemanz's Avatar
 
Join Date: Jun 2001
Location: Sydney
Posts: 2,421
Default

The drives are very much oem disks.
These would probably be disks that are sold to OEM's such as HP/IBM for rebaging.
No model numbers on them at all, not in firmware either.
That said, 12 months warranty from supplier for early failures, they are really high spec enterprise drives so I'm expecting a good life from them.
Even if I had to replace each one in a 5 year period I would be ahead.
I'm going to keep a cold spare for my raid 5. I pulled a disk for a rebuild test and it rebuilt in 8 hours which is quite fast for this much data.
They are brand new in sealed bags with zero hours on the smart info.

They are loud like a sas disk but my god they are fast, fastest sata disk ive ever used.
I'm everything i can gather these disks would be Toshiba MG04ACA600E.

I'm certainly not concerned about storing data on them, I trust them more than some bloody slow desktop disks. Why the great concern Gaidin?
__________________
At least 15 years and $30k worth of trades...

Last edited by Danthemanz; 29th January 2017 at 2:46 PM.
Danthemanz is offline   Reply With Quote
Old 29th January 2017, 3:32 PM   #9
StratosFear
Member
 
Join Date: Jun 2001
Location: Melbourne, Australia
Posts: 7,899
Default

"4 Available" for $210 each Pickup from Thomastown Victoria or $12 shipping

http://www.ebay.com.au/itm/Toshiba-O...D/182436669159

I haven't bought from them. Although based on their feedback they do look legit.
StratosFear is online now   Reply With Quote
Old 29th January 2017, 3:33 PM   #10
Gaidin
Member
 
Gaidin's Avatar
 
Join Date: Jun 2001
Location: Vic
Posts: 1,952
Default

Now the OP has checked back into the thread and provided some more detail then they sound fine, might be time to invest and roll the 2012 R2 server over to 2016 too
Gaidin is offline   Reply With Quote
Old 29th January 2017, 4:05 PM   #11
Danthemanz Thread Starter
Old School Admin
 
Danthemanz's Avatar
 
Join Date: Jun 2001
Location: Sydney
Posts: 2,421
Default

Sorry for the delay, my house was flooded and i had to move out.
The server was ok though so not all was lost
__________________
At least 15 years and $30k worth of trades...
Danthemanz is offline   Reply With Quote
Old 29th January 2017, 4:25 PM   #12
NSanity
Member
 
NSanity's Avatar
 
Join Date: Mar 2002
Location: Canberra
Posts: 16,014
Default

ya'll crazy wasting ~$1600 on a Raid card in 2017 (which is the approx retail for a SAS2 Hardware Raid w/ CacheVault and Cachecade).

Just use a HBA and Storage Spaces already (or BTRFS or ZFS).

Cross-Flashing cards is not my cup of tea when you're talking about $1000's in storage.

Last edited by NSanity; 29th January 2017 at 4:28 PM.
NSanity is offline   Reply With Quote
Old 29th January 2017, 4:36 PM   #13
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 1,032
Default

Quote:
Originally Posted by NSanity View Post
ya'll crazy wasting ~$1600 on a Raid card in 2017 (which is the approx retail for a SAS2 Hardware Raid w/ CacheVault and Cachecade).

Just use a HBA and Storage Spaces already (or BTRFS or ZFS).

Cross-Flashing cards is not my cup of tea when you're talking about $1000's in storage.
Quote:
Originally Posted by Danthemanz View Post
Thought i would share some info and a really good lead on some cheap disks. Happy to take questions and run any benchmarks anyone wants to see.

Im one to store alot of data at home on various servers.
Ive never used RAID 5 at home as ive always micro managed space and multiple copies.
Im an IT consultant with a background in server admin who uses RAID everyday, so i finally decided to build something to last many years now that i have a baby due in April. Also needed a decent server 2016 box for training.
Data is setup for one direction replication nightly to my HTPC with Seagate 8TB Archive disks.

Server specs:
Supermicro X11SSH-LN4F Motherboard
Xeon E3-1230 v5
64GB ECC DDR4
Nvidia Quadro K4200
Silverstone TJ08-E uATX case
Intel X520-DA2 (10Gb dual port SFP+ NIC (on the way)

4 x 2.5" hotswap bay for additional disks
Samsung 512GB 950 Pro (oem) (for VMs)
Samsung 256GB 850 Pro (For boot)
2 x Seagate 4TB 2.5" SMR HDDs
5 x OEM Toshiba 6TB 7200rpm Enterprise HDDs, 512e
LSI 9265-8i 1GB DDR3 with capacitor (M5016)

The Toshiba disks on their own are good for about 200MB/s reads and 180MB/s writes. Disks are an absolute BARGAIN at $225 + delivery. I owned 2 for the past 6 months then bought another 3 cause they are so fast and cheap. They are a little noisy and hot though as they are datacentre disks. Looking to get another for a cold spare.

Originally built with an LSI 9260 (m5015) which was fast (link below), but i wanted something better and newer that i can trust for many years.

Now i understand that most of these numbers are from the 1GB cache, but i still wasnt expecting ~3.5GB/s of writes. Makes me wonder what the newer PCI-e 3.0 controllers can do.
This controller is FAST. Ive no doubt used it before in client servers with SAS disks but i never properly tested it. I only paid about $270 for a new controller with a 2nd hand capacitor, all from china.

Im happy to run any other benchmarks anyone is interested in seeing. Math would suggest the disks themselves should be able to do close to 1GB/s Reads and maybe ~700MB/s writes.
Card is running from a PCI-e 3.0 8x slot off the CPU but running in PCI-e 2.0 mode.

image

http://imgur.com/a/kqj97

Comparison to the same array on an LSI 9260 512MB (M5015).
http://imgur.com/a/IB5aH

.
*shrug*
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote
Old 29th January 2017, 5:01 PM   #14
Danthemanz Thread Starter
Old School Admin
 
Danthemanz's Avatar
 
Join Date: Jun 2001
Location: Sydney
Posts: 2,421
Default

Quote:
Originally Posted by NSanity View Post
ya'll crazy wasting ~$1600 on a Raid card in 2017 (which is the approx retail for a SAS2 Hardware Raid w/ CacheVault and Cachecade).

Just use a HBA and Storage Spaces already (or BTRFS or ZFS).

Cross-Flashing cards is not my cup of tea when you're talking about $1000's in storage.
Paid $250 for the card after trying storage spaces and getting terrible performance. I didn't flash the card either.
Have you used parity with storage spaces? Write speed was about 1/20th of the LSI controller, that was after tweaking.
Windows was a requirement for the server so ZFS is out...

I was very excited for storage pools under 2016 but after days of playing it didn't suit what I needed.
Striped and mirror was still good though. Pity you have to match the "raid" type for tiered storage, i wanted to use a 950 pro for caching but I wasn't going to buy 3 of them..
__________________
At least 15 years and $30k worth of trades...

Last edited by Danthemanz; 29th January 2017 at 6:26 PM.
Danthemanz is offline   Reply With Quote
Old 29th January 2017, 7:01 PM   #15
NSanity
Member
 
NSanity's Avatar
 
Join Date: Mar 2002
Location: Canberra
Posts: 16,014
Default

Quote:
Originally Posted by Danthemanz View Post
Paid $250 for the card after trying storage spaces and getting terrible performance. I didn't flash the card either.
Have you used parity with storage spaces? Write speed was about 1/20th of the LSI controller, that was after tweaking.
So - yeah.

First thing is first though. Parity spaces (raid card or otherwise) suck at IO - so hosting VM's on them is always going to end in tears. And sure, for 1-2 vm's it will be ok, but the second you get a bit of an io blender, it falls apart.

Second thing - why in god's name are you building a Raid5 array with 6TB members? You're looking at about a week+ to rebuild the array in the event of a failure. Thats a *long* time.

But in terms of building a large parity space for typical File Serving - no worries flatlining 1GbE with 5 drives (roughly about 550MB/sec or so - but it depends *entirely* on your file mix). You need to enable the power protection flag and a WBC / Tiering helps *immensely*.

For your specific use case mentioned (vm labs/training), I would be just grabbing a 1-2TB SSD and never thinking about it again.

Quote:
Originally Posted by Danthemanz View Post
Windows was a requirement for the server so ZFS is out...
Ehhhh, You can get around it (e.g KVM/ESXi serving up from NFS - but even i'll admit its a stretch).

Quote:
Originally Posted by Danthemanz View Post
I was very excited for storage pools under 2016 but after days of playing it didn't suit what I needed.
There is really bugger all info out there regarding Storage Spaces design - definitely more in 2016 than 2012r2 (which was *entirely* Fujitsu's whitepapers) though.

For "budgetish" builds, try;

https://blogs.technet.microsoft.com/.../14/kepler-47/
https://www.tech-coffee.net/2-node-h...s-server-2016/
NSanity is offline   Reply With Quote
Reply

Bookmarks

Tags
9260, 9265, lsi, raid5, server

Sign up for a free OCAU account and this ad will go away!

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time now is 1:30 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
OCAU is not responsible for the content of individual messages posted by others.
Other content copyright Overclockers Australia.
OCAU is hosted by Micron21!