Overclockers Australia Forums

OCAU News - Wiki - QuickLinks - Pix - Sponsors  

Go Back   Overclockers Australia Forums > Specific Hardware Topics > Storage & Backup

Notices


Sign up for a free OCAU account and this ad will go away!
Search our forums with Google:
Reply
 
Thread Tools
Old 7th June 2017, 7:26 PM   #1
Revenger Thread Starter
Member
 
Join Date: Aug 2002
Location: Armadale / Perth
Posts: 3,488
Default Help me sort this storage system out

Ok I am wanting advice and ideas on sorting my data system out.

It's been a mess of multiple single drives floating on on my benches for ages.

Currently have:

7 WD Red 3TB
1 WD Green 3TB

1 WD Red 3TB to replace in my server bad sectors

Extras
1 Segate 3TB (noting important as the drive isn't too reliable)
4 WD in a Raid 5 not touched in a couple years.
4 WD in raid 5 to recover still if I ever can
A bunch of assorted sizes 120gb etc.

Most drives are nearly full to about 100gb free on each.

Eventually I want to transition to bigger drives.

--------------
All the 3TB drives are all single partition NTFS drives except the server which is EXT4.

Most drives are floating on my bench stacked ontop of each other when not in use.

My main computer case Norco 4020 has no drive rails in it anymore and they all had issues broke etc.

--------------
I sometimes use other PC's and devices and would eventually would like some storage accessible to other systems and android devices like ebooks, PDF's, music, my growing video and other collections etc.

I want to get this properly sorted for good as I know this temp setup I have cant last for ever.

I looked the Thermaltake W100 case if that could be a good solution for the time being as eventually I also need a new computer and case upgrade planned end of this yeat to mid next which should also be taken into consideration.

I thought of small nas devices but want to transition to bigger drives.

--------------
So OCAU recommend me a proper decent cost solution to sort this mess out, hardware I need, drives I need, software I need etc etc etc.

Also post any reasons and thoughts on this.

Thanks
__________________
COMPUTER: I7 950, Gigabyte UD3Rv2 , 12GB Corsair DDR3-1600 CL7 ram, NVIDIA GTX 590 + 560, 24TB+ HDD space, 3x u2410, 7.1 HDMI passthrough
flickr - OCAU Night Owl

Last edited by Revenger; 8th June 2017 at 3:20 PM.
Revenger is offline   Reply With Quote

Join OCAU to remove this ad!
Old 13th June 2017, 1:12 AM   #2
EvilGenius
Member
 
EvilGenius's Avatar
 
Join Date: Apr 2005
Location: _Rocky Status:_Folding!
Posts: 9,022
Default

A quick low fuss/non complicated solution would be to build whatever pc you want, plug all your existing drives in and use stablebit drivepool to create a storage pool across all of them. It'll handle mismatched drive sizes and types etc, it doesn't care. Basically pool everything up, then you can add new drives and remove old ones at your leisure. It will balance the storage across them as required. The only redundancy this option would give you is the ability to duplicate any/some/all of your data as you wish, it makes sure there's a copy on at least 2 hard drives at all times. This of course doubles the amount of storage space you need.


Another option is flexraid. It works in a very similar manner, except that you can configure it with storage parity. You need at least one hard drive as big as your largest drive that is completely empty. You set it as the parity drive and it creates a software raid of all your data. In theory you can lose any 1 hard drive and recover all the data from the parity drive. You can choose to have more than one parity drive if you like for better redundancy etc. Otherwise it works in a very similar way to stablebit. You can choose to outright duplicate important files so you don't need to worry about rebuilding lost files from a parity drive, and it creates a storage pool of all your drives. You can also add new drives and remove old ones. This is more complicated to setup though and more can go wrong with it, but it's no more or less risky in terms of data integrity.



These would basically be an interim option between piles of drives on your desk and a new system, be that a NAS or proper server build. Over time you could relatively easily consolidate your data onto larger disks removing the oldest/smallest. Also both of these programs save your data on disk as just standard files. Any disk can be removed and read on any other machine, so you're not locking yourself into whatever system you get going with this method.
__________________
i7-4820k @ 4.6 | X79-Deluxe | 64GB GsKill Ares PC-14900 | EVGA GTX 970 | Corsair HX-850 | CM690II
Once more unto the breach dear friends, once more
Cry fold for Team24, OCAU and all the world!
Wanted - N64 console - decent controller/s
EvilGenius is offline   Reply With Quote
Old 13th June 2017, 1:51 AM   #3
Revenger Thread Starter
Member
 
Join Date: Aug 2002
Location: Armadale / Perth
Posts: 3,488
Default

Oh my....
Thanks for the reply EvilGenius.
I will definitely be looking those up at some stage soon.

Would need a new case for my main system to house the drives temporary.

It's given me something good to look into and the dual parity non lock in sounds great as I still have to recover a raid 5 that accident had 2 drives removed when one has bad sectors and I was checking them with gsmart.

I'll create a topic on that when I get a spare drive with no other drive issues.

For the best solution instead of interim what would people recommend for long term scalability?

With growing collections such as videos, arcade, retro systems, consoles etc etc etc, there is still a lot to get as I haven't started mame yet (for my arcade) and that's a 2TB collection with the extras.

So ultimately I am looking at long term and expansion.

So I guess I have a few options.

1. Dedicated Nas
2. Storage server (second PC) with some type of pool.
3. New main system with pooled storage.

As my main I use for gaming and reinstall every so often and open all my files (rarely get unlucky) might not be the best setup for pooled storage.
Plus I want to keep my main system easy and easy to use othe os systems or whatever.

Which leads me to a Nas or a second computer for long term and 8tb drives or so.
I know some basic pros and cons but enlighten me if it was you.

While I may have a Linux server I don't know much at all about storage and things like unraid and all the other storage things.
Nor any Nas's that are easy decent priced scalable and support 8tb drives or so.

Thanks Evil and others reading and considering input.
__________________
COMPUTER: I7 950, Gigabyte UD3Rv2 , 12GB Corsair DDR3-1600 CL7 ram, NVIDIA GTX 590 + 560, 24TB+ HDD space, 3x u2410, 7.1 HDMI passthrough
flickr - OCAU Night Owl

Last edited by Revenger; 13th June 2017 at 1:55 AM.
Revenger is offline   Reply With Quote
Old 13th June 2017, 2:46 AM   #4
Copie
Member
 
Copie's Avatar
 
Join Date: Jul 2009
Location: Newcastle
Posts: 11,956
Default

All depends on your budget, there is a lot to be said about a gen 8 microserver and upgrading your drives to 6 or 8tbs, then plex backend for media distribution
__________________
My Solar stats
Trump 2016
Make America Great Again!
Copie is offline   Reply With Quote
Old 18th June 2017, 2:30 AM   #5
Revenger Thread Starter
Member
 
Join Date: Aug 2002
Location: Armadale / Perth
Posts: 3,488
Default

I was looking at the Synology DS1815+ just now based on storage review and seems like a nice price and you can get the two 5 bay expansion units for 18 drives.

With 8tb drives that's a potential 144TB storage.

That seems like good upgradability to scale to my needs.

How's the scaling with replacing drives adding new ones and expanding storage?

What's Synology's long term support like?

How's the power usage? Is it power friendly and won't eat through my wallet.

Anyone use these or similar and can comment on them with my needs?

Also smart data and issues can anyone comment?

Something like this seems to be a good for investment for me for long term of its easy to use and swap drives with issues etc.

Thanks
__________________
COMPUTER: I7 950, Gigabyte UD3Rv2 , 12GB Corsair DDR3-1600 CL7 ram, NVIDIA GTX 590 + 560, 24TB+ HDD space, 3x u2410, 7.1 HDMI passthrough
flickr - OCAU Night Owl
Revenger is offline   Reply With Quote
Old 21st June 2017, 3:54 AM   #6
Revenger Thread Starter
Member
 
Join Date: Aug 2002
Location: Armadale / Perth
Posts: 3,488
Default

I just came across U-NAS cases and they look very interesting.

The price seems nice butot I don't know the cost of all the other stuff motherboard, power supply raid card etc.

Also software FreeNAS etc, U-NAS have there own software for $30 us and other parts but I'm sure if one is there own mitx system in it may be a cheaper solution.

Any thoughts about this route?

So thoughts about Synology and a diy U-NAS case?
__________________
COMPUTER: I7 950, Gigabyte UD3Rv2 , 12GB Corsair DDR3-1600 CL7 ram, NVIDIA GTX 590 + 560, 24TB+ HDD space, 3x u2410, 7.1 HDMI passthrough
flickr - OCAU Night Owl
Revenger is offline   Reply With Quote
Old 21st June 2017, 11:22 AM   #7
davros123
Member
 
Join Date: Jun 2008
Posts: 2,782
Default Help me sort this storage system out

Recommendation #1: sell all your existing hdd's and start with fresh new storage drives. Estimate your growth for the next 1-2 years and buy space accordingly.

Budget is required.

Pls. provide separate budget costing for hdd's as this will likely be the major portion and can easily be estimated - ensure you allow two drives to account for raid6/raidz2/parity.

You can make a storage server for $300 or $3000+. Before suggesting parts, we need to know your budget. Hdd's will be a significant and known cost..we can work from there.
__________________
Want a nas, you may find my Esxi/Solaris ZFS NAS build thread of interest.
Quote:
Originally Posted by Stanza View Post
yeah well I just reported my own post...ferk....
Quote:
Originally Posted by Blinky View Post
If you have become content with the size of your e-penis, sticking clear of rack mounted stuff will save you heaps of $$$.

Last edited by davros123; 21st June 2017 at 11:28 AM.
davros123 is offline   Reply With Quote
Old 21st June 2017, 11:29 AM   #8
vladtepes
Member
 
Join Date: Sep 2015
Location: Brisbane, Qld
Posts: 1,008
Default

FWIW I think a larger array of more drives, rather than an array of larger drives is probably a better (and cheaper) option - especially as you already have quite a few drives.

A single drive failure of an 8Tb drive in an (appropriate) NAS array would mean a VERY long time for the resilvering process to occur for the replacement drive.

Of course, I could just be talking out my arse, there are certainly people here know far more about it than me.
__________________
My Dual System Build Thread

Members willing to do pickups for other members

successful trades: see profile.
vladtepes is offline   Reply With Quote
Old 21st June 2017, 11:33 AM   #9
davros123
Member
 
Join Date: Jun 2008
Posts: 2,782
Default

Quote:
Originally Posted by vladtepes View Post
FWIW I think a larger array of more drives, rather than an array of larger drives is probably a better (and cheaper) option - especially as you already have quite a few drives.



A single drive failure of an 8Tb drive in an (appropriate) NAS array would mean a VERY long time for the resilvering process to occur for the replacement drive.



Of course, I could just be talking out my arse, there are certainly people here know far more about it than me.


It's somewhere between 14 and 18 hours...not too bad imho.
__________________
Want a nas, you may find my Esxi/Solaris ZFS NAS build thread of interest.
Quote:
Originally Posted by Stanza View Post
yeah well I just reported my own post...ferk....
Quote:
Originally Posted by Blinky View Post
If you have become content with the size of your e-penis, sticking clear of rack mounted stuff will save you heaps of $$$.
davros123 is offline   Reply With Quote
Old 21st June 2017, 11:46 AM   #10
broccoli
Member
 
broccoli's Avatar
 
Join Date: Feb 2010
Location: Perth
Posts: 15,775
Default

Quote:
Originally Posted by Revenger View Post
Any thoughts about this route?
When I last looked, M-ITX motherboards and small form factor power supplies were few and far between, and not budget-priced. Raid cards cost a bomb. Costwise, you might be looking at a fair bit of moolah to DIY. Then, you also need to decide on an OS, and figure out how to use it. Not a big deal if all you want to do is stick files on it, but more time and effort if you want to install/use other programs.

Quote:
Originally Posted by Revenger View Post
So thoughts about Synology and a diy U-NAS case?
If you buy a pre-configured NAS, you get the software, and the other programs that work on it. If you want to "do stuff", have a look at the application downloads on Synology's site and see what is available to you. I have prebuilt NASes and some of the available applications (with instructions) have saved me a lot of time and effort "doing stuff".

If I were you, I'd go for a preconfigured option. After you've sorted your messy arrangements, look into the available DIY OSes and, if you think you want to give that a go, then do it. You really should get things under control ASAP, and having something "plug and play" is the quicker way to do that.

(You will need to get a backup, so consider that when deciding what you need)
__________________
Quote:
Originally Posted by elvis View Post
People are idiots.
Quote:
Originally Posted by Sankari View Post
I expected this to be bad, but they really put the effort in and took it to a whole new level of atrocious.
broccoli is online now   Reply With Quote
Old 21st June 2017, 11:49 AM   #11
Revenger Thread Starter
Member
 
Join Date: Aug 2002
Location: Armadale / Perth
Posts: 3,488
Default

I'm don't exactly have a budget as such.
Basically have no immediate budget but can save for something.

What I am doing is currently assessing my options and what would be ideal as ongoing migration etc.

The Synology I keep hovering towards with its hybrid raid ability to have a array and expand just by adding or replacing a few drives at a time.

I could easily start setting it up on the 3tb drives I have and migrate all my current drives to it now and upgrade little by little to 8tb drives and the extra two expansion bays.

That's what I am currently leaning towards on my mindset as it appeals to me with growing little by little over say the next 5 years.

Overall as said I'm assessing my options here for simple and easy to migrate over to.

Don't have a budget as said 1-2k would be a rough figure of I had to put a figure on anything but at the same time I need a new PC here etc and it takes me a while to save.

Edit: broccoli that's my thoughts exactly and why the Synology appeals to me as I can use my current drives and migrate from 1 drive to all current 3tb drives bit by bit then expand up to now storage from there with the cost of just the Nas and a couple extra drives for that needed spare space.

As well I have data in a non connected raid and one to try to recover.

So the Synology really appeals with scalability.

It would be something is be aiming for later in the year and take it from there.

Yep plan to look at the Synology modules available today as I didn't sleep last night doing soo much research lying on bed about all this.
__________________
COMPUTER: I7 950, Gigabyte UD3Rv2 , 12GB Corsair DDR3-1600 CL7 ram, NVIDIA GTX 590 + 560, 24TB+ HDD space, 3x u2410, 7.1 HDMI passthrough
flickr - OCAU Night Owl

Last edited by Revenger; 21st June 2017 at 11:58 AM.
Revenger is offline   Reply With Quote
Old 21st June 2017, 11:51 AM   #12
broccoli
Member
 
broccoli's Avatar
 
Join Date: Feb 2010
Location: Perth
Posts: 15,775
Default

Quote:
Originally Posted by Revenger View Post
I could easily start setting it up on the 3tb drives I have and migrate all my current drives to it now
Bear in mind that the Synology will reformat your drives, you can't put them in and keep/read the existing data, you'll have to move it off first
__________________
Quote:
Originally Posted by elvis View Post
People are idiots.
Quote:
Originally Posted by Sankari View Post
I expected this to be bad, but they really put the effort in and took it to a whole new level of atrocious.
broccoli is online now   Reply With Quote
Old 21st June 2017, 12:06 PM   #13
Revenger Thread Starter
Member
 
Join Date: Aug 2002
Location: Armadale / Perth
Posts: 3,488
Default

Quote:
Originally Posted by broccoli View Post
Bear in mind that the Synology will reformat your drives, you can't put them in and keep/read the existing data, you'll have to move it off first
Already know this and that's why I mentioned migrate bit by bit get a 8tb initially to copy some 3's, then put those drives in the Synology and copy data to those and expand the drives bit by bit by moving data round to free up drives to expand onto.

I've done this quite a few times (constantly) especially as normal drives when just sorting out and consolidating data so that's no issue for me.

The hybrid raid appeals to me in that way that you can expand from 1 to all drives with just adding new drives in once there freed up.
__________________
COMPUTER: I7 950, Gigabyte UD3Rv2 , 12GB Corsair DDR3-1600 CL7 ram, NVIDIA GTX 590 + 560, 24TB+ HDD space, 3x u2410, 7.1 HDMI passthrough
flickr - OCAU Night Owl

Last edited by Revenger; 21st June 2017 at 12:10 PM.
Revenger is offline   Reply With Quote
Old 21st June 2017, 12:16 PM   #14
davros123
Member
 
Join Date: Jun 2008
Posts: 2,782
Default Help me sort this storage system out

"Set this up properly" and "little by little over say the next 5 years" does not reconcile. I do not recommend little by little. Save and do it right.

I'd get the following:
Super micro 1151 mobo ~$300 for an X11on eBay (better now as there is 20%off atm )
16 GB EEC ram $300
E3-1220 $300 (guess)
Fractal R5 $140 - lots of drive space and cool and quiet. You do NOT need hotswap drive bays...once in, you will never open the case.
Seasonic psus or the like with a good 12v rail $ 200
60gb ssd for os
Freenas - free.

Drives of your choosing - $ lots

Edit: doing it bit by bit = half assed and is out of my skill set. Will leave it to others to assist.
__________________
Want a nas, you may find my Esxi/Solaris ZFS NAS build thread of interest.
Quote:
Originally Posted by Stanza View Post
yeah well I just reported my own post...ferk....
Quote:
Originally Posted by Blinky View Post
If you have become content with the size of your e-penis, sticking clear of rack mounted stuff will save you heaps of $$$.

Last edited by davros123; 21st June 2017 at 12:18 PM.
davros123 is offline   Reply With Quote
Old 21st June 2017, 12:17 PM   #15
broccoli
Member
 
broccoli's Avatar
 
Join Date: Feb 2010
Location: Perth
Posts: 15,775
Default

ah ok, gotcha
Quote:
Originally Posted by Revenger View Post
The hybrid raid appeals to me in that way that you can expand from 1 to all drives with just adding new drives in once there freed up.
If you mix big drives and small ones, you'll lose most of the capacity of the big ones. You might want to consider just using your drives as single drives in the nas.
__________________
Quote:
Originally Posted by elvis View Post
People are idiots.
Quote:
Originally Posted by Sankari View Post
I expected this to be bad, but they really put the effort in and took it to a whole new level of atrocious.
broccoli is online now   Reply With Quote
Reply

Bookmarks

Sign up for a free OCAU account and this ad will go away!

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time now is 6:50 PM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
OCAU is not responsible for the content of individual messages posted by others.
Other content copyright Overclockers Australia.
OCAU is hosted by Micron21!