Help me sort this storage system out

Discussion in 'Storage & Backup' started by Revenger, Jun 7, 2017.

  1. Revenger

    Revenger Member

    Joined:
    Aug 7, 2002
    Messages:
    3,536
    Location:
    Armadale / Perth
    Ok I am wanting advice and ideas on sorting my data system out.

    It's been a mess of multiple single drives floating on on my benches for ages.

    Currently have:

    7 WD Red 3TB
    1 WD Green 3TB

    1 WD Red 3TB to replace in my server bad sectors

    Extras
    1 Segate 3TB (noting important as the drive isn't too reliable)
    4 WD in a Raid 5 not touched in a couple years.
    4 WD in raid 5 to recover still if I ever can
    A bunch of assorted sizes 120gb etc.

    Most drives are nearly full to about 100gb free on each.

    Eventually I want to transition to bigger drives.

    --------------
    All the 3TB drives are all single partition NTFS drives except the server which is EXT4.

    Most drives are floating on my bench stacked ontop of each other when not in use.

    My main computer case Norco 4020 has no drive rails in it anymore and they all had issues broke etc.

    --------------
    I sometimes use other PC's and devices and would eventually would like some storage accessible to other systems and android devices like ebooks, PDF's, music, my growing video and other collections etc.

    I want to get this properly sorted for good as I know this temp setup I have cant last for ever.

    I looked the Thermaltake W100 case if that could be a good solution for the time being as eventually I also need a new computer and case upgrade planned end of this yeat to mid next which should also be taken into consideration.

    I thought of small nas devices but want to transition to bigger drives.

    --------------
    So OCAU recommend me a proper decent cost solution to sort this mess out, hardware I need, drives I need, software I need etc etc etc.

    Also post any reasons and thoughts on this.

    Thanks
     
    Last edited: Jun 8, 2017
  2. EvilGenius

    EvilGenius Member

    Joined:
    Apr 26, 2005
    Messages:
    9,157
    Location:
    _Rocky Status:_Folding!
    A quick low fuss/non complicated solution would be to build whatever pc you want, plug all your existing drives in and use stablebit drivepool to create a storage pool across all of them. It'll handle mismatched drive sizes and types etc, it doesn't care. Basically pool everything up, then you can add new drives and remove old ones at your leisure. It will balance the storage across them as required. The only redundancy this option would give you is the ability to duplicate any/some/all of your data as you wish, it makes sure there's a copy on at least 2 hard drives at all times. This of course doubles the amount of storage space you need.


    Another option is flexraid. It works in a very similar manner, except that you can configure it with storage parity. You need at least one hard drive as big as your largest drive that is completely empty. You set it as the parity drive and it creates a software raid of all your data. In theory you can lose any 1 hard drive and recover all the data from the parity drive. You can choose to have more than one parity drive if you like for better redundancy etc. Otherwise it works in a very similar way to stablebit. You can choose to outright duplicate important files so you don't need to worry about rebuilding lost files from a parity drive, and it creates a storage pool of all your drives. You can also add new drives and remove old ones. This is more complicated to setup though and more can go wrong with it, but it's no more or less risky in terms of data integrity.



    These would basically be an interim option between piles of drives on your desk and a new system, be that a NAS or proper server build. Over time you could relatively easily consolidate your data onto larger disks removing the oldest/smallest. Also both of these programs save your data on disk as just standard files. Any disk can be removed and read on any other machine, so you're not locking yourself into whatever system you get going with this method.
     
  3. Revenger

    Revenger Member

    Joined:
    Aug 7, 2002
    Messages:
    3,536
    Location:
    Armadale / Perth
    Oh my....
    Thanks for the reply EvilGenius.
    I will definitely be looking those up at some stage soon.

    Would need a new case for my main system to house the drives temporary.

    It's given me something good to look into and the dual parity non lock in sounds great as I still have to recover a raid 5 that accident had 2 drives removed when one has bad sectors and I was checking them with gsmart.

    I'll create a topic on that when I get a spare drive with no other drive issues.

    For the best solution instead of interim what would people recommend for long term scalability?

    With growing collections such as videos, arcade, retro systems, consoles etc etc etc, there is still a lot to get as I haven't started mame yet (for my arcade) and that's a 2TB collection with the extras.

    So ultimately I am looking at long term and expansion.

    So I guess I have a few options.

    1. Dedicated Nas
    2. Storage server (second PC) with some type of pool.
    3. New main system with pooled storage.

    As my main I use for gaming and reinstall every so often and open all my files (rarely get unlucky) might not be the best setup for pooled storage.
    Plus I want to keep my main system easy and easy to use othe os systems or whatever.

    Which leads me to a Nas or a second computer for long term and 8tb drives or so.
    I know some basic pros and cons but enlighten me if it was you.

    While I may have a Linux server I don't know much at all about storage and things like unraid and all the other storage things.
    Nor any Nas's that are easy decent priced scalable and support 8tb drives or so.

    Thanks Evil and others reading and considering input.
     
    Last edited: Jun 13, 2017
  4. Copie

    Copie Member

    Joined:
    Jul 24, 2009
    Messages:
    12,287
    Location:
    Newcastle
    All depends on your budget, there is a lot to be said about a gen 8 microserver and upgrading your drives to 6 or 8tbs, then plex backend for media distribution
     
  5. Revenger

    Revenger Member

    Joined:
    Aug 7, 2002
    Messages:
    3,536
    Location:
    Armadale / Perth
    I was looking at the Synology DS1815+ just now based on storage review and seems like a nice price and you can get the two 5 bay expansion units for 18 drives.

    With 8tb drives that's a potential 144TB storage.

    That seems like good upgradability to scale to my needs.

    How's the scaling with replacing drives adding new ones and expanding storage?

    What's Synology's long term support like?

    How's the power usage? Is it power friendly and won't eat through my wallet.

    Anyone use these or similar and can comment on them with my needs?

    Also smart data and issues can anyone comment?

    Something like this seems to be a good for investment for me for long term of its easy to use and swap drives with issues etc.

    Thanks
     
  6. Revenger

    Revenger Member

    Joined:
    Aug 7, 2002
    Messages:
    3,536
    Location:
    Armadale / Perth
    I just came across U-NAS cases and they look very interesting.

    The price seems nice butot I don't know the cost of all the other stuff motherboard, power supply raid card etc.

    Also software FreeNAS etc, U-NAS have there own software for $30 us and other parts but I'm sure if one is there own mitx system in it may be a cheaper solution.

    Any thoughts about this route?

    So thoughts about Synology and a diy U-NAS case?
     
  7. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    2,816
    Recommendation #1: sell all your existing hdd's and start with fresh new storage drives. Estimate your growth for the next 1-2 years and buy space accordingly.

    Budget is required.

    Pls. provide separate budget costing for hdd's as this will likely be the major portion and can easily be estimated - ensure you allow two drives to account for raid6/raidz2/parity.

    You can make a storage server for $300 or $3000+. Before suggesting parts, we need to know your budget. Hdd's will be a significant and known cost..we can work from there.
     
    Last edited: Jun 21, 2017
  8. vladtepes

    vladtepes Member

    Joined:
    Sep 30, 2015
    Messages:
    1,188
    Location:
    Brisbane, Qld
    FWIW I think a larger array of more drives, rather than an array of larger drives is probably a better (and cheaper) option - especially as you already have quite a few drives.

    A single drive failure of an 8Tb drive in an (appropriate) NAS array would mean a VERY long time for the resilvering process to occur for the replacement drive.

    Of course, I could just be talking out my arse, there are certainly people here know far more about it than me.
     
  9. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    2,816


    It's somewhere between 14 and 18 hours...not too bad imho.
     
  10. broccoli

    broccoli Member

    Joined:
    Feb 21, 2010
    Messages:
    16,276
    Location:
    Perth
    When I last looked, M-ITX motherboards and small form factor power supplies were few and far between, and not budget-priced. Raid cards cost a bomb. Costwise, you might be looking at a fair bit of moolah to DIY. Then, you also need to decide on an OS, and figure out how to use it. Not a big deal if all you want to do is stick files on it, but more time and effort if you want to install/use other programs.

    If you buy a pre-configured NAS, you get the software, and the other programs that work on it. If you want to "do stuff", have a look at the application downloads on Synology's site and see what is available to you. I have prebuilt NASes and some of the available applications (with instructions) have saved me a lot of time and effort "doing stuff".

    If I were you, I'd go for a preconfigured option. After you've sorted your messy arrangements, look into the available DIY OSes and, if you think you want to give that a go, then do it. You really should get things under control ASAP, and having something "plug and play" is the quicker way to do that.

    (You will need to get a backup, so consider that when deciding what you need)
     
  11. Revenger

    Revenger Member

    Joined:
    Aug 7, 2002
    Messages:
    3,536
    Location:
    Armadale / Perth
    I'm don't exactly have a budget as such.
    Basically have no immediate budget but can save for something.

    What I am doing is currently assessing my options and what would be ideal as ongoing migration etc.

    The Synology I keep hovering towards with its hybrid raid ability to have a array and expand just by adding or replacing a few drives at a time.

    I could easily start setting it up on the 3tb drives I have and migrate all my current drives to it now and upgrade little by little to 8tb drives and the extra two expansion bays.

    That's what I am currently leaning towards on my mindset as it appeals to me with growing little by little over say the next 5 years.

    Overall as said I'm assessing my options here for simple and easy to migrate over to.

    Don't have a budget as said 1-2k would be a rough figure of I had to put a figure on anything but at the same time I need a new PC here etc and it takes me a while to save.

    Edit: broccoli that's my thoughts exactly and why the Synology appeals to me as I can use my current drives and migrate from 1 drive to all current 3tb drives bit by bit then expand up to now storage from there with the cost of just the Nas and a couple extra drives for that needed spare space.

    As well I have data in a non connected raid and one to try to recover.

    So the Synology really appeals with scalability.

    It would be something is be aiming for later in the year and take it from there.

    Yep plan to look at the Synology modules available today as I didn't sleep last night doing soo much research lying on bed about all this.
     
    Last edited: Jun 21, 2017
  12. broccoli

    broccoli Member

    Joined:
    Feb 21, 2010
    Messages:
    16,276
    Location:
    Perth
    Bear in mind that the Synology will reformat your drives, you can't put them in and keep/read the existing data, you'll have to move it off first
     
  13. Revenger

    Revenger Member

    Joined:
    Aug 7, 2002
    Messages:
    3,536
    Location:
    Armadale / Perth
    Already know this and that's why I mentioned migrate bit by bit get a 8tb initially to copy some 3's, then put those drives in the Synology and copy data to those and expand the drives bit by bit by moving data round to free up drives to expand onto.

    I've done this quite a few times (constantly) especially as normal drives when just sorting out and consolidating data so that's no issue for me.

    The hybrid raid appeals to me in that way that you can expand from 1 to all drives with just adding new drives in once there freed up.
     
    Last edited: Jun 21, 2017
  14. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    2,816
    "Set this up properly" and "little by little over say the next 5 years" does not reconcile. I do not recommend little by little. Save and do it right.

    I'd get the following:
    Super micro 1151 mobo ~$300 for an X11on eBay (better now as there is 20%off atm )
    16 GB EEC ram $300
    E3-1220 $300 (guess)
    Fractal R5 $140 - lots of drive space and cool and quiet. You do NOT need hotswap drive bays...once in, you will never open the case.
    Seasonic psus or the like with a good 12v rail $ 200
    60gb ssd for os
    Freenas - free.

    Drives of your choosing - $ lots ;)

    Edit: doing it bit by bit = half assed and is out of my skill set. Will leave it to others to assist.
     
    Last edited: Jun 21, 2017
  15. broccoli

    broccoli Member

    Joined:
    Feb 21, 2010
    Messages:
    16,276
    Location:
    Perth
    ah ok, gotcha :thumbup:
    If you mix big drives and small ones, you'll lose most of the capacity of the big ones. You might want to consider just using your drives as single drives in the nas.
     
  16. Revenger

    Revenger Member

    Joined:
    Aug 7, 2002
    Messages:
    3,536
    Location:
    Armadale / Perth
    When I said 5 years I didn't mean screw around for that time but instead get a setup now with the Synology and migrate and expand to better and full 8TB over 5 years basically a 5+ year lifecycle with scalability of the setup.

    Obviously at $470 per 8TB drive that's $$$$ and why one would scale.

    According to Synology its hybrid raid had no wasted space as it automatically allocates space and sets up the redundancy accordingly compared to conventional raid methods if you have 2+ of a drive size installed.
    You can mix sizes as long as you have 2+ and it'll scale the raids.

    Yep this is what I was reading up on all night long and what appeals to me.
     
    Last edited: Jun 21, 2017
  17. whatisk

    whatisk Member

    Joined:
    Dec 11, 2004
    Messages:
    114
    Location:
    Brisbane
    If you get a Synology and use their SHR (allows different sized drives in RAID like setup) I believe you will need to make sure you start it with the smallest drives you want to use. If you start with 3TB drives, I don't believe down the track you can add a 2TB drive or two. They need to be 3TB or greater.

    I have a Synology unit (DS1812+) and a HP microserver running Xpenology. The HP runs SHR with a mixture of 2 and 3TB drives. I have picked up another Synology unit though so the HP is about to be retired.

    I find it great as was mentioned, you can just install packages which makes things a lot easier. There are also 3rd party repositories you can get more applications from that may be useful but not in the official Synology app center.

    And with Docker ability as well the sky is pretty much the limit.
     
  18. Revenger

    Revenger Member

    Joined:
    Aug 7, 2002
    Messages:
    3,536
    Location:
    Armadale / Perth
    Yep your correct SHR scales up not down.

    Not a issue I have all 3TB nothing below that main storage wise I'd use and will be scaling up bit by bit which is the appeal.

    Seems I did most of the research.

    How is the 1812? 5 years is it still going well? How's scalability and how easy is it? thinking of the new version of it 1817+ looked at the 2415 bay but that's a bit older so the new this years would be better I think.
     
    Last edited: Jun 21, 2017
  19. whatisk

    whatisk Member

    Joined:
    Dec 11, 2004
    Messages:
    114
    Location:
    Brisbane
    I haven't had the 1812 for that long. Got it off Gumtree along with 4x 5tb drives for a very good price. But running with no issues what so ever. Obviously CPU is only dual core compared to quad core in later models and it's max ram isn't as high, but for what I use it for it's great. No issues at all and runs latest DSM happily. I primarily use it for storage to stream to other devices, back up pcs to and am running deluge on it for a torrent client.

    One day I may expand it's uses.

    I wouldn't necessarily be that worried about how long the model has been around for. They seem to be rock solid. Just make sure it'll do what you like.
    One reason I've steered clear of the 1815+ or 1515+ is the intel CPU issues. If you buy new now you should be safe from that.
     
  20. EvilGenius

    EvilGenius Member

    Joined:
    Apr 26, 2005
    Messages:
    9,157
    Location:
    _Rocky Status:_Folding!
    We have a bunch of 1815's rolled out at work, they've been pretty solid. Only gotcha I would suggest you look out for is that you buy one that is new current stock. The 1815's were released with an intel CPU that suffers from sudden death at roughly the 2yr mark. Synology are aware of it but did not recall the unit, they simply set up a dedicated warranty support line/email address for affected customers. If you get a current revision though it will have a newer version of the CPU which is not affected.

    I can't comment on power usage as it's not even a consideration for us, guaranteed less than almost any other server setup though. Performance is sufficient to serve an office with ~100 users, that's office docs and general smallish file transfers though, but still you and any family/friends aren't going to stress it too much with a handful of media streams.

    Smart reporting is quite good, they've given us plenty of heads up prior to drive failures we've had and rebuilt without issue.
     

Share This Page