Next gen filesystems (ZFS, BtrFS, ReFS, APFS, etc)

Discussion in 'Storage & Backup' started by elvis, May 20, 2016.

  1. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    43,454
    Location:
    Brisbane
  2. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    43,454
    Location:
    Brisbane
    XFS is quite a mature filesystem, originally designed by Silicon Graphics in 1993 for their IRIX operating system (their own version of UNIX). It was released under the GPL and integrated into Linux in 2001.

    XFS *is not* a next-gen filesystem. It's quite fast and excels at large file performance (much better than either EXT3 or EXT4, and far better than ZFS or BtrFS, but with obvious risks being non-next-gen).

    Some interesting notes have been appearing from the XFS developers about upcoming on-disk structure changes for XFS:

    http://phoronix.com/scan.php?page=news_item&px=XFS-Reverse-Mapping-4.8

    This sounds very much like your usual list of next-gen-fs features, which appear to be planned to integrated into XFS over the coming years.

    XFS is also just a flat filesystem, and doesn't offer any disk/volume management tools currently. It still relies on traditional RAID or LVM structures to offer those features, which make it difficult to truly get an accurate picture of disk health from the file system's point of view. All the same, it's interesting to see a filesystem as old as this looking to update itself under the same banner. (Compare and contrast BtrFS, which is somewhat of an evolution of ext4, but was forked fully to offer a completely new code base without the need to worry about either code or on-disk compatibility).
     
  3. Jazper

    Jazper Member

    Joined:
    Jul 28, 2001
    Messages:
    2,668
    Location:
    Melbourne Australia
    Lots of good info here. Well done for consolidating.

    Until a few days ago I was running mdadm + LVM + ext4 (started as ext3) on an array that started with 4x1.5tb disks in 2009 and grew to 8x2tb disks recently.

    I've had about 4 drive failures(and 3 different server configs) in that time without it missing a beat.

    Now I've moved to ZFS on Freenas under ESXi with a drive controller passed through.

    I have experienced bit rot, rarely but it does happen. I also have non-ecc ram at the moment which will be remedied with a platform upgrade when funds allow.

    Having done extensive research on the topic, I've come to the conclusion that ecc is nice to have, but not by any means necessary so long as you have a backup strategy, and worst case scenario, you're willing to restore from backups.

    It is also (exceedingly) unlikely that you'll experience a scrub of death as threatened by some with non-ecc. I'm prepared to take my chances for the time being, as I do have a backup strategy.

    As Elvis has said, new CoW filesystem W/o ecc > old filesystem.
     
    Last edited: Aug 12, 2016
  4. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    43,454
    Location:
    Brisbane
    ECC is a numbers game. Errors are guaranteed over enough cycles, so you as the individual need to play the gamble of whether or not you think you can beat the odds.

    It's much the same as a game of Russian roulette, but you've got a single bullet and a billion chambers. Highly unlikely you'll do damage if you're only doing low volumes, but the more data you're holding, and the more read/write operations you're doing, the more "trigger pulling" you're invariably doing.

    Bitrot is much the same, and is why we now have these next-gen file systems. Error rates have always been much the same, but we're not seeing drive sizes as standard today that people couldn't dream of filling 20 years ago. As a result, we're having to take measures to solve these things that were once so unlikely, nobody cared.

    I think at some point PC RAM will go all-ECC. In a world of multi-GHz, multi-core CPUs and PCI-E busses pushing data around at the rates we have today, ECC will soon become mandatory even for desktop and low-end server computing.

    In the server game, I don't think you can buy non-ECC RAM any more. Even the low end suppliers don't bother, and sell you ECC whether you ask for it or not.
     
  5. Jazper

    Jazper Member

    Joined:
    Jul 28, 2001
    Messages:
    2,668
    Location:
    Melbourne Australia
    I agree with you completely.

    Please don't see what I'm saying as "ECC is not the best way forward" - ECC is better.

    My rationale (simplified). A bad scrub from errors in ram is likely to cause some issues, which will likely be resolved on the following scrub/access.

    I've never in my 32 odd years of computing, had a catastrophic ram failure. I've had progressive ram failure, but only after heavy use while overclocking the snot out of it and volting the hell out of it.

    In contrast I'm running my server ram at lowish volts within spec

    ECC is a better solution but considering home use is not heavy use and I have a backup strategy for critical data - vs ext4/lvm/mdraid it is a better solution.

    I will go with ECC on the next upgrade, but there's no rush.
     
    Last edited: Aug 13, 2016
  6. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    43,454
    Location:
    Brisbane
    I'm actually in the same boat. While I run a lot of big/expensive gear at work, my home file server is quite modest and very old. It's running BtrFS without ECC purely to save on cost.

    When the time comes to upgrade, then yeah, I'll definitely invest in ECC RAM. By that stage I'd say the cost difference would be far less than it was when my current hardware was new.
     
  7. yanman

    yanman Member

    Joined:
    Jun 4, 2002
    Messages:
    6,609
    Location:
    Hobart
  8. Jazper

    Jazper Member

    Joined:
    Jul 28, 2001
    Messages:
    2,668
    Location:
    Melbourne Australia
    They have been saying it isn't production ready for some time and explained that this could be a problem.

    Frankly this is the reason that I ended up going with ZFS, raidZ2 (aka raid6, just the ZFS version) is what I run. I would have preferred a working Btrfs alternative but it doesn't exist.

    The only gripe I have about both ZFS and Btrfs is the inability to easily increase size by adding drives, but I came to the conclusion that for home use I won't want to run more than 8 drives, for power reasons if nothing else.
     
    Last edited: Aug 14, 2016
  9. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    43,454
    Location:
    Brisbane
    Yup, thread on it here too:
    http://forums.overclockers.com.au/showthread.php?t=1202054

    Single/Dup/RAID1/RAID10 modes are fine, however. The problem is specifically in the parity calculation, which only applies to those modes (RAID5 and RAID6 specifically for BtrFS). I'm still running BtrFS in "single" and "RAID1" modes for various workloads, and am still confident with their performance and safety at this stage.

    If you go back to post 1, you'll see I've mentioned BtrFS RAID5/6 is not production ready. On top of this issue, BtrFS RAID5/6 doesn't fix the RAID write hole, and the current implementation would require a rewrite to do so. Given that the current model is now labelled "fatally flawed", I hope this gives the BtrFS devs a chance to tear out the current implementation and start again, including fixes for the RAID5 write hole in the process.

    The (somewhat) good news is that BtrFS users who were naughty enough to avoid the warnings and use RAID5/6 can convert their data on the fly to something else (RAID1/10). If you're sitting at the 70% or more utilisation, it'll probably mean needing to buy another disk or two to get the space required. However BtrFS's ability to reorganise blocks on the fly on a mounted file system is nice in that regard (one of the selling points of BtrFS, despite some obvious downsides still).
     
  10. fad

    fad Member

    Joined:
    Jun 26, 2001
    Messages:
    2,459
    Location:
    City, Canberra, Australia
    When I was building a Storage spaces box based on a desktop 2011 3930k and 8x8gb non ecc. I had a few errors with bit rot and file corruption. The ram sticks had one bit which would flip if adjacent bits were toggled. During the build I was not sure if it was the chipset or cpu which was causing them. After around 2 weeks of memtest it was the ram sticks at fault. The chips must have had a faulty column inside them. There was only one test in memtest that would fail. The same bit would pass on all other tests.

    The memory errors were big enough that Windows patching bsod. Other wise I might have not known.

    It made me switch to a Supermicro server platform.
     
  11. link1896

    link1896 Member

    Joined:
    Jul 28, 2005
    Messages:
    368
    Location:
    Melbourne


    I'm using r710 with a LSI 9200-8e hba card connected to md1200 in production and at home with Freenas, bullet proof, three years and running. 1GB ECC Ram for every 1TB of storage.

    Have used md1200 with h800 with 12 virtual drives passed through to Freenas, less then ideal but not in a critical environment, with zero dramas.

    All with 2TB SAS drives though.
     
  12. MUTMAN

    MUTMAN Member

    Joined:
    Jun 27, 2001
    Messages:
    8,781
    Location:
    4109
    So...loaded question time.
    Whats the best file system for some Raid5 type storage/serving in a home environment at this time ?
    Or do we just wait for btrfs to be fixed ?
     
  13. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    18,243
    Location:
    Canberra
    If you're just serving media and sucking torrents and the like.

    Easily ZFS. That said, if incremental expansion is on your mind - you'd be better served with an EXT4 based MDADM array.
     
  14. fad

    fad Member

    Joined:
    Jun 26, 2001
    Messages:
    2,459
    Location:
    City, Canberra, Australia
    It is not that it doesn't work. I have a r720xd at work with a LSI HBA. just that its not on the HCL for Dell, so when you go to them for help or warranty it will not be supported.
     
  15. MUTMAN

    MUTMAN Member

    Joined:
    Jun 27, 2001
    Messages:
    8,781
    Location:
    4109
    thanks NSanity

    Growing the array every few years seems inevitable. One of my synology boxes has grown from 2x 500gb + 1x 250gb jbod into a 4x 3tb raid10 through multiple steps. Minimal down time isn't a hard requirement, but it sure was handy.
    It ended up as r10 because i didn't have any other affordable backup solution at the time and also wanted more speed (which i didnt get, but thats another story).
    Now i have the backups covered and want to move back to a raid5... Synology is ext4 with MDADM, so it seems I'm best served sticking to that for now...
    Syno's DSM 6 is doing raid 5 on btrfs though... Is there some sorcery involved ?
     
  16. Jazper

    Jazper Member

    Joined:
    Jul 28, 2001
    Messages:
    2,668
    Location:
    Melbourne Australia
    I wonder if Btrfs on mdraid would be better than ext4
     
  17. Jazper

    Jazper Member

    Joined:
    Jul 28, 2001
    Messages:
    2,668
    Location:
    Melbourne Australia
    Raid 5 on syno dsm with Btrfs is using Btrfs on Mdraid. You get some of the benefits but not all(notably no self healing). ZFS is easily more stable if you are creating new. The drawback is you can't add disks, but you can change the disks for bigger ones.
     
  18. MUTMAN

    MUTMAN Member

    Joined:
    Jun 27, 2001
    Messages:
    8,781
    Location:
    4109
    ZFS is out for now as i only have 8gig ram (but it is ecc)
    single parity disk that's bit rot free sounds like its still a little way away for me yet.
    i had a drive fail under an old raid 5 on ext4 and while the rebuild took close to a week, the fact was my data was never interupted (i did kill the torrents on that box for the duration though, small price to pay) so a raid 5 is a big bonus for peace of mind.
    ummmm, might stick to what i know for now. stay on top of the backup routine and i can keep stuffing around changing the server setup whenever i want.

    thanks for the input guys :thumbup:
     
  19. Jazper

    Jazper Member

    Joined:
    Jul 28, 2001
    Messages:
    2,668
    Location:
    Melbourne Australia
    8gig of ram is plenty for ZFS for home use. For reference, my ZFS VM has 12 gig and 10-11 is used for cache. Can even use nas4free on 4-5 gig of ram happily.
     
    Last edited: Aug 14, 2016
  20. link1896

    link1896 Member

    Joined:
    Jul 28, 2005
    Messages:
    368
    Location:
    Melbourne


    True.

    If it isn't running a flavour of MS you're rooted when you need support anyway, forces me to send a tech to site to meet the dell support contractor as when they spot non MS it gets messy.
     

Share This Page

Advertisement: