Increase storage size and move from mdraid+LVM to ZFS - feedback and comments please

Discussion in 'Storage & Backup' started by daehenoc, Aug 9, 2021.

  1. jhunt31

    jhunt31 Member

    Joined:
    Aug 8, 2005
    Messages:
    222
    Location:
    4032
    No, ZFS is not able to expand RAID-Z vdev's like that. You would destroy current pool, create new pool & copy your data back, or alternatively expand storage capacity by adding another RAID-Z vdev to the pool.

    RAID-Z expansion is in the works but it won't change the existing striping, you would need to rewrite all of your data after attaching a new drive to your existing pool. See recent overview [Ars Technica] & there's a FreeBSD dev summit presentation from Matt Ahrens

    Spin up a VM and have a play with ZFS using some virtual disks :thumbup:
     
  2. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    15,473
    Location:
    Canberra
    The big downside for ZFS at home, is you really have to plan your storage for now and a fair way into the future, because of the cases where you can't just add/remove disks at will. e.g. if you have room for 4 disks now, start with 4 disks, don't think I'll start with 2 and add another later, and the last one later again. not a smart idea.

    other than vdevs being very limited once built, everything on top so highly dynamic. There's also some key pool settings that can't be changed once built, (ashift is the key one to get right)

    Matt's been working on online stripe resilvering for many years now, don't hold your breath for it. (nothing against Matt, he's a wizard with filesystem coding). just it's a big problem, that's bigger than you think it is. it goes against a lot of the things baked into ZFS from the beginning.
     
    jhunt31 likes this.
  3. OP
    OP
    daehenoc

    daehenoc Member

    Joined:
    Nov 4, 2005
    Messages:
    2,887
    Location:
    Mt Gravatt E, BNE, QLD
    HUH. Very interesting!

    I am so educated, thank you for the information! I was very heavily implying that I would be able to purchase one 12Tb drive, add it to the system, create the pool on it, copy my data over to it, start removing my mdraid drives while adding 12Tb drives to the zpool. I will not do that!

    Yes. I will :)
     
    Last edited: Aug 18, 2021
    jhunt31 likes this.
  4. gea

    gea Member

    Joined:
    May 22, 2011
    Messages:
    228
    Raid transformation on ZFS ex Z1->Z2 is not possible with ZFS.
    Vdev remove is in ZFS (OpenZFS mirror only, Solaris removes also raid-z)
    Raid Z expansion (ex add another disk to Z2 vdev) is coming.

    First Alpha preview:
    https://github.com/openzfs/zfs/pull/8853
     
    Last edited: Aug 19, 2021
  5. jtir

    jtir Member

    Joined:
    Oct 1, 2003
    Messages:
    1,181
    Location:
    Sydney
    I now run all mirror vdev setup instead of RAID-Z[1,2,3] , so I add two disks at a time to grow the pool.
    Also much more straightforward to resilver when replacing a failed disk.
     
  6. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    15,473
    Location:
    Canberra
    that link is old news. it's now in code-review and pending no major issues should make the 2.2 release next year.
    https://github.com/openzfs/zfs/pull/12225

    it isn't all sunshine and lollipops though. the fine print shows that while it re-stripes the blocks across your new disk, any existing data remains in the stripe config it was written in. again ZFS 'RAID' isn't 'RAID' in the conventional sense. one of the baked in features I mentioned above.

    e.g. you have a 5-disk Z2 vdev, and add a 6th disk. you gain a disk of free space, all your data is now spread across the 6 disks, but anything pre-existing is still in the old 3 data +2 partiy stripe arrangement, any new data, and any data that gets modified will be written in the new 4d+2p stripe format. you don't lose any redundancy, but you still have a slightly higher parity overhead on the old data. 40% vs. 33%.
     
    gea likes this.
  7. Doc-of-FC

    Doc-of-FC Member

    Joined:
    Aug 30, 2001
    Messages:
    3,396
    Location:
    Canberra
    All this talk and nobody has even mentioned the elephant in the room, ZFS is dead.

    It's yesterday's technology and is centric on block level management, newer filesystems all use extents on disks.

    And before you ask, yes I've used ZFS extensively over the years, check out the NG FS thread for some laughs. ZFS lacks a crucial ability to fix fragmentation and that's the ability to rewrite block pointers I to the filesystem and metadata, this shit gets all screwy when your snapshots also point to a block and dedupe is impossible there after.

    For 99% of let's use ZFS ideas, you will probably get better performance out of BTRFS mirrors or striped mirrors.

    L2ARC, is for frequently hit blocks that live in the ARC ghost list, to hopefully get them back into some cache again

    SLOG for staging writes and coalescing them to disk.

    The fact that these capabilities exist points to how old as a concept ZFS is, it's dead, stop finding uses for it.
     
    NSanity likes this.
  8. grs1961

    grs1961 Member

    Joined:
    Jan 21, 2005
    Messages:
    528
    Location:
    Melbourne
    Just like the mainframe is dead, right??

    And UNIX?? OS/400 was going to destroy UNIX, I remember being told by oh so many clever people back in the 1980s. (To be fair, OS/400 is still around as well, it's called IBM i these days.)

    If and when I can see an alternative that is as reliable as RAIDZn, maybe I'll look into it, but for the moment I'll keep all my critical data on my RAIDZ3 arrays[1], and use Solaris to keep it all under control.

    1 - And back it all up onto tape.
     
  9. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    18,441
    Location:
    Brisbane
    It is. Technical Debt isn't life.

    mostly supporting the previous statement.

    Dead Filesystem that was:

    * Architected *just* before everyone woke up to the fact that a single node as a fault domain is a bit shit
    * Never Finished by its original creators
    * Abandoned for 2-3 years (#bless Oracle)
    * Found the best support in a post Sun world in FreeNAS forums
    * Finally found enough support in Linux (at a time when BTRFS wasn't ready) to build something that the kernel will never fully support due to Oracles litigious nature and that CDDL is not GPL.
    * Has since been surpassed by alternatives but a bunch of people who heard about it one time think its really cool - despite really not understanding what its good for, what its not, how to size/spec it, how it needs to be fed and watered, and most importantly how to fix it when it does get its panties in a knot.

    Depending on your usecase - I'm ok with this.
     
    Last edited: Sep 20, 2021
  10. jtir

    jtir Member

    Joined:
    Oct 1, 2003
    Messages:
    1,181
    Location:
    Sydney
    Surprisingly QNAP have https://www.qnap.com/quts-hero/en-au/ based on ZFS (in beta).
    Synology are backing BTRFS instead.

    OpenZFS is still actively developed - https://github.com/openzfs/zfs but yes Sun's ZFS is dead, and Oracle is known for their litigious behaviour.

    ZFS was one the earlier pioneers of Copy on Write file systems/volume managers.
     
  11. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    18,441
    Location:
    Brisbane
    If you're linux based and you aren't backing BTRFS, you already lost. Its literally a waste of time to do anything else medium-long term. BSD is in an interesting position given they aren't at odds from a licensing perspective - and the BSD kernel maintainers obviously have a lot of time and alignment w/ ZFS.

    Still - if the only reason to use BSD is because of a dead Filesystem and an aging approach to firewalls, why bother.
     
  12. jtir

    jtir Member

    Joined:
    Oct 1, 2003
    Messages:
    1,181
    Location:
    Sydney
    It will be interesting, as TrueNAS Scale will now port from FreeBSD to Linux on ZFS.

    Perhaps there will be a fork or future release of TrueNAS with BTRFS options?

    I have been an avid fan of FreeNAS since v0.7 and have data on pools brought across successfully each incremental version with no issues
     
  13. Doc-of-FC

    Doc-of-FC Member

    Joined:
    Aug 30, 2001
    Messages:
    3,396
    Location:
    Canberra
    Enough said, right there.

    RAIDZ is cheap and nasty and N-WAY mirrors are far more reliable and performant during an online rebuild, this has been known for a very long time.

    RaidZ is also akin to 5-6x on write amplification, all to make a filesystem larger for people who want to cheap out on usable storage space at the expense of write performance.

    BtrFS has raid levels for metadata and data, e.g. raid1c4 for metadata, write my metadata to 4 seperate drives and then maybe raid10 my data, not sure why you'd do that though, however some people are using raid1c4 metadata and raid5/6 for data.

    Enjoy your prehistoric filesystem, I hope you've got ECC.

    And Netapp still managed to build a defragmentation capability into the atomic filesystem - along with snapshots, something ZFS never got. Oracle snapped up SUN and likely also paid Netapp for the alleged COW filesystem IP violations.
     
  14. jtir

    jtir Member

    Joined:
    Oct 1, 2003
    Messages:
    1,181
    Location:
    Sydney
    https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/

    Above article explains benefits of mirror vdevs over RAIDZ.


    For most home users like myself who are not running intensive homelabs or workloads, NAS performance is adequate on RAIDZ (I use 5x mirror vdevs personally).

    Netapp WAFL/Ontap, if there was an open/free edition I would evaluate and consider for home use.

    There is special vdevs for ZFS metadata, I would consider this function to be on parity with BtrFS
     
  15. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    15,473
    Location:
    Canberra
    you can also configure different copy levels for metadata and user data.

    I use this on my external backup HDDs, to provide some level of redundancy (for UREs, of course no extra protection for complete hardware failures) for what is otherwise a single disk vdev.
     
    jtir likes this.
  16. grs1961

    grs1961 Member

    Joined:
    Jan 21, 2005
    Messages:
    528
    Location:
    Melbourne
    And the small fact that Fujitsu re-opened their mainframe division a few years ago because, instead of demand reducing it was increasing?

    You are Rod Speed, AICMFP.
     
  17. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    45,385
    Location:
    Brisbane
    Where "demand increasing" = "the one customer who uses it decided paying millions to maintain the technical debt was cheaper than the tens of millions to re-architect a smarter solution".

    This. It's the IT equivalent of necromancy.
     
    fredhoon and NSanity like this.
  18. grs1961

    grs1961 Member

    Joined:
    Jan 21, 2005
    Messages:
    528
    Location:
    Melbourne
    You really don't know much about what's going on in the IT world, do you?

    As I said before, you are Rod Speed, AICMFP.
     
  19. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    45,385
    Location:
    Brisbane
    Heh. Ya got me. Never touched a dang computer in my life.

    As I said before, you are a necromancer. That name confirms it.
     
  20. tensop

    tensop Member

    Joined:
    Mar 26, 2002
    Messages:
    1,542
    as usual this thread is full of toxic trolling from f25, people trying to show off how smart they are ;)

    dudes talking about a little nas array for its uncle torrence, and in here we have people talking about customers and crap like this is some sort of enterprise deployment for the ato
     
    Last edited: Sep 22, 2021
    darkanjel and daehenoc like this.

Share This Page

Advertisement: