Goodbye RAID, hello ZFS on Linux

Discussion in 'Storage & Backup' started by saba, Jan 31, 2012.

  1. dyu

    dyu Member

    Joined:
    Mar 1, 2012
    Messages:
    4
    Agreed

    Agreed. zfs on FreeBSD is rock solid (I just set it up a week ago). I've tried zfs on both ubuntu 11.10 (4 months ago) and 12.04 (3 weeks ago, rc9). The rollback functionality does not work in ubuntu (while it works in FreeBSD).

    If you want a fully functional zfs, go with FreeBSD.
     
  2. CaptainBlame

    CaptainBlame Member

    Joined:
    Jan 19, 2011
    Messages:
    286
    Everything about FreeBSD is rock solid, I'm still running my first install from 2009 (which has root on ZFS). Once you understand how the base system works and how its independent from the packages, there is really no reason to reinstall FreeBSD.
     
  3. DaveQB

    DaveQB Member

    Joined:
    Apr 30, 2002
    Messages:
    2,695
    Location:
    Narellan, Sydney
    Stuff :leet:

    Most services they offer or of the small file type, DNS, Mail etc.


    What what what???!!! :eek:
     
  4. CaptainBlame

    CaptainBlame Member

    Joined:
    Jan 19, 2011
    Messages:
    286
    So what does virtualisation give you? None of those services are particularly hard to configure under one OS.
     
  5. DaveQB

    DaveQB Member

    Joined:
    Apr 30, 2002
    Messages:
    2,695
    Location:
    Narellan, Sydney
    With fear of derailing this thread....yes you can, but virtualisation offers isolation. Most companies that employ virtualisation are doing so for the isolation of processes and users etc. Easier management and aids in security etc.

    I could run FreeBSD and use virtualbox on that allowing me to run a more mature ZFS implementation. But I do love Proxmox so much....
     
  6. CaptainBlame

    CaptainBlame Member

    Joined:
    Jan 19, 2011
    Messages:
    286
    If all you want to do is isolate processes and users, thats exactly what a freebsd jail does.

    As for virtualisation aiding security, I like Theo's response:
    http://www.tylerkrpata.com/2007/10/theo-de-raadt-on-x86-virtualization.html
    https://forums.virtualbox.org/viewtopic.php?p=17930
     
  7. The Sentinel

    The Sentinel Member

    Joined:
    Jan 30, 2002
    Messages:
    2,838
    I've been playing with ZFS under Solaris Express for the last year or so (on and off). This was via napp-it as well as directly from the command-line.
    Reports of the performance hit running via FUSE in Linux meant I didn't consider using it under Linux. After seeing this thread I've come back to it and it looks very promising!

    With 6 1TB drives in RAIDZ2 I get around 195 MB/sec on a scrub under Ubuntu Server 12.04 LTS.
    I'll post some more detailed performance stats and config details a little later for those that are interested.

    This next question is for those of you running it in a Norco RPC-2440 case (or some other setup with lots of drives):
    How do you easily identify your drives when removing/replacing them?
    (i.e. Pulling a drive to simulate a drive going offline).

    Solaris has the format command to identify the UID of the drive and napp-it had a function to make the drive LED flash to aid in identifying the drive. I don't find an equivalent for Linux other than running a command to flog the drive so the activity LED comes on.

    This currently isn't a big issue as I am still testing but it would obviously be an issue on drives with real data on them.
    I've got the drive serial/model number and drive bay recorded in spreadsheet but there must be a better way to do this.
     
  8. DaveQB

    DaveQB Member

    Joined:
    Apr 30, 2002
    Messages:
    2,695
    Location:
    Narellan, Sydney

    This may tip you off for a better approach

    Code:
    ls /dev/disk/by-                                                                                                                                                                            
    Completing files
    by-id/     by-label/  by-path/   by-uuid/
    

    "by-id" might be best for you in your case.
     
  9. DaveQB

    DaveQB Member

    Joined:
    Apr 30, 2002
    Messages:
    2,695
    Location:
    Narellan, Sydney
    So as to not take this thread off topic I have started a new thread on this matter CaptainBlame. I am keen to discuss this topic.

    http://forums.overclockers.com.au/showthread.php?t=1043489
     
  10. The Sentinel

    The Sentinel Member

    Joined:
    Jan 30, 2002
    Messages:
    2,838
    Wow.
    Just wow.

    On the same hardware under Solaris Express I was unable to get more than 120 MB/s during resilvering (at any stage).

    As you can see below ZFS is resilvering two drives at the same time.
    The speed bounces around a bit but never gets below 205MB/s

    For consistency I filled the pool up with the same 750GB of files I used when testing in under SE about 9 months ago or so.

    Note: This is on a Xeon X3450 with 16GB of RAM.

    Code:
    Sun Jul 29 21:15:22 2012
    
      pool: storage
     state: ONLINE
    status: One or more devices is currently being resilvered.  The pool will
            continue to function, possibly in a degraded state.
    action: Wait for the resilver to complete.
     scan: resilver in progress since Sun Jul 29 21:11:59 2012
        44.9G scanned out of 735G at 227M/s, 0h51m to go
        6.42G resilvered, 6.11% done
    config:
    
            NAME                                           STATE     READ WRITE CKSUM
            storage                                        ONLINE       0     0     0
              raidz2-0                                     ONLINE       0     0     0
                scsi-SATA_ST31000340NS_3QJ04D81            ONLINE       0     0     0
                scsi-SATA_ST31000525SV_5VP5SZ25            ONLINE       0     0     0
                scsi-SATA_ST31000525SV_6VP51MW0            ONLINE       0     0  308K  (resilvering)
                scsi-SATA_ST31000525SV_9VP6HD66            ONLINE       0     0     0
                scsi-SATA_ST31000525SV_9VP6JQ6Q            ONLINE       0     0  264K  (resilvering)
                scsi-SATA_WDC_WD10EACS-00_WD-WCAU42027626  ONLINE       0     0     0
                scsi-SATA_WDC_WD10EADS-65_WD-WMAV51657611  ONLINE       0     0     0
    
    errors: No known data errors
     
  11. Jim G

    Jim G Member

    Joined:
    Jun 20, 2010
    Messages:
    1,008
    Location:
    Newcastle
    Ok, so I got sick of Solaris and gave this a shot for our photos/media/VM storage server (around 40TB).

    Old: OI 151a5 + Napp-it
    New: 12.04, no web management

    It's been stable for about a month now - performance has improved and I can configure things so much more quickly now, as I'm a daily Ubuntu Server user but definitely not a regular Solaris user.

    I'm converted. I don't miss the web management except for the DD-based disk identification - I need to look into how to trigger the activity lights for a specific disk so I know which one to pull when they break.

    Otherwise... have had zero dramas and so much more ease of use. If you're on the fence - try it.
     
  12. ewok85

    ewok85 Member

    Joined:
    Jul 4, 2002
    Messages:
    8,112
    Location:
    Tokyo, Japan
    Waiting for everyone to go home so I can shutdown the VMs and give it a try :)
     
  13. Jim G

    Jim G Member

    Joined:
    Jun 20, 2010
    Messages:
    1,008
    Location:
    Newcastle
    Would very much like to hear about how you go with it - I've just noticed that one of the drives has dropped out of an array so I'm probably going to have to look at figuring out which one it is over the weekend before another one goes and I'm left with no redundancy :p
     
  14. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    3,090
    That's some pretty damn awful stats...I would expect closer to 500+ scrubs (assuming no other disk access)....something is not right here.
     
  15. samarium

    samarium Member

    Joined:
    Jun 6, 2007
    Messages:
    475
    If resilvering, then 120 would be close to single disk speed, which would be all I would expect.
     
  16. ewok85

    ewok85 Member

    Joined:
    Jul 4, 2002
    Messages:
    8,112
    Location:
    Tokyo, Japan
    Step 1 - label each drive and caddy, photograph each drive, and memo where each drive is, makes it much easier :)

    So I was a little surprised, this was dead simple. Goal was zfs, infiniband and iscsi

    Installed Ubuntu in the usual time (~30mins while I had dinner), standard install with only OpenSSH. Setup the network, updates, and then added zfs. Dead simple -
    $ sudo add-apt-repository ppa:zfs-native/stable
    $ sudo apt-get update
    $ sudo apt-get install ubuntu-zfs

    Done and done. Then just use the usual zfs commands to set it up.

    http://zfsonlinux.org/faq.html

    I'm fighting with the infiniband stuff atm, but I'll post benchmarks etc when I'm done
     
  17. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    3,090
    oops skimed over that word.
     
  18. xeghia

    xeghia Member

    Joined:
    Nov 7, 2003
    Messages:
    27
    Location:
    Gold Coast
    I've been watching this thread for a while and doing a little experimenting with junk hardware lying around, finally took the plunge and built the new file server.

    I've just installed Ubuntu Server 12.04 and added the ppa, its all working a treat so far. I just want to do a little more reading on how exactly the filesystem and volumes and pools etc all interact, and how best to set up samba before I go any further.

    The pool itself is 6 x 2TB Seagates in raidz2, initial 'benchmarks' using dd while the pool is empty give 360MB/s write and just over 600MB/s read. Scrub also happening just over 600MB/s. Very impressed.

    Looking forward to setting everything up and giving it a good workout.
     
  19. ewok85

    ewok85 Member

    Joined:
    Jul 4, 2002
    Messages:
    8,112
    Location:
    Tokyo, Japan
    A little swearing, a few threats, and we have this!

    Code:
            NAME                                          STATE     READ WRITE CKSUM
            vmtank                                        ONLINE       0     0     0
              mirror-0                                    ONLINE       0     0     0
                scsi-SATA_ST31000528AS_6V                 ONLINE       0     0     0
                scsi-SATA_ST31000528AS_9V                 ONLINE       0     0     0
              mirror-1                                    ONLINE       0     0     0
                scsi-SATA_Hitachi_HDS7210_JP              ONLINE       3    80     0
                scsi-SATA_Hitachi_HDS7210_JP              ONLINE       0     0     0
              mirror-2                                    ONLINE       0     0     0
                scsi-SATA_ST31000340AS_9QJ                ONLINE       0     0     0
                scsi-SATA_ST31000528AS_9VP                ONLINE       0     0     0
              mirror-3                                    ONLINE       0     0     0
                scsi-SATA_ST31000528AS_9VP                ONLINE       0     0     0
                scsi-SATA_ST31000333AS_9TE                ONLINE       0     0     0
              mirror-4                                    ONLINE       0     0     0
                scsi-SATA_ST31000333AS_9TE                ONLINE       0     0     0
                scsi-SATA_ST31000528AS_9VP                ONLINE       0     0     0
              mirror-5                                    ONLINE       0     0     0
                scsi-SATA_ST31000333AS_9TE                ONLINE       0     0     0
                scsi-SATA_ST31000523AS_5VP                ONLINE       0     0     0
    
    Code:
    NAME              USED  AVAIL  REFER  MOUNTPOINT
    vmtank            619G  4.75T    30K  /vmtank
    vmtank/vmstore1   309G  5.05T    16K  -
    vmtank/vmstore2   309G  5.05T  27.3M  -
    The best bit...

    Code:
    Infiniband device 'mthca0' port 1 status:
            default gid:     fe80
            base lid:        0x3
            sm lid:          0x3
            state:           4: ACTIVE
            phys state:      5: LinkUp
            rate:            20 Gb/sec (4X DDR)
    
    Infiniband device 'mthca0' port 2 status:
            default gid:     fe80
            base lid:        0x1
            sm lid:          0x1
            state:           4: ACTIVE
            phys state:      5: LinkUp
            rate:            20 Gb/sec (4X DDR)
    Got two vSphere4 hosts connected to one zvol each with ib SRP - now I need to work out if I can have both hosts connected to both zvols for manual HA, or if I need to edit the scst config before it will work :Paranoid:

    Edit: Dunno if I should be happy or not

    [​IMG]
     
    Last edited: Aug 20, 2012
  20. daehenoc

    daehenoc Member

    Joined:
    Nov 4, 2005
    Messages:
    2,925
    Location:
    Mt Gravatt E, BNE, QLD
    Hihi, I'm thinking about putting zfs on Ubuntu to store my MythTV files.

    I currently use md raid and LVM to store my files. (2x2Tb, md raid1, chopped up into bits with LVM for movies, mythtv recordings, etc)

    I'm interested in deduplication in ZFS, has anyone used this on Ubuntu? I've had a bit to do with implementing Netapp v-series in the past, so I know how that worked, does dedup on ZFS work across all file systems?
     

Share This Page

Advertisement: