1. OCAU Merchandise is available! Check out our 20th Anniversary Mugs, Classic Logo Shirts and much more! Discussion in this thread.
    Dismiss Notice

FreeBSD + ZFS + Web-Interface ready for testing

Discussion in 'Storage & Backup' started by sub.mesa, Jun 23, 2010.

  1. Samuka

    Samuka Member

    Joined:
    Jul 9, 2004
    Messages:
    367
    Location:
    Melbourne
    Seems there is on avg $100AUD from here to there on ECC 8GB DDR3 ram.
    http://www.staticice.com.au/cgi-bin/search.cgi?q=8gb+ecc+ddr3+-non&spos=1

    MY brother lives in the UK but it seems its expensive over there also from what I can see on some sites over there but then I dont know what is a good one to look at.

    When I do put something together with new parts I will put it high on the list. It will be a project over time in stages im guessing. I just got a RR 2680 Raid card of someone here tonight. Stage 1. best parts I have....Stage 2 New Parts :D

    Thank again
     
  2. Samuka

    Samuka Member

    Joined:
    Jul 9, 2004
    Messages:
    367
    Location:
    Melbourne
    Had a go with the on a more recent PC. 4GB E6750. Got it up and running on the LiveCD but as this is not my prefered load methof as i needed the CD drive else where so i tried getting it to run from USB. This did not work when I tried with the binary img. When running from the LiveCD where is the setting info stored?

    i tried using winimage but could not work out how to use it,. every time I tried i stated that I did not have a valid fat/fat32 partition. I tried again with win32diskimager. This copied to the USB and did boot but failed halfway through the install.

    Are you looking to create a easy bootable USB setup or install from CD to run on USB?
     
  3. OP
    OP
    sub.mesa

    sub.mesa Member

    Joined:
    Jun 23, 2010
    Messages:
    271
    Location:
    Europe
    New Release: 0.1.7-preview3

    My Christmas present to you guys: preview3 release!
    • ZFSguru 0.1.7-preview3 LiveCD (.iso) containing experimental ZFS v28 system image
    • ZFSguru 0.1.7-preview3 web-interface available for web-update
    • FreeBSD 9.0-001 amd64 system image containing experimental ZFS v28 code
    Warning: the ZFS v28 system image, including the one included on the LiveCD, is considered very experimental, and you should not use this on your existing valuable data unless you got a very good backup! This release is for testing only!


    System version 9.0-001 amd64 (bleeding edge)
    • based on FreeBSD 9.0-CURRENT 20101221 code + ZFS v28 patch + head patches
    • highly experimental 'bleeding edge' release; DO NOT USE ON REAL DATA!
    • it actually does work, however! Imports your existing pools fine.
    • supports ZFS v28 including de-duplication and RAID-Z3
    • booting support in Virtualbox proved difficult; Virtualbox does not yield access to other devices than the boot device. Please test more thoroughly in the real world! Create a RAID-Z2 and boot from it, have multiple pools of different ZFS versions, etcetera.
    • has the new MPT Fusion 2 'mps' driver supporting LSI SAS2008 and other 6Gbps-family chips. This adds support for SuperMicro USAS2 6Gbps HBAs.
    • has a few fixes for disk labeling (no more gptid labels)
    • has a new menu instead of login prompt, with very basic functionality; but it does display the IP address the web-interface is running on!
    • updated all packages, including Samba to 3.5.6 (up from 3.4.x); this may have performance implications.

    ZFSguru web-interface version 0.1.7-preview3
    • fixed issues with ZFSv28 new data format
    • adds De-duplication support to Files page, on pools and systems that support it.
    • changed automatic memory tuning to new defaults (kmem=1.5x the RAM)
    • fixed automatic memory tuning on Root-on-ZFS installations
    • some important of under-the-hood changes

    How to upgrade?
    • first update your Web-interface on the System->Update page, to preview3 version
    • next, System->Install page will allow you to perform Root-on-ZFS installation and download the newly available 9.0-001 system image to your system and use it for installation.
    • after installation reboot and you should run the new experimental ZFS v28 system!

    Note that in order to use de-duplication, you must either create a pool of at least pool version 21, or upgrade an existing pool with the zpool upgrade command. Note that if you do this, you cannot go back to stable ZFS versions like the ZFS v15 the stable ZFSguru versions use.

    So please test, but be careful not to burn your fingers! Bleeding edge can bite you. :)

    If there are no critical issues with this release, it will be branched 0.1.7 (final) and next release will be 0.1.8 in new year. The 0.1.7 final will contain the 'stable' ZFS v15 system image based on FreeBSD 8.2-RC; so this really is a testing-only release.

    Cheers!
     
  4. BluBoy

    BluBoy Member

    Joined:
    Jan 20, 2006
    Messages:
    1,899
    Location:
    Brisbane


    Mate... I dunno how you did it, but this is awesome! I needed to build a NAS over the weekend and have had enough experience now to last a life time.
    To be honest, I wasn't expecting much with the old P4 era hardware I decided to dust off (literally!!) for this project, but it is flying along!

    Started off with NexentaStor. BEAUTIFUL GUI, very simple to go through and administer fully from here. However, I was only managing 10-15MBps over gigabit ethernet. Also, I'm not a fan of Solaris. I find it does 'out of the box' stuff wonderfully, but step outside the box, and there just isnt the same level of support that you can find with linux/bsd.
    I did briefly try a few different installs with this, but could not fix the transfer rate - As a noob, I was impressed with the ease you could just import a pool after my original OS HDD died... Who needs a mirrored OS, everything is back to normal in 10 minutes anyway! (*JK'ing people. Back your shit up anyway!)

    Not happy with the throughput, I moved onto a native FreeBSD 9-Current install (following your instructions on the website).
    Amazed at how simple ZFS from the CLI is. Such a powerful tool which is intuitive and fairly common sense based commands. I expected it to be more complicated and found myself googling for second and third opinions on how to do things just cos I didnt believe it. Incredibly, transfer speeds were now going along at 70MBps!!
    ... But alas, I could not get the zfsv28 stuff into the kernel, which I needed for bragging rights over a mate running nexentastor still. And moved onto my next attempt.

    I should point out that only at this stage, did I do the 4Kb align for my WD 2Tb drives...

    Anyway, spent ages reading over the [H] forum thread, and decided to give your OS a go.
    Installed without a glitch, imported the pool no worries, upgraded the pool to zfsv28 perfectly... I must admit that I am 'only' getting about 55MBps now, but I a happy with this, and put it down to the additional features of v28.

    Have since installed SABnzbd, Sickbeard and CouchPotato on it and they are running along perfectly (Massive queue actually, this box is getting a stress test early on with it all). This was the big requirement for me... and had been causing trouble on Solaris.

    Next steps... Get L2ARC (maybe ZIL) going on the OS drive.
    Then just find a way to fill up the drives, so I can add another pool and see how far I can push this thing.

    My constructive feedback for you... There is still a lot I have to do from the CLI. NFS/SMB setup/users was a big one. I really wish I could have the nexentastor GUI on your backend! Having said that, the CLI work is simple to do... But you risk people just using the native FreeBSD install once that has v28 inbuilt.

    Soooo, any ideas on a release date for v0.1.8 ??? :)

    And my fave two GENERAL ZFS bookmarks from this adventure for others:
    http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
    http://www.filibeto.org/~aduritz/truetrue/solaris10/zfs/ZFSNinja-Slides.pdf
     
  5. BluBoy

    BluBoy Member

    Joined:
    Jan 20, 2006
    Messages:
    1,899
    Location:
    Brisbane
    Oh and for those interested:
    I just enabled dedup for my backups... Transfer speed drops from about 55MBps to about 20MBps. I've been aiming to keep it above that 50MBpsmark. RAM and CPU have let me down :(

    Might give compression a go here, apparently a lot less overhead and still a decent amount of 'additional' space provided.

    *NB: This is a SMB directory copy with SABnzbd (downloading at 2Mbps) and Sickbeard (sorting through some media) doing their thing in the background... Current CPU usage is "250" to "400" percent :)
     
  6. sreg0r

    sreg0r Member

    Joined:
    Jul 9, 2001
    Messages:
    1,146
    Location:
    Melbourne
    Depending on your content dedup/compression just adds to your overhead with no noticeable space saving or speed-up. I added a separate pool for a documents backup but considering my documents use up less than a 20th of the space my media does it wasn't really worth it, especially considering the cost of HDD's these days.
     
  7. samarium

    samarium Member

    Joined:
    Jun 6, 2007
    Messages:
    475
    Hope you have sufficient RAM to keep you dedup table in memory. If not, then speed takes a dive as you incur extra IO just to check the dedup table.

    zdb -DD should give you some info
     
  8. barbz

    barbz Member

    Joined:
    Dec 12, 2006
    Messages:
    272
    Location:
    Brisbane
    Whats the best setup for 10 disks connected to two USAS-l8i's?

    Cheers
    Paul
     
  9. young_einstein

    young_einstein Member

    Joined:
    Jan 19, 2006
    Messages:
    2,568
    Location:
    Caroline Springs, VIC
    A single RAIDZ2 pool is what I'd go with.

    That'll give you redundancy for any two drives to fail, and leave you with a 'usable' capacity of 8x drives. [IE - 2TB disks = 16TB usable]
     
  10. barbz

    barbz Member

    Joined:
    Dec 12, 2006
    Messages:
    272
    Location:
    Brisbane
    Ta - any idea how that would be impacted by using 4kb sector disks?

    Atm I am running 8 x 1tb's in a raidz1...time to upgrade to 2tb disks
     
  11. young_einstein

    young_einstein Member

    Joined:
    Jan 19, 2006
    Messages:
    2,568
    Location:
    Caroline Springs, VIC
    It shouldn't be impacted at all.
     
  12. deva_da_man

    deva_da_man Member

    Joined:
    Aug 30, 2007
    Messages:
    347
    Location:
    Brisbane
    how is this project coming along?

    I've recently just tried to setup my own ZFS freeBSD server, with only small amounts of luck.

    currently just having problems with samba. it seems my macbook doesn't like to report its hostname correctly to my DHCP server (router) which is giving me hassles trying to access the ZFS share from my laptop...

    anywho, about to give your web interface a go. Any idea when the 'stable' 0.2 release will happen?

    Do you need any help with anything? I'm more than happy to help out in any small way that I possibly could

    thnx
     
  13. Mad Mike

    Mad Mike Member

    Joined:
    Jun 30, 2001
    Messages:
    780
    Location:
    Melbourne
    It could be awhile, as sub.mesa is MIA at the moment. There is some discussion on his website forums about it. Though it seems most are moving on to other things. We will just have to see if sub.mesa appears again.
     

Share This Page

Advertisement: