aokmans ZFS adventures

Discussion in 'Storage & Backup' started by aokman, Jul 3, 2010.

  1. aokman

    aokman Member

    Joined:
    Jul 12, 2001
    Messages:
    12,546
    Location:
    Melbourne
    Well rather than start a million threads i thought i would just have the 1 from start to finish that can hopefully help some people out and not fall into the holes i did :)

    Initial hardware config:

    E5400 @ 2.7Ghz
    ASUS P5G41C (ICH7)
    2GB Ram
    3 x WD20EARS Drives
    4 x 2TB Hitachi 7k2000
    4GB USB for OS purposes

    [​IMG]

    After some intial testing and reading i setup a freeNAS server, everything was running great in single drive UFS configurations seeing 100MB/s transfers via FTP from the interface. However once i setup ZFS raidz1 speeds plummeted to around 25-30MB/s.

    FreeNAS have since added advanced 4k sector formatting for ZFS which helped slightly but it is still quite slow. Eventually i gave up on the EARS drives as there are too many issues and unknowns with them and i had no troubles disabling the head parking functions on them.

    I replaced the config with 4 x 2TB Hitachi 7k2000 drives and everything went smoothly in freeNAS. Speeds increased to 45-55MB/s which was better but there is still more performance to be had in the config. This lead me to the 2GB of RAM which i thought would be ample for a simple fileserver.

    Well i was wrong on both fronts i loaded 4GB instead and speeds went even higher but the real results showed on the system itself regardless of the fact that it never actually showed the existing 2GB being fully utilized?

    These tests were alternated so i did 1 test added ram, then removing it and added it again to rule out any caching skewing the results. However with both ram configs i am seeing a "Pulsing" data transfer on the network interface and you can hear it from the HDD's also as it stops to write then resumes again which is hard to work out...

    [​IMG][​IMG]

    Finally i installed Nexenta to take for a spin, everything went relatively smoothly except that it takes quite a bit longer to install and seems much more chubby all around compared to freeNAS. Everything installed fine and i was even able to import the ZFS pool into it but i recreated it anyway.

    Once all the groundwork was done i am seeing FTP speeds from 70MB/s-100MB/s depending on the file type which im very pleased with. Though i have to say Nexenta seems to have a much rougher interface than freeNAS. Alot of functions dont seem to work like the realtime status monitors and little things like changing MTU settings which does nothing. Finally i cant seem to find any support for AFP (time machine backups) or power management settings like freeNAS has?

    Anyone have anything to offer about where to head from this point, i love the features of freeNAS but the speed of Nexenta :confused:
     
    Last edited: Jul 4, 2010
  2. Stanza

    Stanza Member

    Joined:
    Jun 27, 2001
    Messages:
    2,878
    Location:
    Adelaide
    do a search about limiting the amount of memory ZFS uses in Freenas.

    Thats probably where your problems lay, as with less than 4gig Freenas and ZFS can be a bit of a slug
     
  3. sreg0r

    sreg0r Member

    Joined:
    Jul 9, 2001
    Messages:
    1,146
    Location:
    Melbourne
    ZFS is very RAM hungry, so running freenas with only 2GB was always going to inhibit you. freenas probably takes up a fair bit of that for its ramdisk + normal ram usage and that doesn't leave you with much for ARC cache.

    Stanza, ZFS should dynamically allocate RAM to the ARC Cache as it needs it. although you can check how much it is using with kstat zfs. Setting c_max to a smaller value and benching that could at least rule it out.


    I'm not sure how to do it manually but if you benchmark using filebench you can specify directio which will bypass the cache and you will have a better idea of how the drive performs. sequential read/write and random read/write stats are always useful to analyse.

    It certainly appears as if something is not quite right. 4x7200rpm drives should be pushing 200MB/s and beyond.

    You don't happen to have dedup or compression on do you?
     
    Last edited: Jul 4, 2010
  4. OP
    OP
    aokman

    aokman Member

    Joined:
    Jul 12, 2001
    Messages:
    12,546
    Location:
    Melbourne
    i will give that filebench a go :) not sure if i will get it running under nexenta though as the console wont echo back about commands for some reason... ill be adding another 4GB in tomorrow to bring it up to 6GB which will hopefully be enough. Not sure what else would be bottlenecking the output, They are configured in RAID5Z1. The USB key wouldn't be holding anything back would it?

    One other thing i noticed is that during the boot of freeNAS it reported all drives at SATA150 instead of 300 even though they support it and so does the controller.

    In Nexenta dedup and compression are off and set to 128k. I did not see any options for these in freeNAS however as it was a very basic list of options...
     
    Last edited: Jul 4, 2010
  5. sreg0r

    sreg0r Member

    Joined:
    Jul 9, 2001
    Messages:
    1,146
    Location:
    Melbourne
    I imagine freenas is also using the default settings which is 128k record sizee with dedup compression off.

    The contents of the USB drive are just dumped to ramdisk so it should get little to no usage once booted up. But with only 2GB of RAM this is using up extra memory that you don't want to lose, but with 6GB this wont be an issue.

    I'm not sure how difficult they make it to run a 3rd party benchmark on those OS' but you could always try using the opensolaris livecd which can also be installed to USB.

    alas, my desktop is now up and running with a new Vertex 2 SSD and le tour is about to start!
     
  6. OP
    OP
    aokman

    aokman Member

    Joined:
    Jul 12, 2001
    Messages:
    12,546
    Location:
    Melbourne
    heres another little update im trying out some various settings as a user put up a web plugin for freenas to tweak the ZFS settings so i was able to limit ram to 3GB and change a couple of other variables...

    going to try filebench shortly once i sort it out...

    Leaving the ram setting on default or 2GB causes the throughput hole across the network as it drops out and comes back in... i am only running 4GB so far as a single 4GB module is pritty pricy.

    freenas:~# dd if=/dev/zero of=/mnt/array1/testfile bs=8192k count=1000
    1000+0 records in
    1000+0 records out
    8388608000 bytes transferred in 58.727706 secs (142839020 bytes/sec)

    freenas:/mnt/array1# dd if=/dev/urandom of=test.100 bs=1m count=1000
    1000+0 records in
    1000+0 records out
    1048576000 bytes transferred in 16.579492 secs (63245363 bytes/sec)
     
    Last edited: Jul 4, 2010
  7. Panadol

    Panadol Member

    Joined:
    Jul 10, 2001
    Messages:
    207
    Location:
    Toowoomba, Qld
    I had a few similar dealings a while back. After testing various configs with freenas (trying to set freenas arc ram usage) I ended up going with opensolaris as its zfs version is alot newer, dedup and compression isnt available in the current freenas zfs version afaik. Like you though, I much prefer the web interface of freenas.

    Here's a couple things I've found with zfs. With the write pulses, you can improve the performance by adding more ram, increasing the time between bulk writes, anywhere up to 30 seconds between writes with enough ram. To fix the problem completely however you'll need a fast zil (ssd). But I don't that's in the scope of your project.
     
    Last edited: Jul 5, 2010
  8. OP
    OP
    aokman

    aokman Member

    Joined:
    Jul 12, 2001
    Messages:
    12,546
    Location:
    Melbourne
    yeah its a shame freeNAS lost so much support as i really love its interface but from i have read its being developed still unofficially running ZFS 13?

    i kept tweaking today and settled on 3gb allocated for ZFS through the tweaker and set vdev.max_pending to 10 from 35.

    i tried prefetch and ZIL disabled but they had no impact on performance really. Only other 2 i have available through the interface are txg.timeout (30) and vdev.min_pending (4)

    however performance has improved quite a bit ive been sticking with the 1 file as they vary speeds greatly across the network with different ones and FTP sits at a solid 52MB/s now... Another gets up to 75-80MB/s

    heres a comparison graph of before and after tweaking also, it shows the holes in the throughput. Ill have to look into the SSD zil you talk about but i guess the best way is to go to solaris but i love this web interface so much...

    [​IMG]
    Click to view full-sized image!
    Hosted by UGBox Image Store
     
    Last edited: Jul 4, 2010
  9. sreg0r

    sreg0r Member

    Joined:
    Jul 9, 2001
    Messages:
    1,146
    Location:
    Melbourne
    The default vdev.max_pending is 10 in the latest version of ZFS and even that is designed more for enterprise hardware. I'm currently running with it set to 1 and although my speeds didn't change, system responsiveness is much better.

    The freenas gui is nice but the cli for zfs is so simple to use you wont miss it much. Once you have it setup you will barely need to look at it anyway, and if you do opensolaris has an easy to use built-in vnc client.

    While RAM is expensive it is still the cheapest and easiest way to improve perforamance. I notice it struggling with 4gb, particularly when I'm running other apps and a windows VM at the same time. I Just ordered another 4gb (max motherboard can take) and hope that will suffice.
     
    Last edited: Jul 4, 2010
  10. OP
    OP
    aokman

    aokman Member

    Joined:
    Jul 12, 2001
    Messages:
    12,546
    Location:
    Melbourne
    it certainly is a great way to boost zfs performance tho id probably change the board to a fullsize DDR3 board as this only has 2 slots and 4gb is almost 4 times the price of 2gb (i can get a new board and 2 x 2gb for the same price almost)

    i shall have a look at opensolaris as it could be doable depending on its support for AFP and power saving features tho im sure its all there :) tomorrow can be another day another play :)
     
  11. Panadol

    Panadol Member

    Joined:
    Jul 10, 2001
    Messages:
    207
    Location:
    Toowoomba, Qld
    Something else I noticed, you've got a spare pci-e slot there, just waiting for one of the cheap perc 5i cards that are going cheap on ebay at the moment (http://forums.overclockers.com.au/showthread.php?t=879827). ~$30 will get you another 4 sas ports for future expansion. Well worth it IMO. The other bonus is that the perc cards based on the lsi 1068e chipset have good support for solaris and bsd.
     
    Last edited: Jul 5, 2010
  12. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    2,929
    What power saving features are you after?
     
  13. Stanza

    Stanza Member

    Joined:
    Jun 27, 2001
    Messages:
    2,878
    Location:
    Adelaide
  14. OP
    OP
    aokman

    aokman Member

    Joined:
    Jul 12, 2001
    Messages:
    12,546
    Location:
    Melbourne
    good support but they dont support JBOD pass-through unfortunately :( there will be a expansion card going in though for more HDD's once i have the groundwork done in ZFS and i know where i stand :)

    Main power save features i want are just HDD spindown when the array is not in use for long periods and CPU throttling if possible, both features worked great on freeNAS with the CPU backing down to 400mhz when idling. Power consumption was about 50w when it was idling :) ive downloaded opensolaris for a play tonight :) atleast i get a gui i suppose, for day to day stuff :)

    cheers for that i noticed that one briefly in my browsing but i shall have a better look over it :)
     
    Last edited: Jul 5, 2010
  15. Stanza

    Stanza Member

    Joined:
    Jun 27, 2001
    Messages:
    2,878
    Location:
    Adelaide
    Might want to have a read of that thread again

    These cards ARE SAS/SATA JBOD Only.... no RAID
     
  16. OP
    OP
    aokman

    aokman Member

    Joined:
    Jul 12, 2001
    Messages:
    12,546
    Location:
    Melbourne
    the perc5i is a hardware raid card isn't it, all the info i could find on it said it will not pass devices on to the host unless you set each hdd up as a individual raid?
     
  17. Panadol

    Panadol Member

    Joined:
    Jul 10, 2001
    Messages:
    207
    Location:
    Toowoomba, Qld
    They operate in IT mode, sure you can do raid, but they're mainly for jbod. Hence perfect for zfs. In fact, sun build their solaris thumper boxes based on this chips for storage.
     
  18. grs1961

    grs1961 Member

    Joined:
    Jan 21, 2005
    Messages:
    496
    Location:
    Melbourne
    Just a bit of a head's up from a recent OSUG meeting - If you are going to run with a separate ZIL device, make sure it is mirrored, because if it goes away, you are stuffed.
     
  19. OP
    OP
    aokman

    aokman Member

    Joined:
    Jul 12, 2001
    Messages:
    12,546
    Location:
    Melbourne
    Oh thanks for that sorry I looked over that thread I thought u were talking about something else completely now i see what you mean :)
     
  20. flain

    flain Member

    Joined:
    Oct 5, 2005
    Messages:
    2,306
    Location:
    Sydney
    Thats one reason to run it on opensolaris under a DEV build. After pool version 19 thats no longer a problem. Best place to get the DEV builds is http://genunix.org/ if anyone is having trouble finding them..
     

Share This Page

Advertisement: