Large Storage Build

Discussion in 'Storage & Backup' started by prod, May 9, 2015.

  1. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    2,930
    Last edited: May 22, 2015
  2. ae00711

    ae00711 Member

    Joined:
    Apr 9, 2013
    Messages:
    1,372
    where's this stated? :shock:
     
  3. Dropbear

    Dropbear Member

    Joined:
    Jun 27, 2001
    Messages:
    9,720
    Location:
    Brisbane
    *sings* in a world of imagination
     
  4. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,643
    Location:
    Canberra
    So.

    I've had major discussions with various Nexenta technicians (both here and in the US) about the topic over the past 2 years - across medium to large to stupid huge deployments with dedupe.

    They've told me what works and what doesn't - and in terms of what is safe to "install and not monitor on a daily basis".

    Here's an Oracle consultant of 20 Years - http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-dedupe

    Here's FreeNAS 9.3 Documentation - http://doc.freenas.org/9.3/freenas_storage.html#deduplication


    This guy is saying that if you care about performance 20GB of Ram / TB - http://permabit.com/zfs-dedupe-just-doesnt-make-any-sense/
    Here's some maths on a real world 10TB scenario - https://pthree.org/2013/12/18/zfs-administration-appendix-d-the-true-cost-of-deduplication/

    Heres a guy saying 6GB/1TB - http://jrs-s.net/2015/02/24/zfs-dedup-tested-found-wanting/

     
    Last edited: May 22, 2015
  5. Butcher9_9

    Butcher9_9 Member

    Joined:
    Aug 5, 2006
    Messages:
    2,095
    Location:
    Perth , East Vic Park
    Man 20GB per TB is crazy though. My 100TB would require 2TB of Ram, madness.

    Bit rot is not really an issue with Media Files anyway.You might end up with a slight stutter in a movie or something like that but otherwise its fine. I doubt corporates would need to take a machine to a LAN so the whole 2in1 PC requirement I have is not an issue for them.

    Also I can't fit 100GB of Ram into my PC (or Any non Server board) so ZFS is not workable based on your statements.
     
    Last edited: May 25, 2015
  6. aokman

    aokman Member

    Joined:
    Jul 12, 2001
    Messages:
    12,619
    Location:
    Melbourne
    You can get away with a lot less ram these days, it used to have high utilization in early builds of FreeNAS but seems to have settled down as it has matured. I only give 6gb to my NAS4Free VM with 20TB and shes been happy for a loooooong time over many drive failures, scrubs etc. I don't even see full RAM usage on the system as is and it will happily saturate my gigabit link.

    Anyone who says there are issues with running FreeNAS / NAS4Free in a VM environment is talking out of their ass, I have been doing it for so long that I cant even remember when I started now on ESXi.
     
    Last edited: May 26, 2015
  7. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,643
    Location:
    Canberra
    Again, on the topic of Ram and ZFS today.

    We're upgrading our 16TB Usable Nexenta box from 64GB to 128GB next week. Why? Because it the arc metadata overflowed onto disk and thus all operations required a page to disk. This happened over a period just 16 days;

    Code:
    root@zeus:/# uptime
    13:43:11    up 15 day(s), 18:36,  1 user,  load average: 0.81, 0.77, 0.76

    This isn't specifically a Nexenta/Illumnos thing - this is a ZFS thing.

    And sure, you probably wouldn't notice it if you just had CIFS shares - because who gives a fuck if you have a 150-400ms read/write latency when you're copying some files. But for VM Usage this pretty much spells death to any kind of performance.

    We could probably get along with 96GB of ram just nicely - but Nexenta recommends 128GB, and its reasonably cheap.
     
    Last edited: Jun 5, 2015
  8. LostBenji

    LostBenji Member

    Joined:
    Oct 5, 2007
    Messages:
    6,077
    Location:
    Up a tower somewhere....
    LMAO @ this thread.

    OP asked a question done to death and likely never bothered searching before posting it. Thread gets derailed quickly and goes on and on, no return of the OP.
     
  9. encode

    encode Member

    Joined:
    May 21, 2003
    Messages:
    490
    Location:
    Sydney
    Yep. Seems strange to decide on CPU before motherboard, and I would suggest that motherboard and HBA are the two most important things for a storage build.
     
  10. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,643
    Location:
    Canberra
    Server boards are pretty good typically. You don't have fuckery where HBA's don't work because of limited option rom and other such bullshit.
     
  11. encode

    encode Member

    Joined:
    May 21, 2003
    Messages:
    490
    Location:
    Sydney
    True, but for 24 drives I'd be thinking more than 32GB ECC ram, if not straight away then at least down the road. However the E3 CPU is limited to max 32GB ram...
     
  12. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,643
    Location:
    Canberra
    I said this earlier.

    FreeNAS zealots all claim that 8GB is overkill. :p
     
  13. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    2,930
    Word.

    Give it 8G or 32, 96GB and you'll see no material difference in a home environment - except to your wallet!

    Commercial server (pending usage pattern) would be a different kettle of fish.
     
    Last edited: Jun 5, 2015
  14. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,643
    Location:
    Canberra
    fixed that for you.
     
  15. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    2,930
    yep. the mob rules over there....and the mob has drunk all the coolaid.
     
  16. fad

    fad Member

    Joined:
    Jun 26, 2001
    Messages:
    2,271
    Location:
    City, Canberra, Australia
    I would suggest LSI 9211 8i cards with a Intel expander.

    On the tpoic of memory sizing, I had 24gb allocated to the vm hosting ominos and nappit. I didnt however enable dedupe. Performance was quite good.
     
    Last edited: Jun 5, 2015

Share This Page

Advertisement: