File Server Build Thread

Discussion in 'Storage & Backup' started by doodz, Jun 21, 2010.

  1. doodz

    doodz Member

    Joined:
    Nov 6, 2002
    Messages:
    5,025
    Location:
    Sydney
    So this is my thread where I'll be making a 32TB file server with scablity.

    Parts will be used are:

    Norco 4220 w/ 120mm fan board.
    Supermicro AOC-USAS-L8i
    Chenbro CK12803
    Intel Q6600
    4GB RAM
    Intel S3210SHLC
    Corsair HX850

    Solaris ZFS.
    Will be running two pools of 2TBx10. So 16TB useable raidz2, then later expand that to another 2TBx10, with total of 32TB useable.

    Samsung 2TB Hard Drives.

    Pictures: (These are the parts that have arrived already)

    Norco 4220: (1 week shipping direct from Norco)


    Click to view full size!



    Click to view full size!



    Click to view full size!


    20x Samsung 2TB HDDs:

    Click to view full size!


    Custom made Norco 120mm fan board:

    Click to view full size!



    Click to view full size!



    Click to view full size!


    2xNoctua NF-R8 80mm Fans
    3xNoctua NF-S12B 120mm Fans

    Click to view full size!


    Mini SAS -> Mini SAS Cables + Screws and molexes from Norco shop:

    Click to view full size!


    Supermicro AOC-USAS-L8I Card:

    Click to view full size!


    Chenbro CK12803 card:

    Click to view full size!



    Click to view full size!


    Intel Server Board + Q6600 +4GB RAM:

    Click to view full size!


    Click to view full size!


    Click to view full size!
     
    Last edited: Jul 22, 2010
  2. Stanza

    Stanza Member

    Joined:
    Jun 27, 2001
    Messages:
    2,878
    Location:
    Adelaide
    Subscribed ;)
     
  3. GoofyHSK

    GoofyHSK Bracket Mastah

    Joined:
    Mar 3, 2002
    Messages:
    1,566
    Location:
    Adelaide Hills
    giggity

    you gonna test bench the mobo>hba>expander before norco?
     
  4. OP
    OP
    doodz

    doodz Member

    Joined:
    Nov 6, 2002
    Messages:
    5,025
    Location:
    Sydney
    Probably not. Dont have time, but might be able to test stuff once I have everything. Drives will be the last thing I buy. So somewhere between getting the drives and putting stuff in I might.

    I need to read up and see how ZFS Solaris works too so yea..
     
  5. Statts

    Statts RIP

    Joined:
    Aug 29, 2004
    Messages:
    2,504
    Location:
    Radelaide, SA
    THIS.

    Do it.
     
  6. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    2,912
    Nice.

    Recommend you download the svn 134 release of opensolaris. You'll find plenty of info on setup stuff in my thread too...see sig. Happy to help out where I can so post often if you have any questions.

    You could just run 3 of the super micro cards if the expanded turns out to be an issue.

    Oh, and you may need to update the bios...if it just sits there and "clicks" when you boot it...it needs to be updated. See here for more info. http://communities.intel.com/message/83643;jsessionid=FADE4ED01F8EA6B6274B33FB22D7FC71.node5COMS

    Also, be patient, as these are not the easiest boards to setup...ie. Pinout for case connectors are a bugger to work out what goes where..worth it though.
     
    Last edited: Jun 21, 2010
  7. Flux

    Flux Member

    Joined:
    Jul 4, 2007
    Messages:
    1,511
    Location:
    Melbourne
    Storing those *nix iso's for Armageddon are we? :Pirate:
     
  8. terrastrife

    terrastrife Member

    Joined:
    Jun 2, 2006
    Messages:
    18,414
    Location:
    ADL/SA The Monopoly State
    dont see why this warrants a new thread, its pretty much on par with teh rest in the 2 massive 'file server' threads we got already :)

    this wont be interesting to me unless it has at least 8000Mbit of full duplex networking bandwidth.
     
  9. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    2,912
    Care to elaborate on the "support Issues"?
     
  10. flain

    flain Member

    Joined:
    Oct 5, 2005
    Messages:
    2,162
    Location:
    Sydney
    Bit of a warning - the mini sas cable that came with my Chenbro card is dodgy, doesn't "click" into the minisas connectors properly and has a tendancy to wiggle. I had more errors on my drives running from that cable than others so i ordered another norco minisas cable. I only mention this because your cable looks the same make.

    If you notice errors on drives running that cable - id change that first :p You will see what i mean about the "click" when you plug it into your SAS card and norco backplane.
     
    Last edited: Jun 23, 2010
  11. flain

    flain Member

    Joined:
    Oct 5, 2005
    Messages:
    2,162
    Location:
    Sydney
    Maybe its time we get our own opensolaris/ZFS subforum? :p
     
  12. passivekid

    passivekid Member

    Joined:
    Sep 10, 2003
    Messages:
    332
    Location:
    Perth, WA
    I reversed the bracket on my Supermicro cards by using nylon spacers and long screws so I could mount them in a PCI-e slots easily, works a dream.
     
  13. EvilGenius

    EvilGenius Member

    Joined:
    Apr 26, 2005
    Messages:
    10,121
    Location:
    Rocky
    I found the silver coolermaster mobo spacers (the slightly longer ones) worked a treat for this. Screw the spacer into the bracket and then the card onto the spacer. Perfect :)
     
  14. kbekus

    kbekus Member

    Joined:
    Aug 28, 2001
    Messages:
    1,069
    Location:
    Pascoe Vale, Melbourne
    a Q6600 is a hefty CPU with a fair power draw.... is this just for file serving? If so, perhaps a way less powerful CPU, you could save 20W or more just there.
     
  15. EvilGenius

    EvilGenius Member

    Joined:
    Apr 26, 2005
    Messages:
    10,121
    Location:
    Rocky
    It came as a bundle off ebay... Should be fine with eist settings left on?
     
  16. flain

    flain Member

    Joined:
    Oct 5, 2005
    Messages:
    2,162
    Location:
    Sydney
    Actually a powerful CPU is pretty beneficial to a ZFS file server. The "smartness" of ZFS is pretty CPU intensive. The main benefit being that doing a weekly zpool scrub is usually CPU limited.
     
  17. sub.mesa

    sub.mesa Member

    Joined:
    Jun 23, 2010
    Messages:
    271
    Location:
    Europe
    There is nothing smart about using 20-25W when doing absolutely nothing. Imagine your car drawing one litre of gasoline per hour when being parked.

    The 65nm CPUs from Intel, and the Q6600 in special, have quite high idle power consumption. 65nm AMD cpus do a lot better than Intel's; but both achieve sub-5W idle power consumption on 45nm/32nm CPUs.

    So don't use an older chip which wastes a lot of power when idling. As NAS systems are idling > 99%, the idle power consumption is the only thing that counts. So you better have a 125W TDP chip that idles at 3W than a 45W TDP chip that idles at 20W.

    Both AMD and Intel have low-power options, and you do not need to go the Atom-route to get something really low-power. But the main constraint will be the power supply, and using an efficient power supply like PicoPSU wil mean you are limited to using 2,5" notebook HDDs instead of 3,5" desktop-class HDDs which use ~33W when spinning up.

    And i would say ZFS is more memory hungry than CPU hungry, especially when not using compression, encryption or de-duplication.
     
  18. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    2,912
    Agree a q660 is overkill for a "pure" nas, but perhaps it'll also do server duties via my q6600 rarely goes over 60 percent on all cores even when runing scrubs and 3 vm's.

    I'd be surprised if Someone who spends this much on a server is not going to be fussed about saving 20 or 30 bucks a year by swapping cpu's...not to mention he already has this one and would prob. lose that much in the swap over.

    I am constantly surprised by people's desire to save power with their file servers...I wonder if they are also the type to wear two jumpers rather than put on the heater or unplug all the appliances, tv DVD etc to avoid standby usage, turn down the fridge and pay twice the cost for led tv's...
     
    Last edited: Jun 23, 2010
  19. silentlightning

    silentlightning Member

    Joined:
    Aug 26, 2009
    Messages:
    765
    Location:
    Melbourne
    if you had the option to do that why wouldn't you??? LED tv's are pretty sweet compared to normal lcd ones.

    Also,some people are more aware of the enviromental costs associated with higher power consumption devices rather than the direct $$ value, oh and if search out more power efficient devices in general and do that for 4+ devices @ $15-$20 each your still saving $60 a year minimum every year. and say you buy 2-3 devices a year every year it slowly builds up in savings.
     
    Last edited: Jul 22, 2010
  20. flain

    flain Member

    Joined:
    Oct 5, 2005
    Messages:
    2,162
    Location:
    Sydney
    doodz, just some tips for when you do the build

    - You will probably need to change a jumper on the chenbro expander - my SAS card (same as yours) wouldn't detect it without me doing that

    - once you have drives connected to the expander, the system might not boot past detecting the drives connected to the expander. Go into the SAS card setup and just disable boot support and problem should be solved.

    - i upgraded the firmware on my SAS card, but not sure if it actually did anything :p
     

Share This Page