Rack it, rack it good

Discussion in 'PC Build Logs' started by Aussiejuggalo, Dec 18, 2017.

  1. Aussiejuggalo

    Aussiejuggalo Member

    Joined:
    Sep 14, 2010
    Messages:
    1,552
    Location:
    Sunshine Coast Queensland
    Build log of my rack mount journey

    Decided a few months ago that I wanted to stick my NAS, Game "Server" & networking gear into a server rack instead of having it all in standard cases on garage shelves from Bunnings. My reasoning behind this is simple, it'll all be in a single location that can be moved easy (hopefully) while also being easier to clean and maintain.

    This will be a long term build log, probably over the course of a year if not longer just because of the expense of brand new server rack gear, I try not to buy used or refurbished unless I have to, to many bad experiences :(.

    Current set up

    As I said my current set up is a bit crap, old pic but you get the idea.


    Click to view full size!


    NAS

    Here is my current NAS, specs below.


    Click to view full size!


    OS: FreeNAS-9.10.2-U6.
    CPU / Mobo: SUPERMICRO MBD-A1SRi-2758F-O.
    RAM: Kingston 8GB KVR16LSE11/8KF x2.
    Case: Caselabs NOVA X2M.
    Fans: Noctua 120mm NF-F12 2000RPM.
    Drive Cages: ICY BOX 4bay hot swaps x2.
    OS Drive: Samsung 850 EVO 120GB.
    Storage HDD's: Western Digital Red 4TB WD40EFRX x8.
    PSU: Corsair CS550M.
    RAID Card: LSI 9240-8i (flashed to IT Mode).

    Upgrades

    1. The PSU has to be upgraded to handle the full 24 drives, I'm thinking probably an EVGA Gold or Platinum rated one.

    2. The motherboard needs to be replaced to meet my network requirements.

    3. FreeNAS will be getting reinstalled to 11.

    Game Server

    This one runs my private game servers, folds constantly and does anything else I need it to.


    Click to view full size!


    OS: Windows 10 Enterprise.
    CPU: Intel Xeon E3 1245-V5.
    Mobo: MSI C236M Workstation.
    RAM: Crucial CT8G4WFD8213 8GB DDR4 x2.
    Case: Fractal Design Core 1000.
    OS Drive: Samsung Evo 750 120GB.
    Games Drive: Samsung Evo 750 120GB.
    Storage Drive: Western Digital Red 2TB WD20EFRX.
    PSU: Silverstone SFX 600w.

    Upgrades

    Maybe I'll go for 2011-3, 3647 or AM4, not 100% sure but I do want to keep the ECC RAM and keep TPD around 80w so we'll see.

    I'll also be sticking a DVD drive in it so I can rip CD's to the NAS for playback through Plex.

    The Rack

    Really I don't have heaps to stuff in a rack which is good, it means I have room to expand should I need to, this is my very rough idea of how it'll go.


    Click to view full size!


    As for the rack it's self it'll be a Serveredge CBN-37RU-68FS 37U 800mm deep, yes it is completely over sized for what I need but I don't plan on ever buying another rack again, I may as well buy big now so later on down the line it can't bite me in the ass if I want to expand. I also want a 37U instead of a 42U because a 42U will be to tight a squeeze (height wise) through a standard doorway.

    NAS Chassis

    I actually already have this, it's a Supermicro CSE-846 24 bay, had to get it forwarded from the USA to here which was rather expensive but oh well, also had to buy the SAS 2 backplane for it.

    This will be undergoing some mods as well, the fan wall and PSU shroud will be getting removed so I can fit a standard ATX PSU and the Norco 120mm fan wall so that I can use my current build in it and I'll also be changing the rear fans to Noctua ones.


    Click to view full size!



    Click to view full size!


    And hey it came with rails (those things are damn heavy).


    Click to view full size!


    And finally here's the backplane.


    Click to view full size!



    Click to view full size!


    Back up's

    Was stuck on this for a while as well but I think what I'll be doing is basically a DAS / JBOD or whatever you want to call it by using a 12 bay chassis and the SuperMicro JBPWR2 JBOD motherboard & Supermicro CBL-0167L Backplane Cable with a HBA into the Game Server and I'll run the back up's from the NAS through the Game Server to the JBOD rigs, I may end up building 2 of them for double back up's while still having external drives just as an extra layer of redundancy.

    I know it's not the most elegant solution but it means when I'm not backing up I can turn them off which will save power. For those wondering I'll probably be running 6TB WD Blues as back ups.

    Game Server Chassis

    This one took me a while to figure out but I've finally settled on the TGC Rack Mountable Server Chassis Case 2U with the rails which I have ordered today.

    I was originally going to go for a far smaller chassis but after thinking about it for a while the bigger one will give me more room in the future if I decide I want a dual CPU set up. The only thing I'll be doing to this one is changing out the fans to Notcua again.

    LCD KVM

    I know I don't fully need one but it'd be handy to have, I was looking for one that does 1080p via remote access at least but damn these things are expensive no matter what. Going to keep hunting around for one but as of right now I have next to no clue what I want.

    UPS

    I currently have the CyberPower PRO 1500 VA / 1350 Watts and it hasn't been to bad but because it's not rack mountable it's got to go.

    I'm considering either the Eaton 5PX 1500 VA / 1350 Watts or the 2000 VA / 1800 Watts but I'm not sure how much power I'll need, I don't want to be running the UPS to hard while everything's on but I also don't want a massive power bill if the batteries run to empty.

    Both can have the EATON 5PX Extended Battery 1500VA - 2200VA so I can expand them if need be.

    Network

    Last but not least is my network. This is a no-brainer, 10GBE because I can, going to stick with RJ45, I know it pulls more power than SFP+ but it's just easier and I already have CAT7 cables anyway.

    Switch: Will be the Netgear XS508M 8-Port 10-Gigabit Unmanaged Switch, it's not in Aus yet and I have no clue when it will be here but it's cheap enough to get from B&H.

    Cards: Will be Intel X540-T1, they are a little power hungry but from eBay there dirt cheap and I only need to a max of 3 computers.

    So that's the plan

    As I said this wont be a quick build it'll take at least a year to get it set up if not longer, the most expensive thing will be the HDD's especially for backing up.

    I'll try to keep this updated with what I'm doing, planning etc and try to take better pics :D.​
     
  2. mad_mic3

    mad_mic3 Member

    Joined:
    Jan 18, 2009
    Messages:
    2,305
    Location:
    Nulkaba
    Nothing like a good rack, not to big, not to small:)
     
    josh1990 likes this.
  3. Zyklone

    Zyklone Member

    Joined:
    May 1, 2005
    Messages:
    504
    Location:
    Teh Internets
    Not how I would have done it, but cool none the less. I'll be watching with interest that's for sure. I'm planning something similar, but also putting my "gaming" rig in a 4U in the rack as well.

    When will you be making the first bulk order of parts etc?
     
  4. OP
    OP
    Aussiejuggalo

    Aussiejuggalo Member

    Joined:
    Sep 14, 2010
    Messages:
    1,552
    Location:
    Sunshine Coast Queensland
    Racks of all sizes are fun :D.

    I'm still thinking how it's all going to be laid out and what else I can stuff in it right now so the plan may change over time, I'll know for sure once I have the rack.

    I've already got the NAS chassis and I ordered the Game Server one yesterday which may be here this week. The rack is the next thing with a small shelf for the modem and whatever other odds and ends I may need, that will hopefully be sometime in January.

    After that it'll have to be the UPS (thinking the 1800w one) then the switch and maybe motherboard upgrades for both computers so I can have 10GBe working between them. That's about as far as I've gotten so far.
     
  5. Pugs

    Pugs Member

    Joined:
    Jan 20, 2008
    Messages:
    9,173
    Location:
    Redwood Park, SA
  6. OP
    OP
    Aussiejuggalo

    Aussiejuggalo Member

    Joined:
    Sep 14, 2010
    Messages:
    1,552
    Location:
    Sunshine Coast Queensland
  7. Copie

    Copie Member

    Joined:
    Jul 24, 2009
    Messages:
    13,479
    Location:
    Newcastle
    I wouldnt bother with that switch when you can get the XS708 for the same money in AU if you wait for an ebay sale, only difference is that its a managed L2 switch.

    But anyways, TBH after using 10g extensively at home for years, i ended up going back to 1gb. Wasnt worth the power consumption/noise. Only time it was really handy was moving 50+ tb blocks of data, but if i need to do that again ill use SAS direct or direct connection 40gb QSFP.

    Edit: If the bulk of the gear is in the rack, you can always look at the unifi XG-16 which gives you 12x SFP+ (which is good for your inrack gear) and 4x RJ45 10g. And use direct attach SFP+ cables. And the fiber addon cards are generally cheaper then RJ45.,
     
    Last edited: Dec 19, 2017
  8. OP
    OP
    Aussiejuggalo

    Aussiejuggalo Member

    Joined:
    Sep 14, 2010
    Messages:
    1,552
    Location:
    Sunshine Coast Queensland
    I'll have a look at that, did want to stay away from managed stuff mostly because I couldn't be bothered with the extra steps to set it up.

    I move a lot of files between the Game Server, NAS & my rig so the faster the better, also doesn't help that I'm extremely impatient.

    The main stuff I care about will be in the rack but I do have 3 computers out of it, 2 can have 1GB but my main rig I wanted 10GBe. Just not sure if the SFP+ would do the tight 90° turns without damaging the cable, got a 30m run up stairs to my room and I'm not aloud to drill a hole through the floor :upset:.
     
  9. Copie

    Copie Member

    Joined:
    Jul 24, 2009
    Messages:
    13,479
    Location:
    Newcastle
    Makes no sense, if your going to be wasting money on 10gb you may as well go managed, it’s all very straight forward and via a GUI anyways.

    I would keep the redundant psus from the super micro case, unless your replacing it with a redundant zippy psu it’s a waste of money. As,e with upgrading the psu on your NAS, the 550w is more then fine, I’ve run 24 drives off a Corsair 650w before with no issues.

    As for backup, super micro/Norco/LSI make dedicated JBoD enclosures complete with psus etc. the LSI units are dirt cheap, 150 bucks each on the second hand market.

    If it was me doing something like this I would be heading over to natex grabbing a dual socket 2011 board, 128gb ram and 2x 2670’s shoving that all into the super micro case then run your game severs via VM.

    Lower power consumption, it’s all one box, better performance and easy to expand if required.
     
    Last edited: Dec 19, 2017
  10. OP
    OP
    Aussiejuggalo

    Aussiejuggalo Member

    Joined:
    Sep 14, 2010
    Messages:
    1,552
    Location:
    Sunshine Coast Queensland
    I'll have a look at them and see.

    The Supermicro chassis didn't come with any PSU, I could have gotten one with dual 900w ones but I wasn't paying an extra $300 AUD for it. I also don't trust the cheaper Corsair PSU's like what I'm currently using, already had a few die on me but I'll upgrade that later on down the line anyway.

    I was looking at a Supermicro JBoD chassis for $200, shipping is kind of a bitch at $300 but total it's still cheaper than buying everything separate and doing it myself.

    I could do something like that I suppose, I still haven't decided what exactly I'm doing with both of them yet. I do like having them split in different systems just because I've found it easier to fix problems.

    Edit, I had a look at reviews of the unifi XG-16 and there not great, seems to have quality control issues and setting it up is meant to be a pain in the ass.
     
    Last edited: Dec 20, 2017
  11. Copie

    Copie Member

    Joined:
    Jul 24, 2009
    Messages:
    13,479
    Location:
    Newcastle
    Yes they had QC issues when first released along with a stack of various bugs, but they have been ironed out now.

    Theoretically having 1 box should be simpler to fix any issues.

    Also means you wont need the KVM either, so dollars saved there. If you didnt have the rack already you wouldnt need to go for a 37RU, a 12-18RU would of been fine.
     
  12. OP
    OP
    Aussiejuggalo

    Aussiejuggalo Member

    Joined:
    Sep 14, 2010
    Messages:
    1,552
    Location:
    Sunshine Coast Queensland
    Update 1

    NAS Chassis Stuff

    Norco 120mm Fan Wall

    Yes it's not exactly made for a Supermicro chassis but I should be able to squeeze it in, and as an added bonus it comes with the hot-swap things for the fans which I didn't expect at all.


    Click to view full size!



    Click to view full size!


    Uses a single Molex to power it and a 4 pin fan header for signal, not sure where I'm gonna plug the fan header to just yet.


    Click to view full size!


    SuperMicro mSAS to mSAS 75cm SFF-8087 to SFF-808

    Didn't need to go a Supermicro cable but it was $50 so whatever. Don't think the blue will clash with the orange to much either :lol:.


    Click to view full size!


    Game Server Chassis

    I wasn't actually expecting this until after Christmas but early Christmas present it is :D.




    Click to view full size!


    The front is pretty basic, 2x 5.25" bays, tiny power and reset switch, bunch of LED's, USB 2.0 port and first surprise a USB 3.0 port.


    Click to view full size!


    (Ignore the finger)


    Click to view full size!


    Second surprise, I didn't expect this to have the hot-swap fans either but it comes from the same generic OEM as the Norco stuff (this is branded TCG). One odd thing about the hot-swap, it doesn't have a 4 pin fan header it seems to have a USB plug or something like it, not sure about that one.


    Click to view full size!


    The rails aren't anywhere near Supermicro quality, no surprise there, they do feel pretty solid even extended and I won't have an insane amount of weight in this chassis as with the NAS one.


    Click to view full size!


    My one complaint, the back end of one of the rails came bent, not to big of a deal a quick go with pliers sorted it. Not sure if it was like that or the way Scorptec packaged it.


    Click to view full size!


    That's about it

    So yeah that's it for now, I might try and swap the computers over sometime early Jan but most likely it'll be when I get the rack but that may not be till early Feb now, things have come up as they always do which is annoying.​
     
  13. mad_mic3

    mad_mic3 Member

    Joined:
    Jan 18, 2009
    Messages:
    2,305
    Location:
    Nulkaba
    any updates:)
     
  14. OP
    OP
    Aussiejuggalo

    Aussiejuggalo Member

    Joined:
    Sep 14, 2010
    Messages:
    1,552
    Location:
    Sunshine Coast Queensland
    Aside from ordering a new HBA for the NAS (an LSI SAS 9207-8i) no I haven't done pretty much anything.

    I had some bullshit to deal with over Christmas / New Years so that put a hold on things but now that's over and done with I'm back to focusing on this. Around the middle of Feb I'll be ordering the Rack, hopefully by then I'll have the computers in the cases, been a bit hot to do much lately :mad:.
     
  15. OP
    OP
    Aussiejuggalo

    Aussiejuggalo Member

    Joined:
    Sep 14, 2010
    Messages:
    1,552
    Location:
    Sunshine Coast Queensland
    Update 2

    NAS Chassis

    Finally got around to ripping apart the NAS Chassis today, I thought it was going to be a lot of rivets but was nice to see it was all screws.


    Click to view full size!


    All gutted and ready for the fan wall.


    Click to view full size!


    Got a slightly big gap at the back where the power supply is meant to be, may try to cover that with some metal and put an IEC connector in it.


    Click to view full size!


    Fan wall in after a bit of stuffing around.


    Click to view full size!



    Click to view full size!


    It's a bit of a tight fit because the bit of metal with the grommets in it is actually part of the motherboard try so I can't remove it easily.


    Click to view full size!



    Click to view full size!


    I've encountered a few other problems, one is the nubby bits for the power distributor board are slightly fouling the fan wall, it's also a damn tight fit for cables and the top corner of the fan wall is hitting the edge of the chassis.


    Click to view full size!



    Click to view full size!


    The final box of bits, dunno what I'm going to do with all of it yet.


    Click to view full size!


    Problem Solving

    As we can see I have some fit problems with this wall, what I'm thinking is taking out the nubby bit on the bottom of the chassis and removing the corner of the fan wall so it'll fit under the lip of the chassis, that way I can drill some holes through the bottom and 1 on each side and screw the fan wall into place.

    The other problem I now have is I need a PSU with 6 Molex connectors (I don't really want to use adapters), I was looking at some of the 500w Sea Sonic 2U ones but I'm not sure if 500w will be enough or if it'll put to much stress on the power supply.

    Hmm decisions decisions.

    Game Server

    I'll be swapping the game server over in maybe a few days as well, got to have another look at it's chassis though because I'm having trouble seeing how in the hell to access the 5.25" bays.

    Edit, also here's the weird fan connector in the Game Server Chassis, I've never seen it before...


    Click to view full size!

     
    Last edited: Feb 10, 2018
  16. OP
    OP
    Aussiejuggalo

    Aussiejuggalo Member

    Joined:
    Sep 14, 2010
    Messages:
    1,552
    Location:
    Sunshine Coast Queensland
    Update 3

    NAS Chassis

    Got around to doing work on this thing over the couple weeks, it's been a bit of a pain with all the rain but the NAS Chassis is done. The cables look like utter crap though :upset:.


    Click to view full size!


    What I ended up doing was taking 3 chunks out of the fan wall, 2 on the right and 1 on the left, the right was because it was hitting the top rail and a support peg for the PSU cover, the left was because it was hitting the bent bit of metal with the rubber grommets in it. It's fixed with 2 screws on either side although the right side I've had to use a couple of motherboard stand off's then screws because the gap was to big. The screws do rub against the rails but it's nothing to major, if it turns out to be a problem I'll take a file to them.

    I also swapped out the 2 rear 80mm fans for Noctua NF-A8 ones (the other 4 are for the games server), it works out good because the fit in the hotswap bays without the need for screws thanks to the rubber corners, only annoying thing was I couldn't use the hotswap connectors, standard 4 pin connector's wont lock in place.


    Click to view full size!



    Click to view full size!


    I reused the 120mm Industiral fans I've got as well, they needed a bit of cleaning to say the least.

    These also went into the fan wall hotswaps without the need for screws but I think some of the rubber got torn up, not to fussed about it though.


    Click to view full size!



    Other Things

    I've got to do some tests on the HDD's over the weekend because da3 & da5 were throwing up S.M.A.R.T. errors, not sure if it was because I was using the LSI 9240-8i flashed to IT mode, dust was screwing with the connectors or what.


    Click to view full size!


    I've also changed to an LSI SAS 9207-8i which is a straight HBA instead of the flashed LSI 9240-8i, forgot to take a pic of that before chucking it into the case.


    Old NAS Case

    The old case has seen better days... perks of living with a smoker, tar filled sticky dust, always fun :mad:.


    Click to view full size!



    Click to view full size!


    I'll be cleaning this thing out then selling it I think, it's a great little case but I have no need for such a big ITX case any more.


    Game Server

    Now I've gotten the NAS done I can get to work on this thing, I shouldn't have to do any mods to it but I've got to have a good look and see if I can figure out how the hotswap connector works, I've also got to measure sizing for a 2U PSU and see how in the hell to stick a DVD drive into the thing.


    The Rack

    Last but not least the rack, I'll be buying it some time in the next month, thought maybe this week but not 100% sure I'll be ready for it.
     
    Last edited: Mar 10, 2018
  17. OP
    OP
    Aussiejuggalo

    Aussiejuggalo Member

    Joined:
    Sep 14, 2010
    Messages:
    1,552
    Location:
    Sunshine Coast Queensland
    Quick update, ended up working out everything with the Game Server chassis, most of it was pretty simple but after working out the 8 pin fan connector thing for the hotswaps I'm even more confused as to why they did it this way.

    Here's the power and sense connector.


    Click to view full size!


    Here's the pinout, fan 1 is the closest with fan 4 being the end of the run. Control is pin 4 (furthest on the left, from the pic above), Sense is pin 3 (to the right of control).

    I don't understand why it was done this way and not just thick tracers like the ground and 12v and just to a standard 4 pin connector like the Norco 120mm fan wall (this chassis and the fan wall come from the same OEM)

    I also don't understand why the Sense and Control aren't all on the same side...


    Click to view full size!


    To get this working right I'm going to grab a spare internal USB header, solder the Sense / Control together then run them out to a single 4 pin header and the respective pins like it should've been done in the first place.
     
  18. /invariance\

    /invariance\ Member

    Joined:
    May 23, 2012
    Messages:
    827
    Location:
    Brisbane
    I would guess the reason the sense and control connector aren't on respective sides would be due to the autorouting of the pcb package. Could also be due to how they are arranged at the other end of this fan connector. Via's and extra copper to get it into a logical order add dollars to the production run.
    I would think that the power and control lines are separate to keep the current off the motherboard and to individually control and monitor each fan. EG hdd's on the left are getting a hammering and heating up, speed up the fans behind them; one of the fans has failed, set off an alarm.
    If this was designed to be run off internal USB, it would have extra circuitry to interface the control and sense signals to USB.
    Could it be for a dedicated header on the OEM's motherboard?
    If there is no other ic's, then consider connecting the pwr/gnd to the psu, one sense wire to a 4 pin pcb header, and the controls joined together and also going to the 4 pin header. Not pretty but it will keep the current load of the motherboard's fan ic whilst still controlling the speed. Downside is you don't get to see if a fan is failing, but how often does that happen.
     
  19. OP
    OP
    Aussiejuggalo

    Aussiejuggalo Member

    Joined:
    Sep 14, 2010
    Messages:
    1,552
    Location:
    Sunshine Coast Queensland
    See I can't understand why they did it with such small tracers going to an 8 pin header when it would've been quicker and easier doing 4 thick ones, 2 to the Molex for 12v & ground like they have done and 2 to a fan header for control & sense.

    As far as I'm aware with server chassis it's basically an all or nothing kinda deal, so all the fans run at the same speed, if one side of HDD's is hotter all the fans get sped up.

    Nope just what you see in the pics, it's a single sided PCB flat to the chassis.

    It's just a generic OEM that makes all the Norco / TGC stuff, no motherboards.

    I ended up doing this, nicked an old USB header and a fan header, soldered all the wired together, sleeved it and that was that.


    Click to view full size!



    Click to view full size!



    Click to view full size!


    Far from the best work I've ever done but not bad for 20 mins and a USB soldering iron :leet:. I am a bit worried about the fan header side, it feels pretty weak but it should be ok... I hope.
     
  20. OP
    OP
    Aussiejuggalo

    Aussiejuggalo Member

    Joined:
    Sep 14, 2010
    Messages:
    1,552
    Location:
    Sunshine Coast Queensland
    Update 4

    Game Server Chassis

    Got around to moving everything over today, ended up waiting and ordered a Seasonic 2U 500w PSU. It got sent and delivered pretty quick, ordered yesterday from Scorptec at 2.20pm, Star Track delivered it by 10am this morning.

    500w isn't much but it's gold rated and I'm not going to be running heaps with it.


    Click to view full size!


    The only gripe I have with the cables is the PCI-e cable is connected to the CPU one and I'm never going to use the PCI-e so it's taking up space. Think I may de-pin it sometime.


    Click to view full size!


    I like this kind of connector set up, can't really screw up and plug in the wrong thing.


    Click to view full size!


    The only thing I couldn't attach was this, it's a mounting plate for the cable side so the PSU doesn't get pulled up while you plug and unplug / pull on the cables.


    Click to view full size!


    Finally here's the computer, a lot of cables which I'm not totally happy about and my SSD's are currently not mounted to anything but it works


    Click to view full size!


    Problems

    While this is working I do have a problem, the fan cable I made isn't working right, it's only connecting intermittently so I can't control the speed of the fans at all.

    I don't know why it's doing it because I triple checked the pin out before I did anything, checked continuity to each pin before making the cable and checked again after making the cable.

    I suppose it's not that big of a deal, I did buy Noctua fans knowing full well I may need to run them at 100% but I did want to have the option of controlling them.


    The Rack

    I'll be getting this next month, had bills and shit come up on and off over the last few months which has stopped me from getting it but now I have everything swapped over I'll be getting it ASAP.

    UPS

    This will most likely be the next thing, I've decided on the Eaton 5PX 2000 VA / 1800 Watts and if I need EATON 5PX Extended Battery 1500VA - 2200VA. Not sure if it'll be overkill but seeing I'm stuck to running off 10A this is the biggest I can go.

    LCD KVM

    I've changed my mind on a really expensive one (kinda), currently I'm looking at some single port used ones on eBay and just using my current KVM switch (if it'll work). Not sure if the 1280x1024 res of the cheap ones will shit me though so I am keeping the Serveredge SED-KLVGA-0117 in mind.​
     

Share This Page