File Server Build Thread

[Davo1111]

epic, nice post.

[doodz]

AfterBurner1: I think the norco is slightly thinner than the Alienware one. Anyhow PCCG have sent me the new HSF/Cooler and HDD so yea. Should get them tommorow.

I got the HDD's from pcmeal. They have them on special currently $143 each. It came to $144 delivered for me. I highly recommend them. Had at hitatchi HDD arrive then it broken after I sent it to someone and I requested a RMA and got a replacement within 2-3days.

hate-xfiles Yea it's Raidz2 - aka Raid6 with extra 2 parity bits. it's pretty much n-2, and I'll do a pool of 10 drives for the 20. And being n-2, it will be (10-2)+(10-2)=~32TB. But ive figured in real world terms it's only approx 28TB

In terms of price approx $2000 for the hardware + Whatever the drives cost.

The good thing about this is.. If I'm to expand I can just buy another norco case ~$600 + PSU + Expander + Hard drives. Wont need to buy Mobo/RAM/etc etc as the 2nd system will be connected to the 1st system. But that's long way down the track...

[Zenskas]

I don't recommend PC Meal at all, I had a 500GB die on me at the start of the year and it was well over a month and countless ignored emails before I got it back
But nice fileserver man, keep the Q6600 unless you really need something with less power usage, in that case grab an E5200 rather than an older E4xxx CPU.

[Snuffy22]

Originally Posted by Zenskas

I don't recommend PC Meal at all, I had a 500GB die on me at the start of the year and it was well over a month and countless ignored emails before I got it back
But nice fileserver man, keep the Q6600 unless you really need something with less power usage, in that case grab an E5200 rather than an older E4xxx CPU.

HDD warranty's are best dealt straight with the manufacturer, they are all pretty damn good with the RMA's (imo)

[doodz]

After much angusih and downloading the wrong iso! I downloaded some opensolaris iso and it was incomplete and didnt boot properly. I had no idea why, thought it was my hardware etc etc... Finally figured out the one I used was wrong! The MD5 has checks were different!

So tip: be sure to do a MD5 check on your iso before burning and installing, it wont work if you have the wrong one!

Anyways currently installing opensolaris:


Click to view full size!
Yes that's a Samsung 40" LED LCD! I didnt have any other smaller screen. lol

Hardware pictures to come later...

Testing the different slots and what they register as in osol:
For referenece:

Without expander:
So it goes slot 1 = c9d0d0, c9d1d0, c9d2d0, c9d3d0, slot 2 = c9d4d0, c9d5d0, c9d5d0, c9d7d0

With expander and L8i Card:

USAS
Port 1 ----- c9d0d0
----- c9d1d0
----- c9d2d0
----- c9d3d0

Port 2 ----- Expander
Port 1 ----- c9d20d0
----- c9d21d0
----- c9d22d0
----- c9d23d0

Port 2 ----- c9d24d0
----- c9d25d0
----- c9d26d0
----- c9d27d0

Port 3 ----- c9d28d0
----- c9d29d0
----- c9d30d0
----- c9d31d0

Port 4 ----- c9d32d0
----- c9d33d0
----- c9d34d0
----- c9d35d0

[Snuffy22]

Originally Posted by doodz

After much angusih and downloading the wrong iso! I downloaded some opensolaris iso and it was incomplete and didnt boot properly. I had no idea why, thought it was my hardware etc etc... Finally figured out the one I used was wrong! The MD5 has checks were different!

So tip: be sure to do a MD5 check on your iso before burning and installing, it wont work if you have the wrong one!

Anyways currently installing opensolaris:

At least you managed to get opensolaris installed, my install last night decided to freeze at about 60%............. sigh
on a brighter note, Norco has fixed me up with 5 mini sas to mini sas cables for $77US delivered thanks for the hookup doodz!

[electronicaz]

why dont you just use ubuntu 10.04 server?

[Soarer GT]

Originally Posted by electronicaz

why dont you just use ubuntu 10.04 server?

I've had major issues with 10.04 server regarding startup services. Stick with 9.10 if you're going to go down that path.

[davros123]

Originally Posted by electronicaz

why dont you just use ubuntu 10.04 server?

With that much data you want zfs, and that's not available in ubuntu ( other than via fuse...and that's not stable).

[Snuffy22]

where is the photo of your modded fan firewall doodz

i just finished setting up SMB on my opensolaris server, sharing to windows machines is a go

[doodz]

Pics pending... too fken sleepy for pics etc...

But I do have:

d00dz@TurdBox:~# zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
Buffy  36.2T   206K  36.2T     0%  ONLINE  -

d00dz@TurdBox:~# zpool status
  pool: Buffy
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        Buffy         ONLINE       0     0     0
          raidz2      ONLINE       0     0     0
            c10t4d0   ONLINE       0     0     0
            c10t5d0   ONLINE       0     0     0
            c10t6d0   ONLINE       0     0     0
            c10t7d0   ONLINE       0     0     0
            c10t20d0  ONLINE       0     0     0
            c10t21d0  ONLINE       0     0     0
            c10t22d0  ONLINE       0     0     0
            c10t23d0  ONLINE       0     0     0
            c10t24d0  ONLINE       0     0     0
            c10t25d0  ONLINE       0     0     0
          raidz2      ONLINE       0     0     0
            c10t26d0  ONLINE       0     0     0
            c10t27d0  ONLINE       0     0     0
            c10t28d0  ONLINE       0     0     0
            c10t29d0  ONLINE       0     0     0
            c10t30d0  ONLINE       0     0     0
            c10t31d0  ONLINE       0     0     0
            c10t32d0  ONLINE       0     0     0
            c10t33d0  ONLINE       0     0     0
            c10t34d0  ONLINE       0     0     0
            c10t35d0  ONLINE       0     0     0

errors: No known data errors

d00dz@TurdBox:~# zfs list
NAME                      USED  AVAIL  REFER  MOUNTPOINT
Buffy                     153K  28.4T  45.4K  /Buffy

After much headache and finally rebooting system I got the system to work with cifs in Windows7

GO Buffy!


Click to view full size!

Here are initial pictures with the old larger HSF:


Click to view full size!


Click to view full size!


Click to view full size!


Click to view full size!

And what I used for the UIO card:


Click to view full size!


Click to view full size!


Here are the actual pictures of the build:

Due to the norco case's miniSAS plugs for the backpane being in the middle I had to modify it as the cables I had werent long enough.
Here is my mod:


Click to view full size!

This is pictures of the actual build: Yes messy cables but I was in a rush and need some other parts to make it better:


Click to view full size!


Click to view full size!


Click to view full size!

So far so good everything is as I thought it would be. Transfering data from main pc to the fs is pretty quick. ~80-100MB/s depending on file.
Doesnt seem to be any slowdown.

I do notice that zfs file system uses a shit load of ram.

[Snuffy22]

lol, that HSF barely fits, i like it
the mod of the firewall isnt to bad, alot smaller then i expected, thought you might have taken all the cables through the middle at the top not the bottom

[doodz]

After much figuring out I got the temps working too!

Download the source tarball from here: (at time of writing smartmontools-5.39.1) - http://sourceforge.net/project/showf...group_id=64297

# gunzip smartmontools-5.39.1.tar.gz
# tar xvf smartmontools-5.39.1.tar
# cd smartmontools-5.39.1

Then you have to setup the complier on your server

# pkg install gcc-dev

Then in the same dir as the smartmontools dir

# ./configure
# make
# make install

Then deleted (commented out, put a #) for the "DEVICESCAN" line in /usr/local/etc/smartd.conf

Then created Temps.sh and added the script from: http://breden.org.uk/2008/05/16/home...r-drive-temps/

Where it says "(Name, OF, YOUR, Drives)" replace each one with the name of your drives. ie. c10t6d0 etc...

#!/usr/bin/bash

# is there a way to populate the array by querying the zfs pool directly?
drives=(Name, OF, YOUR, Drives)

temp=""

drivetemp () {
drive=$1
# printf "drive passed in = %s\n\n" $drive
temp=`/usr/local/sbin/smartctl -a -d scsi /dev/rdsk/$drive | \
grep Current | awk '{print $4 $5}'`
return
}

drives_count=${#drives[@]}
index=0

while [ "$index" -lt "$drives_count" ]
do
drivetemp ${drives[$index]}
printf "Drive %s temp is %s\n" ${drives[$index]} $temp
let "index = $index + 1"
done

Then a quick chomod +x Temps.sh

Then I ran the file ./Temps.sh

And got:

d00dz@TurdBox:/Buffy/Other# ./Temps.sh
Drive c8d0 temp is
Drive c10t4d0 temp is 32C
Drive c10t5d0 temp is 33C
Drive c10t6d0 temp is 32C
Drive c10t7d0 temp is 30C
Drive c10t20d0 temp is 33C
Drive c10t21d0 temp is 34C
Drive c10t22d0 temp is 33C
Drive c10t23d0 temp is 33C
Drive c10t24d0 temp is 33C
Drive c10t25d0 temp is 32C
Drive c10t26d0 temp is 34C
Drive c10t27d0 temp is 33C
Drive c10t28d0 temp is 32C
Drive c10t29d0 temp is 34C
Drive c10t30d0 temp is 33C
Drive c10t31d0 temp is 32C
Drive c10t32d0 temp is 29C
Drive c10t33d0 temp is 31C

Not sure why it doesnt work for the SATA drive connected to the mobo's Sata port. I read something about the smartd not liking Intel ICHR chip or something.

Help taken from:
http://cafenate.wordpress.com/2009/0...n-opensolaris/
and
http://forums.overclockers.com.au/sh...&postcount=134


Also some reference for the intel board:

Front panel pinout is documented in the server board TPS, page 95. Normally you'll need to connect the following:

Power switch: 11-13

Reset button: 15-17

Power LED: 1-5

HDD LED: 7-9

http://communities.intel.com/thread/...e6COM?tstart=7

[Snuffy22]

cheers for the rundown on the smart monitoring, Ive got it working on my box as well now , very useful for when summer comes around

[Simboo]

Originally Posted by doodz

After much figuring out I got the temps working too!




Not sure why it doesnt work for the SATA drive connected to the mobo's Sata port. I read something about the smartd not liking Intel ICHR chip or something.

It depends on wether the drive supports the temperature reading, and if you are grepping for the correct string. So simply adding your intel onboard drive device into that script may not specifically work.

Did you try any alternative methods to get the temperature manually?