Overclockers Australia Forums
OCAU News - Wiki - QuickLinks - Pix - Sponsors  

Go Back   Overclockers Australia Forums > Specific Hardware Topics > Storage & Backup

Notices


Sign up for a free OCAU account and this ad will go away!
Search our forums with Google:
Reply
 
Thread Tools
Old 18th February 2012, 9:33 AM   #676
davros123 Thread Starter
Member
 
Join Date: Jun 2008
Posts: 2,214
Default

mike, great to see a clean pool...albeit with no data

I really do encourage everyone to use at lease a raidz2 array and keep backups on a sep. server. it might seem expensive or overkill, but when things like this happen, it's a few minutes inconvenience as opposed to days or weeks.

You mention lessons learned, would you be able to summarise what you learned from this so I and other can benefit from you experience?

ie tool used, what you tested etc

cheers, dav
__________________
Want a nas, you may find my Esxi/Solaris ZFS NAS build thread of interest.
Quote:
Originally Posted by Stanza View Post
yeah well I just reported my own post...ferk....
Quote:
Originally Posted by Blinky View Post
If you have become content with the size of your e-penis, sticking clear of rack mounted stuff will save you heaps of $$$.

Last edited by davros123; 18th February 2012 at 9:41 AM.
davros123 is online now   Reply With Quote

Join OCAU to remove this ad!
Old 18th February 2012, 1:05 PM   #677
mike-s
Member
 
mike-s's Avatar
 
Join Date: Dec 2003
Location: Sydney, Australia
Posts: 1,845
Default

It's not so much tools used, as it was only cfgadm, zpool oh and zdb that i was mucking with. There are some good tutorials out there though.

The biggest lesson was seriously one about complacency and caution. Had i not made a mistake when pulling a drive and reinserting it, this would have only ever been a whinge about replacing a drive and how long resilvering is taking.
__________________
Successfull trades: RakOon, drfbro, mR_CaESaR, Spyfox
If I've got crap for sale on ebay you like, click here.
Old hardware giveaway thread here.
Sony Vaio Club Member #21
mike-s is offline   Reply With Quote
Old 18th February 2012, 7:03 PM   #678
rubix
Member
 
Join Date: May 2007
Location: Melbourne
Posts: 626
Default

Not sure if this is the right place to ask but was just wondering if someone could clarify something for me please. Been reading through this thread and some of the still active links in the OP about zfs and was just wanting to know more about expanding a an array under zfs.

If I was wanting to expand something like a 4 disk raidz2 array, I wouldn't be able to just add one disk at a time to grow it would I? I'd have to either remove a disk and add a larger one and let it resilver the array then repeat the process for the other disks, or I would have to add another 4 disk raidz2 array and merge them?

sorry again if this wasn't the right place to ask about it.
rubix is offline   Reply With Quote
Old 18th February 2012, 7:36 PM   #679
BAK
Member
 
BAK's Avatar
 
Join Date: Jan 2005
Location: Melbourne
Posts: 397
Default

Quote:
Originally Posted by rubix View Post
If I was wanting to expand something like a 4 disk raidz2 array, I wouldn't be able to just add one disk at a time to grow it would I? I'd have to either remove a disk and add a larger one and let it resilver the array then repeat the process for the other disks, or I would have to add another 4 disk raidz2 array and merge them?
Correct on all counts. To expand a 4 disk RaidZ2 vdev (although I'm not actually sure you can have only 4 disks in RaidZ2 configuration, you may need 5) you'd need to either:

a: Add larger disks one by one, allowing it to resilver each time, the array should expand to fit once the last disk finishes
b: Add another vdev of 4(5, if the above is correct) disks in Raidz2* to the pool

*I'm not 100% sure that the "expansion" vdev needs to be in the same Raid configuration, somebody could confirm this.
__________________
NeOnServ 2.0: Ubuntu 12.04LTS | Q6600 | 8gb | G/A 8EP35-DS3P | AOC-USAS-L8i | 15x 2tb Samsung HD204UI (ZFS Raid-Z2) | CMStacker 810
BAK is offline   Reply With Quote
Old 22nd February 2012, 8:50 PM   #680
judge25
Member
 
Join Date: Dec 2007
Posts: 26
Default

Well my build is up and running, went 10 disk Raidz2 with hot spare, 4k aligned. Not all that bad really all this 4k stuff. It benches at about 900MB/s. Pretty happy with that.

As I already use Esx at work, Esxi part was a breeze. I am new to Solaris 11.11 and it's fun but had to do a lot of knowledge refreshing as haven't used unix in many many years...it's all coming back though!

Anyone using Solaris 11.11 and Esxi?

It appears Vmware tools does not currently work with it and I want to have a UPS do graceful shutdowns of the system if needed. I can set it up ok, and will works on the windows VM's but with out Vmware tools not sure if anyone has found a way to do it with Solaris 11.11?
judge25 is offline   Reply With Quote
Old 22nd February 2012, 9:09 PM   #681
Hive
Member
 
Hive's Avatar
 
Join Date: Jul 2010
Location: Brisbane
Posts: 3,822
Default

Quote:
Originally Posted by judge25 View Post
Well my build is up and running, went 10 disk Raidz2 with hot spare, 4k aligned. Not all that bad really all this 4k stuff. It benches at about 900MB/s. Pretty happy with that.

As I already use Esx at work, Esxi part was a breeze. I am new to Solaris 11.11 and it's fun but had to do a lot of knowledge refreshing as haven't used unix in many many years...it's all coming back though!

Anyone using Solaris 11.11 and Esxi?

It appears Vmware tools does not currently work with it and I want to have a UPS do graceful shutdowns of the system if needed. I can set it up ok, and will works on the windows VM's but with out Vmware tools not sure if anyone has found a way to do it with Solaris 11.11?
How did you align the partition for 4k?
__________________
Today a young man on acid realized that all matter is merely energy condensed to a slow vibration, that we are all one consciousness experiencing itself subjectively, there is no such thing as death, life is only a dream, and we are the imagination of ourselves. Heres Tom with the Weather
Hive is offline   Reply With Quote
Old 22nd February 2012, 9:55 PM   #682
rubix
Member
 
Join Date: May 2007
Location: Melbourne
Posts: 626
Default

Have had my system running for a few days now and it's been pretty interesting playing around with it and tweaking it. Currently have 1 x 160gb OS drive and a raidz pool consisting of 6 x 500gb assorted drives. Only problem at the moment is getting my Intel CT Desktop NIC to play nice and function at 1gbit instead of 100mbit...

I noticed a few people in the consolidated ZFS thread were using the same NIC and was wondering if there is any special trick to getting it working at 1gbit? It says on the Oracle HCL that it's fully certified - http://www.oracle.com/webfolder/tech...8_11/9404.html

any ideas?
rubix is offline   Reply With Quote
Old 22nd February 2012, 10:43 PM   #683
davros123 Thread Starter
Member
 
Join Date: Jun 2008
Posts: 2,214
Default

Hi rubix, there is no trick to the CT, it's plug and play. What o/s are you running? Are you sure the cable, switch and endpoint card is ok?

ps. suggest you mirror the rpool too

Judge, congrat's on getting the system up and running! And reeely nice speed mate! Can we get a few more details on your setup and some pic's?

re. the UPS, I do something similar..
I have an "all in one" server running of a 2400 APC UPS. The Solaris 11 Express vm is setup under ESXi 4.1 serving a number of vm's. It is the first to boot and last to shutdown. I have installed apcupsd on this S11E machine (vm) and passed through the serial port from the APC UPS...(you could just as easily pass the usb port from the ups if that is what your uses).

apcupsd monitors the ups. If it needs to shutdown due to power loss, it runs a script to ssh into ESXi and send the shutdown command to the other vm's and then a "general" shutdown command to ESXi which also shuts down the S11E machine and the whole server.

You can read about an example of how I setup the ssh in my post here

You can find the esxi command line commands on the web...they give you great control of the vm's...

eg, to shutdown the vm with the machine id of "600", where the ip address of the esxi box is 192.168.0.31, the command on the S11E machine would look like this.
Code:
ssh 192.168.0.31 -l root "vin-cmd vmsvc/power.shutdown 600"
I hope that helps.
__________________
Want a nas, you may find my Esxi/Solaris ZFS NAS build thread of interest.
Quote:
Originally Posted by Stanza View Post
yeah well I just reported my own post...ferk....
Quote:
Originally Posted by Blinky View Post
If you have become content with the size of your e-penis, sticking clear of rack mounted stuff will save you heaps of $$$.

Last edited by davros123; 22nd February 2012 at 10:46 PM.
davros123 is online now   Reply With Quote
Old 22nd February 2012, 11:32 PM   #684
rubix
Member
 
Join Date: May 2007
Location: Melbourne
Posts: 626
Default

Quote:
Originally Posted by davros123 View Post
Hi rubix, there is no trick to the CT, it's plug and play. What o/s are you running? Are you sure the cable, switch and endpoint card is ok?
Thanks for the reply mate. Ended trying another cable and it worked fine, can't believe I forgot to try that ha. I'm running Solaris 11, also just wondering what conf file do I need to edit so that my server automatically joins the workgroup on startup? At the moment I'm just doing it manually using
Code:
smbadm join -w WORKGROUP
rubix is offline   Reply With Quote
Old 22nd February 2012, 11:35 PM   #685
Rezin
Member
 
Rezin's Avatar
 
Join Date: Oct 2002
Posts: 8,628
Default

Quote:
Originally Posted by rubix View Post
I'm running Solaris 11, also just wondering what conf file do I need to edit so that my server automatically joins the workgroup on startup? At the moment I'm just doing it manually using
You should only have to do that once, and not keep repeating it every time the box boots.
Rezin is offline   Reply With Quote
Old 24th February 2012, 2:46 PM   #686
rubix
Member
 
Join Date: May 2007
Location: Melbourne
Posts: 626
Default

Can anyone recommend somewhere to buy LSI cards at a reasonable price? I'm after a LSI SAS 9201-16i and the cheapest I've managed to find them in Aus is around around $630AUD, they're selling for around $400USD through amazon but they're region restricted to where they can ship to...
rubix is offline   Reply With Quote
Old 24th February 2012, 3:09 PM   #687
Rezin
Member
 
Rezin's Avatar
 
Join Date: Oct 2002
Posts: 8,628
Default

I'd check eBay, but if there's no good ones there, consider using one of those US postal services where they forward the package onto you (for a fee, of course).
Rezin is offline   Reply With Quote
Old 26th February 2012, 1:16 PM   #688
judge25
Member
 
Join Date: Dec 2007
Posts: 26
Default

Although not recommended by all, as I was building a new zpool without any data I used a patched binary that auto sets ashift=12 for 4k.

Used the Solaris 11.11 one found at solarismen.de

I did initially try building the zpool using zfsguru live cd and exporting and importing the zpool. It has built in option to set 4k parameters. Although it built and exported fine and looked ok, Solaris did not seem to like it and said it was unreadable or corrupt. Not sure what If anything I did wrong there :-)

Quote:
Originally Posted by Hive View Post
How did you align the partition for 4k?

Last edited by judge25; 27th February 2012 at 1:09 PM.
judge25 is offline   Reply With Quote
Old 26th February 2012, 2:08 PM   #689
judge25
Member
 
Join Date: Dec 2007
Posts: 26
Default

Davros, as always thanks mate! Great find on the apcupsd did not know about it, that will help tie it all together. Now I know it can be done.
Been looking at used UPS's on ebay but not sure which are quiet, or better still silent!?

My builds is a little bit of a cross between your setup and some of the ones in the Norco GB.

My setup:

Norco 4224 server case.
3 x 12cm fan wall/Noctua Fans all round
Tyan S5512GM2NR MB (wanted the 4NR but no stock)
E3-1270 Xeon
Noctua NH-U9B SE2 Cpu Cooler
16 Gb Kingston KVR1333D3E9SK2/8GI Ecc Ram
Corsair x850 psu
3 x M1015 sas cards, flashed to IT mode.
1 x Intel 520 ssd 120gb for VM's
12 x samsung F4's 2tb HDD's. 10 in a raidz2. 1 hot spare. 1 as second store and data drive for other VM's (saves spinning up the whole array all the time).
I wanted 5k Hitachi's but can not get them. So far the samsungs are going well with no problems encountered. They are quiet and power friendly.

Esxi 5, Solaris 11.11 and currently one Windows 2008r2 vm (more to come).
Esxi boots from and internally located 1Gb Flash drive (motherboard has 2 internal headers)

The 3 m1015's and one network port VMdirect pathed through to solaris.

Mounted in a 18ru open rack with Switch and soon Ups!
3 network ports patched in including the motherboard IPMI.

All in all a nice and easily maneuverable quiet setup. Pics and power usage readings will follow if anyone is interested :-)

Quote:
Originally Posted by davros123 View Post
Judge, congrat's on getting the system up and running! And reeely nice speed mate! Can we get a few more details on your setup and some pic's?

re. the UPS, I do something similar..
I have an "all in one" server running of a 2400 APC UPS. The Solaris 11 Express vm is setup under ESXi 4.1 serving a number of vm's. It is the first to boot and last to shutdown. I have installed apcupsd on this S11E machine (vm) and passed through the serial port from the APC UPS...(you could just as easily pass the usb port from the ups if that is what your uses).

apcupsd monitors the ups. If it needs to shutdown due to power loss, it runs a script to ssh into ESXi and send the shutdown command to the other vm's and then a "general" shutdown command to ESXi which also shuts down the S11E machine and the whole server.

You can read about an example of how I setup the ssh in my post here

You can find the esxi command line commands on the web...they give you great control of the vm's...

eg, to shutdown the vm with the machine id of "600", where the ip address of the esxi box is 192.168.0.31, the command on the S11E machine would look like this.
Code:
ssh 192.168.0.31 -l root "vin-cmd vmsvc/power.shutdown 600"
I hope that helps.
judge25 is offline   Reply With Quote
Old 26th February 2012, 4:10 PM   #690
davros123 Thread Starter
Member
 
Join Date: Jun 2008
Posts: 2,214
Default

Nice setup there mate. Similar elements to my kit...same PSU, memory, cooler, I run a slower xeon a 1230, M1015's, an m4 ssd for the vm's and noctura's to keep her cool I would lurve the norco, but was too short sighted to buy it back when the group buy happened...sigh

Here's a pic of mine while i was adding it into the new rack (note no patch cables in yet).

Click to view full size!


I do like the idea of separate data drives so you can spin down the main array...I think I might go that way when the 3TB 5k3000's come back down.

One question, why the hot spare? Why not a raidz3? or are you planning to share this across vdev's?

wrt. the UPS, most (all?) of the line-interactive (not the XL's) smartup's are fanless until they kick in, then the fans start...but that's ok.
I run a 2200 like this one.
http://www.ebay.com.au/itm/APC-Smart...item3f12240a2f

I got mine for ~100 with no batteries and then the batteries were <200 from a battery wholesaler. Nice unit. I use an add in LAN card into to monitor it, but the serial line would work as well.

This runs the server and associated kit for about 45 minutes and then does an orderly shutdown. Prob. overkill, but hey, for that price, why not!
__________________
Want a nas, you may find my Esxi/Solaris ZFS NAS build thread of interest.
Quote:
Originally Posted by Stanza View Post
yeah well I just reported my own post...ferk....
Quote:
Originally Posted by Blinky View Post
If you have become content with the size of your e-penis, sticking clear of rack mounted stuff will save you heaps of $$$.
davros123 is online now   Reply With Quote
Reply

Bookmarks

Tags
home, nas, server, solaris, zfs

Sign up for a free OCAU account and this ad will go away!

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time now is 7:05 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd. -
OCAU is not responsible for the content of individual messages posted by others.
Other content copyright Overclockers Australia.
OCAU is hosted by Internode!