Supermicro AOC-USAS-L8i UIO 8 port sas/sata card

[oli]

iostat is pretty good to gauge the performance once things are running though...

[oli]

Seems they have lower utilisation. What makes you say "slower"?

[FatBoyNotSoSlim]

Originally Posted by oli

Seems they have lower utilisation. What makes you say "slower"?

Miss use of that word. I meant to say where are they outside of the std. deviation of the other drives? (Looking @ rrqm/s and r/s)

Not that I care / complaining. Speed it fine. Standard hdparm test is 500-500MB/s

[oli]

No idea on that one, hehe

[EvilGenius]

Just some pix of the $12.50 DX cables that work with this card:


Click to view full size!


And a comparison with the well known gigabyte sata cables:


Click to view full size!



They cost $12.50 for a reason

[MetalPhreak]

EvilGenius: Yeah but they work fine though (I ruled out cables as my issue)



ATTENTION ANYONE MOVING AN MDADM ARRAY

Before you do *anything*, write down which order your drives are along with their serial numbers.

For Example:

'cat /proc/mdstat'

md1 : active raid5 sdf[7] sdg[6] sdh[5] sde[4] sdd[3] sdc[2] sdi[1] sdb[0]
3418705472 blocks level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

Note down the order of sdb, sdi, sdc, sdd, etc

Then, use 'sudo smartctl -a /dev/sdX | grep Serial\ Number' for each drive, making a note of which Serial Number matches which drive letter. This way you have your array in order of Serial Numbers (which never change)

This way if something cocks up your superblocks, you can manually assemble the array again. Also, moving to a new controller will inevitably give you new sdX assignments for each drive, so now you have the Serial Numbers you can match it up with the new assignments, and recreate your array manually.

I got super confused because I didn't write things down properly, and I overwrote the superblocks. I had nfi where 4 drives went Only took me about 7 educated guesses to get it right.

Each guess let me mount it as an XFS partition with no errors, but once you browse the files inside things are seriously wrong, with lots of errors. DO NOT READ/WRITE ANY FILES. Kept going til I got combo with no errors and everything is fine

Ahaha oh man what a journey This is what I love about software raid though. Unless you do something stupid and write recklessly to a drive, its pretty much always recoverable

BTW, with no info on which order my drives went in, i'd need to try 8*7*6*5*4*3*2*1 = 40,320 combinations eeep~!

[Rampage101]

How does one go about 'bringing up' a array when the above does happen? My kernel auto find's the arrays, meaning I don't have a mdadm.conf file. Is there a command to make one (while its up and fresh)? How do you use the above info when re-creating the array or whatever if super-block's go walkabout?

[davros123]

Metal, that must have sucked! Great tip about using smart for the serial number. I have put some tape on each of my raid array drives and written on it critical details wrt the array...incl. position etc. That way if I am ever rearranging the PC and have to remove the drives, I know which connects to which sata and which should mount as which device.

Rampage, From the little I know, you no longer "need" and mdadm.conf (at least not in Ubuntu 9.10), as mdadm can assemble from the superblocks, but as you said, handy to have one incase this gets damaged.

I think you can do this with the following command...(not sure if you need to "sudo" these).

echo "#DEVICE partitions" > /etc/mdadm.conf 
#mdadm --detail --scan >>mdadm.conf

For example, my mdadm.conf (in a recent test) was as follows:

DEVICE partitions
ARRAY /dev/md0 level=raid6 num-devices=4 UUID=4e9f0798:70e409fb:11df29e6:1d7fe95b
   spares=1

Note the UUID's. These are unique identifiers of the the partitions and mean that it does not matter what order you have the HDD's (sdb can be physically moved about and appear as sde for example). mdadm will handle this using the UUID and assemble the array in the "correct" order.

At least that is what I understand it to be.

[Rampage101]

Had a feeling that's what I needed to do.

[MetalPhreak]

I still have to manually assembly my 500gb hdd array each boot, probably because theres bad info in mdadm.conf. Anyway, if you know the order in terms of drive letters this is the command:

" sudo mdadm --create /dev/md1 --assume-clean --level=5 --raid-devices=8 /dev/sdb /dev/sdi /dev/sdc /dev/sdd /dev/sde /dev/sdh /dev/sdg /dev/sdf "

You'll need to change it to match yours but its self explanatory. /dev/mdX, the raid level, the number of devices, and then the devices of the array LISTED IN THE CORRECT ORDER.

The most important bit here to use is --assume-clean so it doesn't try and recheck the array if it is wrong (in which case the rebuild would destroy it). Also, do note that this WILL OVERWRITE your current superblock info so make sure you have it right. I highly recommend doing 'mdadm -E /dev/sdX' on each device and saving the info it spits out, along with drive serial number<-> drive letter matching before you try the above method.

In most cases, if all your superblock info is intact, and mdadm doesn't automatically start them up, there are other easier solutions (or something silly preventing it).

mdadm --scan --assemble etc should work. If the uuid is wrong on one drive (a fairly common issue apparantly) you can simply write back the correct uuid to that one drive.

In other news, my 1tb array is currently expanding to 7 devices (and then 8 later) @ 25 MB/s which is impressive. When expanding it has to read from 6 devices and write new data to 7 devices, so each HDD is doing 25mb/s read AND write. So yeah these WD greens are getting a floggin

[oli]

Instead of using the drive letters to assemble the arrays why not use the UUID on the drives? That way it doesn't matter what port on the controller (or even what controller) they're on.

[davros123]

I must have something incorrect in my setup as I installed a new drive and the array did not build.

I needed to build the array manually, using the mdadm.conf

I think the command I used was...

sudo mdadm -stop /dev/md0
sudo mdadm -A /dev/md0 --config=/etc/mdadm.conf

So that's part way there....

But I stil needed to mount it manually when I add and remove drives...I am clearly missing something.

Anyone know how to set it up so it mounts using the UUID's in mdadm.conf when I add/remove stand alone drives?

[davros123]

Folks,
From my experience there is an issue running a raid 6 array with the AOC-USASLP-L8i under Ubuntu 9.1 (and I have also seen the issue in CentOS 5.4).

I see sense errors reported when smart information is accessed on the drives...which seems to happen alot. For me the issue is that the drives slow down alot when these "errors" appear and also it has been reported that the card can drop the drives..not good for an array. It can drop multiple drives from the array - taking the array off line.

The issue is described here in the bug report. and can be found reported in a number of forums across the web...incl. here on hard forums. Potential issue with AOC-USAS-L8i in Ubuntu? and http://kerneltrap.org/mailarchive/li...6578733/thread

From what I can see, people running them with Windows and Solaris do not seem to have an issue.

[Dragon_191]

I also had similar issues under SLES 11. I'm running an 11 drive raid 6 array and at one stage it was randomly dropping 3-4 disks at a time and brining the whole array down.

I changed a heap of settings while troubleshooting and managed to get the array rebuilt without any data loss. It has now been running without any issues for the past 2-3 months.

[MetalPhreak]

I get the same sense errors using hddtemp or smartmontools.

Try: cat /proc/mpt/version

mptlinux-3.04.10
Fusion MPT base driver
Fusion MPT SAS host driver


http://www.lsi.com/DistributionSyste...README_318.txt

Linux MPT Driver Release Notes
mptlinux-3.18.00.00-1
02/05/2009

I'm going to try compiling from source see if updated drivers are any good.


Found this blog: http://blog.loftninjas.org/2009/06/0...229-on-ubuntu/

Hopefully this might even fix my issues with the 500GB WD drives...