OpenSolaris/Solaris 11 Express/ESXi : BYO Home NAS for Media and backup images etc

[mike-s]

ok, This is my exact problem. Now trying to hose the zfs install so i can start again. Which is turning into quite the pain in the arse to do.

Further to this when i do a cfgadm i see that sata 1/4::dsk/c7t4d0 is currently connected and unconfigured.
The bastard has dropped it's /dev entries and the system doesn't know how to treat it as a disk. I seriously did not see that coming.

Will see what other gems i come up with on this, step #1 is to try and unhose the /dev entry for c7t4d0 and then see if things are recoverable, and THEN take it from there with recovery *sigh*

[edit]
Can't seem to clear it all, booting off a linux usb & currently running a dd of zero's to all 6 drives simultaneously across 99% of capacity ought to do the job. Next task after that is to just go and try and figure out this /dev/(r)dsk/c7t4s0 issue.[/edit]

[davros123]

hi mike, I must admit i am getting a bit lost in all the things going on with this...

when you say hose the zfs install...what do you mean?

the thread you linked to does not give much info. ofthe issue either (from a quick read)

My first thought is a hardware issue with that port...have you tested another card? Perhaps spin up a LiveCD of solaris and see if it also has an issue with that drive port...if so, it's a hardware fault.

[Stanza]

I am with Davros, either bad cable... bad port... bad drive etc

Pop the drive on another port (motherboard etc) if possible
LiveCD
or even a linux live CD.... wipe the drives (create new partitions and initialize) Boot Solaris Live CD and start again.

Failing that.

Pull all drives bar the one playing up.... pop it on a different port/cable and see if ZFS can create a pool on that single drive (that will tell you if the drive or cable or port is bad)

Work your way forward from there

.

[Stanza]

But hang on a second

What are you actually trying to use command wise for ZFS, are you following INTEL or SPARC instructions?

As you are referencing some things as c7t4d0, then in another step you are referencing them as c7t4s0 <<< ie not a disk but a slice of a disk ??

Sparc instructions for ZFS are slightly different if you are following ones off of the interweb.... it's a good GOTCHA

ie here

NAME                      STATE     READ WRITE CKSUM
        storage                   DEGRADED     0     0     1
          raidz1-0                DEGRADED     0     0     6
            c7t0d0                ONLINE       0     0     0
            13127497109634687241  UNAVAIL      0     0     0  was /dev/dsk/c7t4d0s0
            c7t2d0                ONLINE       0     0     0
            c7t3d0                ONLINE       0     0     0
            c7t1d0                ONLINE       0     0     0
            c15t0d0               ONLINE       0     0     0  36.5K resilvered

.

[mike-s]

I don't know why it was referring to it as a slice, all drives were put in as entire disk volumes. I was unable to recover anything from the zfs volume, it had totally turned up its toes and died a horrible death with one drive dropping for some weird reason while there was a drive accidentally unplugged.

I tried imports, i tried scrubs, i tried many things. I couldn't get it to acknowledge the sata port 1/4 (c7t4d0) until i had dd'd 0's over all drives in the array from linux and THEN deleted zpool.cache. Even with the drives zeroed it still thought the array was there but offline. Once i did my next reboot, all drives are back to normal behaviour and cfgadm detects the sata device correctly again.

Since then i have replaced the sata cable of the offending drive as that is the cheapest/fastest method to try and fix this issue. I already have one faulty sata port on the motherboard, another one means i need to go out and buy another pci-e controller card to replace it (fortunately this board has a shit-tonne of pci & pci-e slots i can utilise if i'm that unlucky). At no point prior to the cable swap did i have a problem accessing the drive under linux.

I think there must have been some funky shit happening when this went down as my router lost it's config at around the same point in time and i had to reconfigure it again.

I'll be checking the zpool status every day for the next fortnight to make sure everything is ok again and if needed will export the pool and unplug drives until a new controller/drive can be obtained IF the need arises. I am putting it down to just one-of-those-things. As a side note I am still surprised at the difficulty that solaris gives you in nuking a corrupted array as it continually blocks you doing so, citing i/o suspended, even when you try to force it to do so.


pool: storage
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c7t0d0 ONLINE 0 0 0
c7t1d0 ONLINE 0 0 0
c7t2d0 ONLINE 0 0 0
c7t3d0 ONLINE 0 0 0
c7t4d0 ONLINE 0 0 0
c15t0d0 ONLINE 0 0 0

errors: No known data errors
[edit]Bombing out again on a different sata port (the one i prev thought was faulty). I think the drive has likely grenaded. Now trying to start everything up with the suspect drive unplugged & running in degraded so i can see if it will run one disk down as a test with duplicate data until i get a spare (reconsolidating data as we speak to free one up).[/edit]

[mike-s]

*sigh*
RMA time, got a WD & Seagate here, both of which are kaput. One starts up seemingly ok, but bombs & crashes everything when you try to use it.

The other is giving SMART errors at startup and doesn't allow you to format it. Fortunately the second was in an external enclosure and was at the time empty of data when it decided to crap out.

Now if only the WD website wasn't timing out. I have the drive details & can try it again tomorrow. Already submitted the seagate RMA, funny enough, this was a drive i was sent as an RMA 9 months ago .

[edit]I mucked the plugs around a bit and have to doublecheck which was the one that crapped out as I've managed to confuse myself again with this, then i'll raise the rma. Am at work at the moment, so can't do it today[/edit]

[mike-s]

Quick question, RBAC. I just cannot su to root from my account, yet i have roles & profiles seemingly set up correctly.

/etc/user_attr:
mike::::type=normal;profiles=Primary Administrator;roles=root
root::::type=role;auths=solaris.*,solaris.grant;profiles=All;lock_after_retries=no;min_label=admin_low;clearance=admin_high

/etc/security/auth_attr is untouched

mike@share:~$ pfexec reboot
reboot: permission denied
mike@share:~$ su -
Password:
Roles can only be assumed by authorized users
su: Sorry
mike@share:~$


Any ideas? i could change root to a normal account instead of a role, but i don't really want to do that unless i have to.

[judge25]

Thought I would try these questions here instead of opening a new topic may not be a good idea, anyhow...

About to create my fist ever zpool/vdev on Solaris 11.
I wanted to get 512b drives (hitachi) but impatient waiting for stock so just got my self some cheap samsung spinpoint f4's in the short term interim, knowing full well that they are 512e drives.
Picked up 8 drives for raidz2 array.

What I am wondering about is performance issues.
2 things come up. 4k alignment and Magic Disk number.

How do I put this.... are they mutually exclusive or both things I should look at?

Should I align to 4k **and** have a magic disk number, i.e 6 or 10 instead of 8. Or if I do one or the other the performance hit from 4k with 512 emulation does not become an issue?
I believe I may need to do both!

Any advice greatly appreciated before I go too far or buy extra drives that I don't really need.

Thanks!

[davros123]

Hi Judge, copngrat's on taking a plunge into the zfs pool (pardon the pun).

Given it takes all of 10 seconds to create and share a pool, I would just do that and see what speeds you get - as ymmv. Chuck on some data, run a scrub, look at iostat, then when it's done, try some cifs transfers, then some nfs, then perhaps an iperf if needed...then if you are really keen install and run Bonnie ++ etc.

you should see speeds of around 500MB read and just a bit less write.

re, the magic number, this does make small diff...but not enough to justify buying extra drives.

mike-s, not sure...was so long ago, I set it up so I could use pfexec (which is not available by default) using steps like this...http://lildude.co.uk/solaris-11-expr...assword-gotcha

[judge25]

Thanks Davros, I started this build (unknowingly) on the weekend before HDD prices went up because of the floods (could have purchased drive component before the rise but didn't know what was happening).

I couldn't wait any longer and had to take the final dive....:-)

I suppose you are right, I can test and see what results are returned. At worst I would have to wait another 4-5 days before I could address any drive related issues. I think I can do that seeing as I have waited this long already

[mike-s]

Cheers, still looking into this. I have no idea why it's not bloody working. Absolutely nothing in the configs to indicate why it isn't functioning.


I might try removing the config files and then re-creating the auth file and see what happens.

[davros123]

yeah, just remoted in and had a look at my setup, and I did convt it into a normal user from a role...

I know it's not "the right thing to do" but I got used to being able to pfexec bash and run commands from there. After all, it is just a home server

judge, why would you need to wait? it literally takes seconds to build and destroy zfs arrays...no waiting here...and the test transfers would only take a few minutes....or am I missing something?

At worst I would have to wait another 4-5 days before I could address any drive related issues

ps. bummer about the timing...a mate and I had the flip side luck...we ordered 8 x 3TB drives on the day before the massive price hikes!

[judge25]

Hehe It's just picking them up is a little travel distance, and I can't get there until the weekend

Originally Posted by davros123

judge, why would you need to wait? it literally takes seconds to build and destroy zfs arrays...no waiting here...and the test transfers would only take a few minutes....or am I missing something?

[mike-s]

Found the dud drive, is a 7200.11. It could be stuck in the BSY state as it isn't even spooling up, but i cannot be arsed with taking it out of that state even though i have setup a cable to do this. So have gotten an RMA from their website and will be dropping it off with my other dud drive (a 7200.12 which i got as a refurb in late 2010!) @ UPS Mascot.

Will be buying a new hard drive this week to replace it and will then have to figure out what to do with the two new 500's once i get an RMA.

[mike-s]

New drive installed to replace the one that turned up it's toes & pool erased & re-created to make 100% sure it was all ok. The two replacement RMA drives will be added as removables and used as secondary backup to mitigate any massive data loss ever happening again.

Lesson learned, that was not fun.

  pool: storage
 state: ONLINE
 scrub: scrub completed after 0h0m with 0 errors on Sat Feb 18 03:07:11 2012
config:

        NAME         STATE     READ WRITE CKSUM
        storage      ONLINE       0     0     0
          raidz1-0   ONLINE       0     0     0
            c7t0d0   ONLINE       0     0     0
            c7t1d0   ONLINE       0     0     0
            c7t2d0   ONLINE       0     0     0
            c7t3d0   ONLINE       0     0     0
            c7t4d0   ONLINE       0     0     0
            c15t0d0  ONLINE       0     0     0

errors: No known data errors