ZFS drives in PC-BSD disappearing after moving unrelated drives

Discussion in 'Other Operating Systems' started by DrFrag, Dec 19, 2012.

  1. DrFrag

    DrFrag Member

    Joined:
    Aug 26, 2003
    Messages:
    1,309
    Location:
    South Australia
    This has been driving me nuts and I'm not sure how to search for the solution.

    I have a bunch of drives in my server, all individually formatted with ZFS and no RAID options. They're totally independent of each other.

    When I remove one of the drives, several of the others are missing the next time I reboot. Presumably because the da0, da1, da2 etc designations get reshuffled. I can bring them all back using the zpool import/export commands, except for one which ZFS says doesn't exist. The hardware is fine, the drive shows up in camcontrol, and I can zpool import it fine if I reinsert the other drive that I removed. This is annoying the hell out of me because the drives have nothing to do with each other and imo I should be able to pull one drive without the rest disappearing.

    How can I tell ZFS that the drive "da0" numbers have changed?
     
  2. mwil7034

    mwil7034 Member

    Joined:
    Jan 15, 2003
    Messages:
    612
    Location:
    Woy Woy
    Sounds like the PCBSD udev equivalent (devd?) is renaming them on reboot

    Might have to write a rule to manually assign them the same name each time

    Not a BSD user sorry so cant really elaborate to much :(

    Is there a means of specifying drives by UUIDs?
     
  3. samarium

    samarium Member

    Joined:
    Jun 6, 2007
    Messages:
    476
    I'm not a bsd user, however IIRC you can use some of the utilities to label the drives and you will get block devices created with the device names and you should use those names to reference the devices, not the da names. Have a look in the freebsd manual. You might have to use some special arguments with zpool import too. Let us know how you go.
     
  4. CaptainBlame

    CaptainBlame Member

    Joined:
    Jan 19, 2011
    Messages:
    286
    Unlike Linux the default behaviour of FreeBSD is to number your drives based on the slot you plugged the drive into. Therefore if you plug a new drive into the same sata slot it will get the same number of the replaced drive. For this reason you use labels for zfs so that when you change motherboards your drive numbering won't affect zfs.

    However this behaviour can be changed and I'm not sure if pcbsd does this by default. You should post the output of zpool status before and after pulling out the drive. Also you should post the contents of your /etc/sysctl.conf file.
     
  5. OP
    OP
    DrFrag

    DrFrag Member

    Joined:
    Aug 26, 2003
    Messages:
    1,309
    Location:
    South Australia
    I've found part of the problem. It looks like the drive I erased and the drive I pulled out weren't the same drive.

    I keep meticulous notes on where each drive is and what they contain - stickers on each drive, volume labels to match, a sheet of which drives are in which bays and what their serial numbers are. It appears three of the drives (and maybe others) are rotated out of sequence. This must have happened when I set it up, because the filesystem labels don't match the physical stickers. I've already wiped the drive I pulled out so I'm restoring the erased data from backups.

    I still have the problem of "da" numbers changing. After every reboot I have to export-import two of the drives to get them to show up. I'll have another look at this when my file restore is finished, although I might simply accelerate my migration away from BSD rather than fiddle with it.
     
  6. Stanza

    Stanza Member

    Joined:
    Jun 27, 2001
    Messages:
    2,899
    Location:
    Adelaide
    Not sure on PC-BSD

    But here is what I do on Different Linux systems

    You should instead of drives da0 da1 etc

    if PC-BSD has it

    use the /dev/disk/by-id ...then no matter where you attach the drive (eg switch cables or pull out a drive) the name can't change

    eg

    Have a look in the /dev/disk/by-id folder, it will show you all the drives the system recognises....it will also show you the serial numbers and models etc

    So it should be easy to work out which drive is which 1st off to set up your fstab etc

    say instead of having the normal

    Code:
    Filesystem     1K-blocks     Used Available Use% Mounted on
    /dev/sda1       46137344  7852804  35940860  18% /
    /dev/sdg1      961432904 57275228 855319636   7% /VM2
    /dev/sdb1      961432904   204452 912390412   1% /VM1
    
    I instead have

    Code:
    Filesystem     1K-blocks     Used Available Use% Mounted on
    /dev/disk/by-id/ata-STEC_MACH16_M16ISD2-50UCU_STM00014B73D-part1       46137344  7852804  35940860  18% /
    /dev/disk/by-id/ata-HUA721010KLA330_43W7625_42C0400IBM_GTA060PBHTURHE-part1      961432904 57275228 855319636   7% /VM2
    /dev/disk/by-id/ata-HUA721010KLA330_43W7625_42C0400IBM_GTA060PBHTVADE-part1      961432904   204452 912390412   1% /VM1
    
    1st partition of a 50gig STEC SSD mounted as root
    1st partition of a 1TB Hitachi mounted as VM2
    1st partition of a 1TB Hitachi mounted as VM1

    so for example say on the above....

    if the 50gig STEC SSD was on the 1st SATA port
    nothing was connected on the 2nd SATA port
    and the 2 x Hitachi drives were on 3rd and 4th SATA ports

    as you can see... if using normal /dev/sda or in your case /dev/da0 etc

    pop a drive onto the 2nd SATA port and things shuffle around (which naturally is unwanted in lots of cases)

    in my example above.... before changing to the /dev/disk/by-id way of doing things

    you can see there is already something screwy going on with the drive numbering

    Code:
    /dev/sda1 <<1st SATA port
    /dev/sdg1 <<3rd SATA port
    /dev/sdb1 <<4th SATA port
    
    you would think it would go sda sdb sdc eh?

    /dev/disk/by-id lock it in.... so even if you swap say sdg to the 2nd SATA port... the system will still find it and place it correctly.

    .
     
  7. kripz

    kripz Member

    Joined:
    Sep 29, 2004
    Messages:
    2,834
    Location:
    Near Frankston
  8. CaptainBlame

    CaptainBlame Member

    Joined:
    Jan 19, 2011
    Messages:
    286
    But labels resolve the issue. If you using /dev/da1 etc, you not using labels.
     

Share This Page

Advertisement: