Consolidated Business & Enterprise Computing Rant Thread

Discussion in 'Business & Enterprise Computing' started by elvis, Jul 1, 2008.

  1. EvilGenius

    EvilGenius Member

    Joined:
    Apr 26, 2005
    Messages:
    10,804
    Location:
    elsewhere
    Vendor overnighted spare parts up and had it repaired the next day. We were running on mains for a night and a day.

    Nope, it was the 'master control board' apparently.

    Also I think the firefighters would have been able to smell the difference between standard electrical smoke and batteries that raised the white flag. That I've smelt myself, it's hard to mistake for anything else...
     
    Last edited: Jan 10, 2015
  2. tensop

    tensop Member

    Joined:
    Mar 26, 2002
    Messages:
    1,538
    pretty sure UPS's have their smoke in a raid5 array - one per phase
     
  3. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,471
    Location:
    qld.au
    I've had a sparkie wire up a new UPS incorrectly before, that was a fun experience. 3 wires, all with different colours. 3 terminals, all clearly labelled (Eaton UPS). Result? Active to earth will throw not only the breaker for the rack but the entire room.

    Hard shutdowns of lots of old Solaris systems just added to the fun.

    The sparkie initially tried to blame the UPS, but was quite apologetic when he arrived on site to find that one of his employees had mis-wired it. I don't think I ever received an invoice for that job :)
     
  4. Fred Nurk

    Fred Nurk Member

    Joined:
    Apr 5, 2002
    Messages:
    2,245
    Location:
    Cairns QLD
    I've managed to observe recent experiences where a product lets the magic smoke out badly, then after the inevitable tripped breaker is reset, it manages to keep working again. Doesn't exclude the device from being a piece of shit, but utterly astounding as to how the manufacturer never saw the failure mode in their testing, and even more so, that the device manages to continue functioning to a point after its shat itself.

    Its almost like the drunk taxi driver that still gets you home after taking a wrong turn and driving through a creek.
     
  5. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,471
    Location:
    qld.au
    Speaking of magic smoke, the worst example I've both smelt and seen involved 3 phase power filters for a building needing to meet certain security requirements. If you've smelt electronics letting the smoke out, multiply it by about 500. These EMI filters letting go where insanely horrid and well before OH&S were there to decide what's "safe". Every room in the building smelt. The aircon smelt. Outside if the building smelt. It was an entire week before the headaches went away!
     
  6. EvilGenius

    EvilGenius Member

    Joined:
    Apr 26, 2005
    Messages:
    10,804
    Location:
    elsewhere
    Sounds like the smell from a UPS fire I had in another of my offices a few years back. A couple of failed batteries surrendered their acid, shorted out on the casing and the place never smelled the same again.


    Click to view full size!


    Click to view full size!
     
  7. FiShy

    FiShy Member

    Joined:
    Aug 15, 2001
    Messages:
    9,682
    UMSS

    Uninterruptible magic smoke supply.
     
  8. cbb1935

    cbb1935 Guest

    :wired::sick: that's not good.

    The smell of electrical components surrendering is unmistakable. Add in battery acid and you're right, it'll never smell the same again. No amount of Febreeze or "Nilodour" is going to resolve that issue!

    Thankfully the world of power supplies has changed somewhat though.

    1. Less people using shit like "Shaw 400w I'll fire out a massive flame sometime in the next 12 months" edition, and more using quality power supplies.

    2. We no longer deal in "240v to the case button". That lead to some "interesting" shorts in years gone me. Like "I melted the front of my case" bad.
     
  9. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,471
    Location:
    qld.au
    It's an unmistakable smell :) I've seen some lovely examples where batteries let go. It seems that many (at least of the older types) of the the small UPS's don't have decent current overload protection.

    I've seen a little 500VA under the desk type job where someone had plugged in a fan heater to keep their feet warm. Of course when there was a power outage, it flicked over to battery and promptly tried to deliver 10A. It left burn marks in the carpet where the battery let itself go :)
     
  10. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    18,314
    Location:
    Canberra
    Paid FreeBSD/ZFS support in Australia. Who is there to go to? Currently in talks w/ ISH but curious to hear feedback on other solutions.

    Legit had a gutful of Nexenta and their lack of support - ran up a spare box w/ FreeNAS to take some overflow, and it runs rings around it.
     
  11. HobartTas

    HobartTas Member

    Joined:
    Jun 22, 2006
    Messages:
    1,054
    What are you interested in? If its primarily FreeBSD with ZFS thrown in as an added bonus then I can't suggest anything.

    On the other hand if its ZFS and your using FreeBSD because of it then I could suggest you get Solaris 11 and download the Solaris 10 Hardware Compatibility Tool and run it on your non-Oracle hardware, if that runs OK and passes the test then Oracle will support it if you purchase an OS license which I believe is about $1K p.a. per CPU last time I looked a while back.

    Hope this helps.

    Cheers
     
  12. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,471
    Location:
    qld.au
    Two years ago FreeNAS was lagging a bit in terms of performance and features, but now it's rock solid. Granted I only use it for D2D backups (about 70TB raw, mostly ZFS2) and not VM/SAN storage but I haven't had any issues.

    What particular issue are you trying to diagnose?
     
  13. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    18,314
    Location:
    Canberra
    On BSD? Nothing (yet).

    On Solaris/Debian - Horrendous Read/Write latency over NFS to VMware.

    In short - we err'd on the safe side and went with a paid, in theory supported, Nexenta option in March last year. Since then we have barely been able to keep it running in a fashion that can power a tiny sprinkling of low load, low iops VM's. The second you Move datastores, deploy from Template or sometimes just nothing at all - latency shoots up to 80-500ms. No Scrubs. No Resilvers.

    Kit is Supermicro E5-Xeon, 64GB ECC ram, 9211 HBA, 2x 8x2TB 7.2k SAS RaidZ2's in stripe, 2x 240GB Intel DC3500 L2ARC, 1xSTEC ZeusIOPS ZIL. OS is 2x 40GB Intel DC3500 in mirror. Intel 10GbE nic.

    No Dedup. lz4 compression (standard). 8k Records. 10G 9k jumbo NFS presented on its own lan (completely - not vlan'd).

    zpool pool0 iostat shows *crazy* latency on disk members when the wheels fall off. like upto 2 seconds. RDS Guests become literally unusable. Some windows VM's are so irresponsive - remote access software such as teamviewer declares them dead.

    On Friday we threw together a FreeNAS box. x53xx Xeons, 16GB Ram, 1015 HBA we had, 8x 4TB WD Red RaidZ2, 1x 120GB Intel 530 L2arc, 1x 120GB Intel 530 Zil. OS on 8GB USB - misaligned. 10GbE nic.

    Fine. Literally nothing wrong at all. Dumped the RDS, ADFS/SQL, 2x Exchange on it. Slam it with Storage vMotion and you see latencies peak at 30-40ms. You see some performance impact on the guests - but its still usable (and in my opinion, heavily oversubscribed).


    Nexenta don't want to talk to us because of some migration paperwork that they pulled out of thin air the second we logged a ticket. Local vendor is clearly out of their depth.

    We're just tired of a lack of support. We want people who know what they are doing and a product that works out of the box. We thought we'd get that with Nexenta - didn't.

    FreeNAS forums is literally 90% of the ZFS based assistance we've got over the last year - and after speaking w/ ISH, we're seriously looking forward to just paying $160/hr to get shit looked at when it doesn't work.


    RE: Solaris/Oracle. I don't know if i have faith in Solaris anymore. Its not Sun Hardware. Its not Oracle Hardware. Oracle don't give a shit about ZFS or Solaris.
     
  14. Jase

    Jase Member

    Joined:
    Jun 28, 2001
    Messages:
    196
    Location:
    Sydney 2081
  15. mooboyj

    mooboyj Member

    Joined:
    Sep 13, 2005
    Messages:
    1,048
    We have just started looking at/playing with Nexentastor and OmniOS with Napp-IT. I have also fired up FreeBSD but I don't think we'd go down that path.

    It is very interesting news about the poor support regarding Nexenta. We probably weren't going to bother with the support, but that poor performance is alarming:Paranoid:

    We have decided against any Linux/ZFS at this time. FYI, we were looking at 12x3TB disks in a RAID6/Z2 arrangement. It would be some underlying storage for scratch VMs and a "dumb" storage location for backups and the like.
     
  16. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    18,314
    Location:
    Canberra
    I don't think ZFS is lacking - particularly after the night and day difference between FreeBSD and Nexenta.

    But its early days w/ BSD.

    Nexenta performed well in testing too.
     
  17. QuakeDude

    QuakeDude ooooh weeee ooooh

    Joined:
    Aug 4, 2004
    Messages:
    8,565
    Location:
    Melbourne
    Guys,

    Question - for those of you with Tier 1 storage - what do you do with your old hardware once it runs out of maintenance?

    We have a Gen 2 XIV and a DS 8100 which I'm probably going to give to the metal recyclers unless I can find a better home. HW maintenance on the Gen 2 alone is around 30K a year (for 72TB) so its not exactly cheap to maintain.

    Could keep it as dev disk.. although we probably won't use it, not to mention it runs on 3 Phase power :Paranoid: I just....dunno.
     
  18. GumbyNoTalent

    GumbyNoTalent Member

    Joined:
    Jan 8, 2003
    Messages:
    9,940
    Location:
    Briz Vegas
    Sell it to a hardware vendor for parts.
     
  19. Urbansprawl

    Urbansprawl Member

    Joined:
    May 5, 2003
    Messages:
    846
    Yep scrap it and write it off, or if you really want to keep it get third party support from Interactive or the like.
     
  20. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,471
    Location:
    qld.au
    Most of the spec looks fine, except for the vdevs. 8 drives in RAIDZ2 isn't ideal (it's write performance is under 6 drive configs under some scenarios) because the striping isn't evenly distributed. From memory the perfect alignment for Z2 is 6 / 10 /18.

    There's 3 big misconceptions with ZFS. The first is that it's performance oriented. This is only correct with large numbers of vdevs. Have a look at Sun's original thumpers, this is where they've built the system from. The absolute pinnacle of ZFS is the data integrity. There's no write hole nor chance of silent corruption with ZFS.

    The second misconception is that RAIDZ performs like traditional RAID. It doesn't. IOPS wise RAIDZ (and Z2 / Z3) is probably somewhere around 30% worse off. Not an issue if it's planned for, you just need more vdevs. Vdevs add iops, that's the magic rule. For VM usage (and especially with low numbers of spindles) you want the drives in 2 x mirrored vdevs. For your configuration of 16 drives, that'd be 8 mirrored vdevs. You'd be looking at a 4x increase in iops (roughly) and lower latency.

    The last misconception is that the SLOG is a write cache. ZIL acceleration via a SLOG can certainly increase synchronous write speeds but it's not a magical fix. It's completely different to a write cache. The way ZFS works is to store the writes to RAM (where they can be reordered so that they're more sequential) and the ZIL provides the transactional assurity that the writes were completed. This is different to filesystem journaling as well.

    If you don't have high speed SSD's for the SLOG then the ZIL still occurs. The difference is that it writes it to the vdevs. This means that for synchronous calls, the latency is significantly higher. When you want to return the confirmation to the application quickly, this makes a massive difference.

    I'm not sure how much if this you know, but reiterating it at least for the sake of others.

    ZFS is excellent when configured correctly and used in the right context. Especially when you get in the 40-100+ spindles, you get the benefit of both performance and data integrity. For the same thing through a SAN vendor, you're paying massive dollars. For smaller (> 10 spindles) installs, ZFS can still give you that data integrity, but it won't perform the same as a 10 spindle traditional RAID.

    I'm hoping that btrfs picks up momentum a bit more to become a replacement. Oracle are closing all of their source code progressively (look at MySQL / Solaris for example). They want you to pay and that's really their only motivator.
     

Share This Page

Advertisement: