Network frock ups

Discussion in 'Business & Enterprise Computing' started by fR33z3, Jun 11, 2012.

?

Have you made an epic IT frockup?

  1. Yes I'm loud n proud of my frockups!

    36 vote(s)
    44.4%
  2. No way. I configure like a boss!

    17 vote(s)
    21.0%
  3. I ninja my coworkers so they make the frockup

    15 vote(s)
    18.5%
  4. Moar wasabi!

    13 vote(s)
    16.0%
  1. biatch

    biatch Member

    Joined:
    Jun 18, 2002
    Messages:
    1,692
    Location:
    North Brisbane
    There are ways to go about getting your point across, but sabotaging the network is not really one of them.
     
  2. ics

    ics Member

    Joined:
    Jul 8, 2002
    Messages:
    209
    Worst I've been involved in, was a deployment of a south American IT hub. The clients office was very large and was fitted out with a dedicated server room with multiple racks designed to provide connectivity for the company's entire South American operations. After the delivery of all the hardware the last part to be delivered was the massive UPS. Total fit out was several hundred thousand dollars.

    When the sparky went to wire up the UPS he realized the building didn't have enough power capacity. After contacting the power utility the client was advised that they couldn't provide said power capacity. The entire room sat without power for 2 years whilst the power utility upgraded it's network.

    IT manager at the time assumed that power wouldn't be an issue...

    T
     
  3. OP
    OP
    fR33z3

    fR33z3 Member

    Joined:
    Jul 16, 2001
    Messages:
    2,164
    Location:
    Perth
    Wow, nasty way to learn the old rule to never assume. I suppose thats why 'IT managers' shouldn't try and be engineers.
     
  4. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    14,844
    Location:
    Canberra
    In a similar situation, finance approved purchase of nice new shiny HP blade chassis, fully populated with specced out blades (many $100k).

    But wouldn't approve us getting in a sparky to swap some 2x10A GPOs to 15A GPOs as required for the chassis power supplies. Each 2x10A was on its own 20A circuit, so there was no technical problem. We just couldn't get in a sparky. Hardware went unused til it ran out of warranty and was disposed of. All for a few hundred dollars we couldn't get approved.
     
  5. KriiV

    KriiV Member

    Joined:
    Feb 24, 2011
    Messages:
    1,400
    Location:
    The 3-thousand

    And where was my call to come pick them up. :p

    Wow, might have been cheaper to buy and extended warranty?
     
  6. bubblegoose

    bubblegoose Member

    Joined:
    May 18, 2007
    Messages:
    4,508
    Location:
    Molesworth - Tasmania
    Man that is just crazy messed up... How is that even allowed to happen lol
     
  7. Hive

    Hive Member

    Joined:
    Jul 8, 2010
    Messages:
    6,355
    Location:
    AvE
    Did anyone actually talk to them about it?
     
  8. miicah

    miicah Member

    Joined:
    Jun 3, 2010
    Messages:
    7,184
    Location:
    Mount Cotton, QLD
    Should have just all chucked in on an office pool then claimed it on tax:

    Amount - Reason
    $100 - cheapskate finance dept.
     
  9. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    14,844
    Location:
    Canberra
    You really think we did nothing about it, for 3 years while struggling on already outdated and failing hardware.

    Welcome to large organsiation bureaucracy
    I'm out of there now (thank f***)
     
  10. The_Derro

    The_Derro Member

    Joined:
    Jul 15, 2001
    Messages:
    547
    Location:
    Sydney
    I work in the Cisco TAC :D Many many many stories, but had a similar one to above once.

    Was working on weekend shift one day and we had a flood of cases come in, all from 1 HUGE customer.. Fortune10 type company.

    I had a number of cases hit my queue.. all to do with failed PSU's/Power and how to recover the systems.

    Noting the sudden flurry of activity for all the teams on the floor, I asked the contact I had on the phone "Hey, WTF just happened? You guys have just raised like 40 cases in one hit.. WTF?"

    The guy on the other end replied, in the most downcast and annoyed voice I've ever heard... "We had a guy in to make some changes to the power in the DC... You know those big red buttons with warnings and 'DO NOT PUSH EXCEPT IN EMERGENCY' and multiple safety clips on them... well, he undid the multiple safety clips... and pushed the big red button..."


    Your fuckups are my day job :D Thanks for keeping life interesting guys.

    There is a quote from Terry Pratchett, I'm paraphrasing, but it's basically..

    'The last thing heard before the end of the Universe won't be a gun or bomb going off.. it will be a voice saying "Hey I wonder what happens if I press this button...." '
     
  11. KLSsandman

    KLSsandman Member

    Joined:
    Mar 15, 2002
    Messages:
    1
    I did this one just yesterday.

    Changing disk controller from SAS to paravirtualized on 61 virtual servers. Majority needed a helper disk creating with a new controller set to paravirtual so the server didn't blue screen on boot up. Once booted and the paravirtual drive installed, shutdown, remove helper disk and change the existing controller type to paravirtual and machine boots up with the new controller. All good. Until I removed what I though was the helper disk that turned out to be our HR Departments HR SQL database. :shock:

    Thankfully, the virtual machine is backed up every hour, but its that feeling, just at the point where you realise whats you've just done.
     
  12. PabloEscobar

    PabloEscobar Member

    Joined:
    Jan 28, 2008
    Messages:
    14,474

    I think everyone knows the feeling.

    When you are grabbing an image from something, and just after you hit enter, you realise you had the source and destination the wrong way round :).
     
  13. Yak

    Yak Member

    Joined:
    Jan 9, 2005
    Messages:
    249
    Amongst things:

    A) Tripped over wan connection to a branch office, plugged back in, and called Branch office to say "i've just fixed the network problem you've had"
    Got praise for excellent "service" on that one..

    B) late 90's. After refusing to remove the SMTP anti-relay server protecting our aging notes server, watched the "other" network administrator struggle as powertel shut our (2meg!!) internet connection down because their single mail server could not cope with the spam coming from our server.

    C) a colleague was testing deploying a screen save via GPO. To test he had set the delay to 2 seconds. Unfortuantly he deployed the "Test" to the whole domain over 16/32K WAN links. About 100 ppls round the country had to keep their keyboard/mouse moving continuously for 2 days till the new policy refreshed.

    D) +1 on the multicast..

    Yak.
     
  14. SpaceFrog

    SpaceFrog Member

    Joined:
    May 29, 2002
    Messages:
    2,125
    Location:
    Brunswick, Melb.
    I had no idea what this thread was about.

    I thought maybe you dressed up a router in a nice dress.

    so disappointed.

    I'm off, need to fap to an unsecured wifi access point in heels
     
  15. ShadowBurger

    ShadowBurger Member

    Joined:
    Feb 19, 2008
    Messages:
    2,280
    Location:
    Melbourne
    -Had a disk die in a RAID 6 array hosting files for the CAD guys.
    -Removed faulty disk
    -Had another spitting out SMART errors, so removed that too
    -Inserted two new disks and went away while waiting for the rebuild
    -While waiting, put the SMART error disk into another machine, ran some tests and formatted it to take home and use for myself in my cluster until it died
    -When nobody could access their files, I realised the server was configured in RAID 5 and the array had been "named" "RAID6" by mistake
    -The server hosting the backups was without mainboard at the time
    -Took 4 days before backup server was repaired
    -once repaired found that it did not contain backups of the server that failed
    -Had to explain to the CEO the CAD guys would need to re-do the complete designs for an entire truck-mount drill rig (~6mths work) and that they would be unable to start production of said rig worth 3mil+ until the designs were recompleted.

    Needless to say i almost shat out all my organs when I realised, and again when the backups weren't there. Fortunately I wasn't responsible for the backups and the guy who was hadn't bothered to do them, ergo i was not held responsible.
     
  16. Creekin

    Creekin (Taking a Break)

    Joined:
    Jun 21, 2003
    Messages:
    10,109
    yep :wired:
    pics or gtfo :weirdo:
    :lol: epic story :thumbup: and i know that feeling too :wired: :upset:
     
  17. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    14,844
    Location:
    Canberra

    Assuming it really was a RAID6 array.....
    Dunno why you'd pull two drives on a RAID6 array anyway. Yes, it should still work, but you've removed all redundancy, so if another drive we to die you'd have lost it all. The disk spitting errors, as long as the array hadn't dropped it already, was still providing some redundancy.

    One drive at a time.
     
  18. ShadowBurger

    ShadowBurger Member

    Joined:
    Feb 19, 2008
    Messages:
    2,280
    Location:
    Melbourne
    This is true, but being at the same site as the backups AND the low chance of having another disk go while rebuilding I felt could justify doing it, not to mention I wasn't actually aware the backups wouldn't be available. there is also backups of our backups but the backup server needs to either be functioning (or in the event its destroyed, replaced) in order to access them
     
  19. rusiakid

    rusiakid (Banned or Deleted)

    Joined:
    Apr 15, 2011
    Messages:
    196
    Location:
    rockhampton.qld.au
    My FUCK ups.

    Disabled the default domain policy, and our finance, and pretty much every other server in the company had disabled rdp connections and dns servers.....

    Note to self do not DISABLE default domain policy.

    Second one,

    Accidently set a WLAN to 192.168.24.93, instead of 27.93 /),- felt like an idiot.
     
  20. eyeLikeCarrots

    eyeLikeCarrots Member

    Joined:
    Jan 1, 2002
    Messages:
    4,386
    Location:
    Canberra Is Shit Sex: Yes
    Not exactly a network whoopsie, but a tiny mistake I made affected a entire university network gateway.

    When I worked level 2 support I was on this guy's laptop fixing something and the guy sakes me about 'file sharing' applications. So I tell him all about torrents and that sort of thing.

    The next day I get into the office and the networks guys are shitting rubber chickens because something was smashing the gateway. It was fairly quickly identified as an ip in the section I was working in the previous day.... The retard had promptly gone home and installed a client and pulled sown episodes of the west wing... Only he didn't turn the bloody thing off when he got to work the next day and plugged himself in....

    It would have saturated the entire upstream link but the he'd in the laptop could only spin so fast seeding the west wing torrent....
     

Share This Page

Advertisement: