Network frock ups

Discussion in 'Business & Enterprise Computing' started by fR33z3, Jun 11, 2012.

?

Have you made an epic IT frockup?

  1. Yes I'm loud n proud of my frockups!

    36 vote(s)
    44.4%
  2. No way. I configure like a boss!

    17 vote(s)
    21.0%
  3. I ninja my coworkers so they make the frockup

    15 vote(s)
    18.5%
  4. Moar wasabi!

    13 vote(s)
    16.0%
  1. fR33z3

    fR33z3 Member

    Joined:
    Jul 16, 2001
    Messages:
    2,164
    Location:
    Perth
    This is a thread for anyone who wants to comment on a network frockup. It could be as a user. It could be as an operator. Maybe as a manager or a designer. Maybe you're a server guy who wants to smack the network guys upside the head. It could even be your confessional if you were responsible for a good frockup in your younger days.

    I remember crippling the network in my first job, using ghost multicast. Terrible thing was I was telling everyone it's not me because I'm using "multicast" not "broadcast". Never mind the fact that the complaints always came in when I was ghosting. Learnt my lesson on "cause and effect"!

    Anyways, anyone else got a pearler or a good rant?

    Obligatory poll included.
     
  2. rainwulf

    rainwulf Member

    Joined:
    Jan 20, 2002
    Messages:
    4,222
    Location:
    bris.qld.aus
    do you have some aversion against swearing?
    Cos damn, when you are in that situation, you dont say "frock"
     
  3. Crusher

    Crusher Member

    Joined:
    Aug 27, 2001
    Messages:
    3,069
    Location:
    Sydney & Adelaide
    One of my biggest... building a new data centre for a company I worked for a part of a move. Needed a 200 amp 3 phase submain installed.

    Anyway, due to my oversight, it went onto the build spec as 1200 amp 3 phase... 400 amps a phase. So the sparkies dragged massive cables up 18 stories of a CBD skyscraper to supply said 1200 amps.

    Not my finest hour trying to explain the $50k electrical variance :(
     
  4. LostBenji

    LostBenji Member

    Joined:
    Oct 5, 2007
    Messages:
    6,078
    Location:
    Up a tower somewhere....
    Networking FUCK-UP's, don't be a skirt about saying it how it really is.....


    Best one I did was deliberate, placed a sneaky loopback between 3x 48-port Gbit switches in the ISP I worked for, the current NOC team (I wasn't NOC, I was the dumb rigger) were twats with no clue. Placed a loop between the servers (Oh, it was a bright red crossover cable to) to see how long it would take for these ID10T's to find the reason for the network jamming...

    No STP or RIP used, they felt it was outdated tech.

    2.5 weeks it took.



    Personal Fuck-Ups, installed four PtMP radios on a tower facing four points of the compass. Turned on the north and only North radio and had shitty connections. We thought it was an issue with the radios but turned out, I marked the cables in opposites, North was South and East was West. It was a shitty week, OK.
     
  5. roger895

    roger895 Member

    Joined:
    Aug 27, 2007
    Messages:
    178
    Location:
    Hobart, TAS
    Only just last year I had created an "all computers, including unknown" in SCCM, and was advertising (not forcing) task sequences to it for testing.

    Got the point where I was happy with testing so far, so I pushed one out to the same group, forcefully, which ended up being all machines.

    Realised my frock-up, shutdown the server, and waited for calls coming in from people with now wiped computers.

    Ended up being a couple of desktops, and 1 server (that we worked out the hard way wasn't backed up) out of about 1050 computers.

    Lesson learnt!
     
  6. s3kemo

    s3kemo Member

    Joined:
    May 13, 2003
    Messages:
    5,889
    Location:
    in a house
    Crashed routers/switches with too verbose logging before, but nothing that caused a serious outage from memory. I got lucky, but these days I'm super cautious with that sort of thing.
     
  7. b00n

    b00n Member

    Joined:
    May 2, 2003
    Messages:
    235
    Location:
    Brisbane
    Did close to the same but with application deployment for a large site with a small link, ended up flooding the link so shut down the sccm server.

    i always have a password to place on pxe task sequence so machines dont get wiped like this, as learnt in my deployment test farm
     
  8. bcann

    bcann Member

    Joined:
    Feb 26, 2006
    Messages:
    5,683
    Location:
    NSW
    was tracing some fibre across a meat processing plant, where every fibre termination point was unlabelled and was working out what went where. accidently plugged in a fibre and looped the network on a non STP enabled switch.... whole plant stopped (kill room floor and PLC's controlling machinery), 50 pc's and 3 servers immediately stopped responding.

    Was not a happy moment. took us an hour to get it all going again, including one PLC nic that completely crashed and needed a power cycle.
     
  9. Urbansprawl

    Urbansprawl Member

    Joined:
    May 5, 2003
    Messages:
    550
    Knocked loose a 10Gbps interconnect while doing some patching at the ISP I was working for. Got a call 23 seconds later saying 'Was that you'. Luckily the ISP had clue and it failed over as designed.
     
  10. Alfonzo

    Alfonzo Member

    Joined:
    Apr 7, 2003
    Messages:
    13,152
    Location:
    4152, Brisbane
    *knock wood* my network 'frock up' has been pretty minor so far, I was trying to tidy some cable runs in the rack in our Toowoomba office, and I brushed past the core switch power cable with the lightest of glances... and knocked the bugger out the socket. Site was down for about 5 - 10, but fortunately 90% of the office was out on site, the receptionist giggled and nobody external to the office noticed.
     
  11. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    12,327
    Location:
    Canberra
    That's what we call a crysitunity.

    Now your uber data centre can be a cash cow, rent out excess capacity.
    Make back that 50k in no time.
     
  12. Myst

    Myst Member

    Joined:
    Feb 26, 2004
    Messages:
    1,246
    Location:
    Hobart, Tasmania
    Damn!

    When our new core switches arrived, we were told by the supplier they were dual power supply, we just hadn't purchased them. So I put all of that through management and ordered redundant power supplies. Turns out it's just a fan unit in the Cisco 3750e, not a slot for another PSU. That was only a $1200 mistake though not 50k...!!
     
  13. PabloEscobar

    PabloEscobar Member

    Joined:
    Jan 28, 2008
    Messages:
    13,446
    Why would you do this?
     
  14. looktall

    looktall Working Class Doughnut

    Joined:
    Sep 17, 2001
    Messages:
    24,708
    because he's a dumb rigger.
     
  15. Gecko

    Gecko Member

    Joined:
    Jul 3, 2004
    Messages:
    2,715
    Location:
    Sydney
    The worst I've done would be reloading a core switch (I was configuring a trunk between it and a new satellite switch, lost track of which terminal window was which). It was probably the longest 5 minutes of my life while it rebooted.

    I've issued halt commands to the wrong boxes a few times as well - really need to pay more attention to the hostname on the screen...
     
  16. MrvNDMrtN

    MrvNDMrtN Member

    Joined:
    Dec 24, 2001
    Messages:
    1,355
    Location:
    SW Syd
    Yea.. who puts the shutdown button right next to the restart/logoff button.

    :lol::lol:
     
  17. Primüs

    Primüs Member

    Joined:
    Apr 1, 2003
    Messages:
    3,374
    Location:
    CFS
    I decided to clear a BGP route table in the middle of peak on accident once. Took a bit for traffic to start flowing while it re-learnt the internet, whoops.

    Boss the other week, we were in the DC, flicked the mains power off to test the UPS. Noticed the warning beep wasn't on and noted down to fix it. Started talking about another issue, walked out of the room. 20~ minutes later all services just went dark, WTF! ran down to the DC, silence. Dead. Freaking. Silence.
     
  18. maddhatter

    maddhatter Member

    Joined:
    Jun 27, 2001
    Messages:
    4,798
    Location:
    Mackay, QLD.
    I've got a couple.

    Not entirely sure what happened, but I was working on a server once clicking through a bunch of updates when a dialogue for the raid card jumped up in amongst it all and I may have clicked force rebuild; server shit itself instantly and wouldn't boot :/... was a medical centre; that was a long day. (Backups; albeit slow to restore worked beautifully.)

    I was at a largish site once doing some work on a PLC when someone accidentally bridged the corporate network with the control network... It turns out the PLC can serve out DHCP requests faster than the core DHCP server... This was a site with 30 or so remote sites and roughly 1500 seats...

    CTRL + ALT + DEL on a KVM with both Windows and Linux servers is never a good idea.

    Crashed many routers via too much logging, managed to sneak in a copy running start once after the fact once too; good times.
     
  19. Hive

    Hive Member

    Joined:
    Jul 8, 2010
    Messages:
    5,410
    Location:
    Some place far away
    Forgot to check the status light of the second power feed on a server when i yanked out the other to connect it to a new UPS

    *whooosh fans spin down*

    "Oh fuck me"
     
  20. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,460
    Location:
    qld.au
    Best one I've seen recently is Telstra, letting Dodo take down their entire network: http://bgpmon.net/blog/?p=554 Anytime I worry about making mistakes I remember what the largest ISP in Australia has done.

    As for personal frockups my list is too long. Some "highlights" are changing the config on a HA clustered system live, only to realise the multicast is blocked by the switch. Trying to recover from a "split brain" scenario when two systems are fighting for the same IP isn't overly fun.

    Wrong terminal window I've done twice now, thankfully just rebooting instead of deleting but still never nice to issue a reboot and suddenly there are alerts firing off everywhere.

    I've had a network switch let the smoke out so bad it threw the breaker on one section of the power board. Proceeding to try and reset the breaker meant it cut out the entire rail (thankfully most systems were dual powered).

    I've also seen remote router configs uploaded to the wrong device, it doesn't win you any friends when you drop all traffic to an entire remote region!

    For anyone that's seen RJ45 connectors on embedded devices, don't always assume it's for networking. I saw two switches killed and the tech went to grab a third (mumbling about faulty switches) before I jumped in to stop him :) They don't seem to enjoy a 24V tickle as much as the device it was intended to be plugged into.
     

Share This Page

Advertisement: