Network frock ups

fR33z3 · 11th June 2012, 9:43 PM

This is a thread for anyone who wants to comment on a network frockup. It could be as a user. It could be as an operator. Maybe as a manager or a designer. Maybe you're a server guy who wants to smack the network guys upside the head. It could even be your confessional if you were responsible for a good frockup in your younger days.

I remember crippling the network in my first job, using ghost multicast. Terrible thing was I was telling everyone it's not me because I'm using "multicast" not "broadcast". Never mind the fact that the complaints always came in when I was ghosting. Learnt my lesson on "cause and effect"!

Anyways, anyone else got a pearler or a good rant?

Obligatory poll included.

rainwulf · 11th June 2012, 10:53 PM

do you have some aversion against swearing?
Cos damn, when you are in that situation, you dont say "frock"

Crusher · 11th June 2012, 10:58 PM

One of my biggest... building a new data centre for a company I worked for a part of a move. Needed a 200 amp 3 phase submain installed.

Anyway, due to my oversight, it went onto the build spec as 1200 amp 3 phase... 400 amps a phase. So the sparkies dragged massive cables up 18 stories of a CBD skyscraper to supply said 1200 amps.

Not my finest hour trying to explain the $50k electrical variance

LostBenji · 12th June 2012, 7:25 AM

Networking FUCK-UP's, don't be a skirt about saying it how it really is.....

Best one I did was deliberate, placed a sneaky loopback between 3x 48-port Gbit switches in the ISP I worked for, the current NOC team (I wasn't NOC, I was the dumb rigger) were twats with no clue. Placed a loop between the servers (Oh, it was a bright red crossover cable to) to see how long it would take for these ID10T's to find the reason for the network jamming...

No STP or RIP used, they felt it was outdated tech.

2.5 weeks it took.

Personal Fuck-Ups, installed four PtMP radios on a tower facing four points of the compass. Turned on the north and only North radio and had shitty connections. We thought it was an issue with the radios but turned out, I marked the cables in opposites, North was South and East was West. It was a shitty week, OK.

roger895 · 12th June 2012, 8:34 AM

Only just last year I had created an "all computers, including unknown" in SCCM, and was advertising (not forcing) task sequences to it for testing.

Got the point where I was happy with testing so far, so I pushed one out to the same group, forcefully, which ended up being all machines.

Realised my frock-up, shutdown the server, and waited for calls coming in from people with now wiped computers.

Ended up being a couple of desktops, and 1 server (that we worked out the hard way wasn't backed up) out of about 1050 computers.

Lesson learnt!

s3kemo · 12th June 2012, 10:10 AM

Crashed routers/switches with too verbose logging before, but nothing that caused a serious outage from memory. I got lucky, but these days I'm super cautious with that sort of thing.

b00n · 12th June 2012, 10:32 AM

Quote:

Originally Posted by roger895

Only just last year I had created an "all computers, including unknown" in SCCM, and was advertising (not forcing) task sequences to it for testing.

Got the point where I was happy with testing so far, so I pushed one out to the same group, forcefully, which ended up being all machines.

Realised my frock-up, shutdown the server, and waited for calls coming in from people with now wiped computers.

Ended up being a couple of desktops, and 1 server (that we worked out the hard way wasn't backed up) out of about 1050 computers.

Lesson learnt!

Did close to the same but with application deployment for a large site with a small link, ended up flooding the link so shut down the sccm server.

i always have a password to place on pxe task sequence so machines dont get wiped like this, as learnt in my deployment test farm

bcann · 12th June 2012, 11:10 AM

was tracing some fibre across a meat processing plant, where every fibre termination point was unlabelled and was working out what went where. accidently plugged in a fibre and looped the network on a non STP enabled switch.... whole plant stopped (kill room floor and PLC's controlling machinery), 50 pc's and 3 servers immediately stopped responding.

Was not a happy moment. took us an hour to get it all going again, including one PLC nic that completely crashed and needed a power cycle.

Urbansprawl · 12th June 2012, 11:33 AM

Knocked loose a 10Gbps interconnect while doing some patching at the ISP I was working for. Got a call 23 seconds later saying 'Was that you'. Luckily the ISP had clue and it failed over as designed.

Alfonzo · 12th June 2012, 11:39 AM

*knock wood* my network 'frock up' has been pretty minor so far, I was trying to tidy some cable runs in the rack in our Toowoomba office, and I brushed past the core switch power cable with the lightest of glances... and knocked the bugger out the socket. Site was down for about 5 - 10, but fortunately 90% of the office was out on site, the receptionist giggled and nobody external to the office noticed.

cvidler · 12th June 2012, 11:42 AM

Quote:

Originally Posted by Crusher

One of my biggest... building a new data centre for a company I worked for a part of a move. Needed a 200 amp 3 phase submain installed.

Anyway, due to my oversight, it went onto the build spec as 1200 amp 3 phase... 400 amps a phase. So the sparkies dragged massive cables up 18 stories of a CBD skyscraper to supply said 1200 amps.

Not my finest hour trying to explain the $50k electrical variance

That's what we call a crysitunity.

Now your uber data centre can be a cash cow, rent out excess capacity.
Make back that 50k in no time.

Myst · 12th June 2012, 2:29 PM

Quote:

Originally Posted by Crusher

One of my biggest... building a new data centre for a company I worked for a part of a move. Needed a 200 amp 3 phase submain installed.

Anyway, due to my oversight, it went onto the build spec as 1200 amp 3 phase... 400 amps a phase. So the sparkies dragged massive cables up 18 stories of a CBD skyscraper to supply said 1200 amps.

Not my finest hour trying to explain the $50k electrical variance

Damn!

When our new core switches arrived, we were told by the supplier they were dual power supply, we just hadn't purchased them. So I put all of that through management and ordered redundant power supplies. Turns out it's just a fan unit in the Cisco 3750e, not a slot for another PSU. That was only a $1200 mistake though not 50k...!!

PabloEscobar · 12th June 2012, 7:25 PM

Quote:

Originally Posted by LostBenji

Best one I did was deliberate, placed a sneaky loopback between 3x 48-port Gbit switches in the ISP I worked for, the current NOC team (I wasn't NOC, I was the dumb rigger) were twats with no clue. Placed a loop between the servers (Oh, it was a bright red crossover cable to) to see how long it would take for these ID10T's to find the reason for the network jamming...

Why would you do this?

looktall · 12th June 2012, 7:51 PM

Quote:

Originally Posted by PabloEscobar

Why would you do this?

because he's a dumb rigger.

Gecko · 12th June 2012, 8:22 PM

The worst I've done would be reloading a core switch (I was configuring a trunk between it and a new satellite switch, lost track of which terminal window was which). It was probably the longest 5 minutes of my life while it rebooted.

I've issued halt commands to the wrong boxes a few times as well - really need to pay more attention to the hostname on the screen...

View Poll Results: Have you made an epic IT frockup?
Yes I'm loud n proud of my frockups!	36	44.44%
No way. I configure like a boss!	17	20.99%
I ninja my coworkers so they make the frockup	15	18.52%
Moar wasabi!	13	16.05%
Voters: 81. You may not vote on this poll

11th June 2012, 9:43 PM	#1
fR33z3 Member Join Date: Jul 2001 Location: Perth Posts: 2,092	Network frock ups This is a thread for anyone who wants to comment on a network frockup. It could be as a user. It could be as an operator. Maybe as a manager or a designer. Maybe you're a server guy who wants to smack the network guys upside the head. It could even be your confessional if you were responsible for a good frockup in your younger days. I remember crippling the network in my first job, using ghost multicast. Terrible thing was I was telling everyone it's not me because I'm using "multicast" not "broadcast". Never mind the fact that the complaints always came in when I was ghosting. Learnt my lesson on "cause and effect"! Anyways, anyone else got a pearler or a good rant? Obligatory poll included. __________________ I need a Wiimote. Broken is ok!

11th June 2012, 10:53 PM	#2
rainwulf Member Join Date: Jan 2002 Location: bris.qld.aus Posts: 2,651	do you have some aversion against swearing? Cos damn, when you are in that situation, you dont say "frock" __________________ derp

12th June 2012, 8:34 AM	#5
roger895 Member Join Date: Aug 2007 Location: Claremont, Tasmania Posts: 94	Only just last year I had created an "all computers, including unknown" in SCCM, and was advertising (not forcing) task sequences to it for testing. Got the point where I was happy with testing so far, so I pushed one out to the same group, forcefully, which ended up being all machines. Realised my frock-up, shutdown the server, and waited for calls coming in from people with now wiped computers. Ended up being a couple of desktops, and 1 server (that we worked out the hard way wasn't backed up) out of about 1050 computers. Lesson learnt! __________________ Dell XT3 - SSD - 8GB DDR3 Dell Optiplex 9010 - SSD - 8GB DDR3 - i7 3770 - 2x 7470 - 3x BenQ G2220HD

12th June 2012, 11:39 AM	#10
Alfonzo Member Join Date: Apr 2003 Location: Carindale, BrisVegas Posts: 10,835	knock wood my network 'frock up' has been pretty minor so far, I was trying to tidy some cable runs in the rack in our Toowoomba office, and I brushed past the core switch power cable with the lightest of glances... and knocked the bugger out the socket. Site was down for about 5 - 10, but fortunately 90% of the office was out on site, the receptionist giggled and nobody external to the office noticed. __________________ tweet

11th June 2012, 10:58 PM	#3
Crusher Member Join Date: Aug 2001 Location: Sydney Posts: 3,061	One of my biggest... building a new data centre for a company I worked for a part of a move. Needed a 200 amp 3 phase submain installed. Anyway, due to my oversight, it went onto the build spec as 1200 amp 3 phase... 400 amps a phase. So the sparkies dragged massive cables up 18 stories of a CBD skyscraper to supply said 1200 amps. Not my finest hour trying to explain the $50k electrical variance

12th June 2012, 7:25 AM	#4
LostBenji (Taking a Break) Join Date: Oct 2007 Location: Not around here anymore.. Posts: 5,298	Networking FUCK-UP's, don't be a skirt about saying it how it really is..... Best one I did was deliberate, placed a sneaky loopback between 3x 48-port Gbit switches in the ISP I worked for, the current NOC team (I wasn't NOC, I was the dumb rigger) were twats with no clue. Placed a loop between the servers (Oh, it was a bright red crossover cable to) to see how long it would take for these ID10T's to find the reason for the network jamming... No STP or RIP used, they felt it was outdated tech. 2.5 weeks it took. Personal Fuck-Ups, installed four PtMP radios on a tower facing four points of the compass. Turned on the north and only North radio and had shitty connections. We thought it was an issue with the radios but turned out, I marked the cables in opposites, North was South and East was West. It was a shitty week, OK.

12th June 2012, 10:10 AM	#6
s3kemo (Taking a Break) Join Date: May 2003 Location: in a house Posts: 5,929	Crashed routers/switches with too verbose logging before, but nothing that caused a serious outage from memory. I got lucky, but these days I'm super cautious with that sort of thing.

12th June 2012, 11:33 AM	#9
Urbansprawl Member Join Date: May 2003 Posts: 139	Knocked loose a 10Gbps interconnect while doing some patching at the ISP I was working for. Got a call 23 seconds later saying 'Was that you'. Luckily the ISP had clue and it failed over as designed.

12th June 2012, 8:22 PM	#15
Gecko Member Join Date: Jul 2004 Location: Sydney Posts: 2,722	The worst I've done would be reloading a core switch (I was configuring a trunk between it and a new satellite switch, lost track of which terminal window was which). It was probably the longest 5 minutes of my life while it rebooted. I've issued halt commands to the wrong boxes a few times as well - really need to pay more attention to the hostname on the screen...