Wierdest problems you've ever experienced in a enterprise computing environment

[Squeezer]

One of our workgroups were conducting a training session and they grabbed a crusty old Switch as there were approx 15 people with PCs laptops int he training session. Someone managed to put a cat5 cable from one port of the switch straight into another port on the switch.

Causes a few problems as the switch further up the rung wasnt configured to stop broadcast storms

[The_Derro]

Originally Posted by Iceman

They were.. briefly.

"Why can't I copy files while he's copying files" "Why do I keep getting thrown out of MSN"

That one was a tricky one to track down. It sounds simple once you know the answer but who compares MAC addresses to check for duplication? It wasn't like they were all 00's or FF's..

Yep, been there, done that We had 5 PC's.. all on the same switch, all with the same MAC address. It presented in odd ways too, but wasn't consistent - a case of one user saying "Hey, I just lost access"..

It was only when I was trying to do a copy on all 5 at once and noticed that only a single PC would work at once.. check ipconfig on all and sure enough.. same MAC address on each one. Back to the wholesaler they went!

[tensop]

Previous network admin saved the AD DNS zonefile, inserted some odd invisible(to dnsmgmt atleast) characters, then re-loaded it back into dns+active directory on his last day.

[checkers]

Just discovered that the NIC that comes with Dell Vostro desktops (intel onboard something-or-other) breaks vmware. With a FreeBSD VM, I kept on getting errors that received packets were 4 bytes larger than the max allowed MTU... no matter what the MTU was set to on the host & in guest!

This confused me for a few days while I tested across ubuntu 64 and windows hosts and saw it fails, then moved to other identical machines (which since they are dells had other random NIC brands) to see it work fine.

[fR33z3]

oh...i bought a carton for this cockup in my younger days...

Take work laptop home, need to copy files from PC to xbox, but can't be arsed moving it closer to the PC, so connect xbox to laptop, and bridge wired + wireless connections. Brilliant. Works a treat.

Get back into work. Plug in. Splat. Everyone's workstation either grinding, or ground to a halt. Insane and unusual messages on peoples screens left right and centre. Drive mappings all failing.

I discovered what spanning tree was for that day!

[Andy_lee]

Here's one:
A small SAN consisting of 6 disk drawers via 2 FC loops to the controllers is having reliability problems. One loop of 3 drawers is fine. The 2nd has a problematic disk drawer which for some reason, can kill off all access to the other 2 drawers in that loop (despite redundant cabling)

We try:

• Updating all firmwares
• Reseat & replacing all fiber and SFPs to and from the drawer
• Reseat & replacing every part in and including the drawer seperately
• Moving all disks to other drawers

Still we get the same problem.

A power cycle of the SAN gets everything working fine again for a while, but the problem returns after a period of time (ranging from an hour to a week). We try take the drawer out of the loop entirely and problem disappears. Its now been running stable for the last 8 months or so.

[brainbug]

We have 3 sites connected through a 10mbit, 4mbit and 2mbit pipes. One day connections between all 3 sites were running really slow. We'd had a problem with a couple of switches before so we cycled the power and that did nothing. We spent half a day looking for the source, turns out the 10mbit site was trying to ram XP Sp3 to ~50 computers at the 4mbit site and about 60 at the 2mbit site.

The other guy here neglected to tell me that he added XPSP3 to the list of allowed updates on the WSUS server :P

[Xenon3d]

I ran into an interesting issue last Friday :

A client box needed a video card upgrade. I installed the card no problems, turned on the box and have no vga output. Checked the card, did the usual stuff, still no go. I unscrewed the pci slot screws and unplugged a nic and sound card, suddenly its working. I chalk it down to pci cards not playing nice, rearranged them into a config that worked, then screwed them in again. Now the vga stops working. I unscrew the cards, take them back out, all working. I put them back one at a time, all ok. Screw them back in, stops working again.

Eventually figured out that the cheap case chassis was dodgy, screwing the new vga card in was just enough to push the mobo onto the case chassis and short it out. Had me thinking I had gone crazy for awhile.

[TonyR]

This one goes waaaay back.

I worked for a company in England which had installed a new computer room. The mainframe type IBM MVS, SYS370, raised flooring etc.

First day of running , everything went perfectly until 3:30 pm when the system suddenly crashed. Restarted and all was OK.

Next day, same time, give or take a few minutes, down it went.

And the next day

And the next

And the next

etc.


Eventually, they realised that the system crash coincided with the arrival of the tea lady with her trolley and two big urns, one of tea and one of coffee. The fault was traced to a cable trapped under a support for the false floor. The cable was compressed when the trolley ran over it and a short happened bcause the insulation between two strands was slightly damaged. Once the trolley moved on, the cables separated and the system could restart successfully.

[checkers]

Originally Posted by TonyR

This one goes waaaay back.

Did I read this on Daily WTF recently??

[stalin]

Sounds like you had lovely relaxed computer room entry requirements. Our mainframe (zSeries, with 390 and 370's before that) rooms, and with the 390 and 370 it was rooms didn't let anyone in there without depositing your children as collateral. Thats what an old boss told me.....

Our series 9 MF's today are similar requirements, though now its presenting the old eyeball and thumb.. im sure the next step in security, will be when getting 'intruder lockout' it will cut your thumb off....

anyhoo

For those ultra newbies:

complete S/370 systems




complete s/390 system

[PsyKo-Billy]

There seems to be an aweful lot of empty space in those rooms...

To get back on topicsomewhat once upon a time as an IBM repair agent IBM insisted on sending us PSUs set to 110V. You only forgot to check a few times. Makes a fair old noise when your head is inches away when you're trying plug the power cable in... It was also fantastic when they started sending them at 230V, before referting sporadically to 110.

[Oppressa]

Originally Posted by Iceman

I'm sure there is much worse but the only one that immediately springs to mind was when a vendor once shipped me 15 PC's with identical MAC addresses..

Ahh yes we had the same... about 60 desktops so we had to go around to each one and run an update disc.

[DigitalStill]

Originally Posted by Squeezer

One of our workgroups were conducting a training session and they grabbed a crusty old Switch as there were approx 15 people with PCs laptops int he training session. Someone managed to put a cat5 cable from one port of the switch straight into another port on the switch.

Causes a few problems as the switch further up the rung wasnt configured to stop broadcast storms

Had a similar issue at one of my old jobs at a large retailer, only had about 20 workstations and 1 server, but all the workstations used POS that required TCPIP connectivity to function at all. The building was very large with many many network ports, after replacing the switch and router, finally decided to take a walk around and in an unused office behind a desk was a cat 5 cable coming out of one port in the wall into the port right next to it. Turns out the cleaner had done it while vacuuming!!

[TonyR]

Originally Posted by stalin

Sounds like you had lovely relaxed computer room entry requirements. Our mainframe (zSeries, with 390 and 370's before that) rooms, and with the 390 and 370 it was rooms didn't let anyone in there without depositing your children as collateral.

Actually, we had the same stringent requirements, but the outer area was also raised and the cable was trapped very close to the dividing wall, very close to where the trolley was parked after crushing the cable.