Strange Network Issue

Discussion in 'Networking, Telephony & Internet' started by Piroteknik, Dec 6, 2011.

  1. Piroteknik

    Piroteknik Member

    Joined:
    Jun 17, 2002
    Messages:
    379
    Location:
    Bat Country
    Hi guys,

    I am currently trying to figure out an issue with a reasonably large small business network.

    The main switches (2 x Cisco and 4 x Netgear, all managed) are all running flatout all the time.

    As most would know the indication lights on a switch, especially a managed switch should only be showing the path that a packet would take, in this case ALL ports that have connectivity are lighting up together, almost as if there were collisions going on.

    The strange thing about this is there appears to be no network degradation/slowing at all. Everything is working fine otherwise.

    Its almost as if there is packets being flooded that none of them can identify.

    We have tried powering them off one at a time with no luck, then tried powering them all down together and bringing them back up.

    We have also tried shutting down everything (Except for a few of the servers, as they could not be taken down, we will try this tomorrow in the early hours).

    We are hoping we can isolate the issue to one of them, but in the meantime, has anyone got any other ideas to try? I'm asking now in the hopes of avoiding an early start on Thursday (Definately happening tomorrow lol.)

    Thanks guys.
     
  2. SpudBoy

    SpudBoy Member

    Joined:
    Jul 30, 2001
    Messages:
    5,281
    Location:
    Under the Bed.
    are all the ports on the same broadcast domain?
     
  3. s3kemo

    s3kemo Member

    Joined:
    May 13, 2003
    Messages:
    5,889
    Location:
    in a house
    Tried popping a box with Wireshark on there during a quiet traffic period and seeing whether there's any broadcast traffic flying around?
     
  4. OP
    OP
    Piroteknik

    Piroteknik Member

    Joined:
    Jun 17, 2002
    Messages:
    379
    Location:
    Bat Country
    yes.

    Single Domain, they are basically just patched one to the next e.t.c.

    Have verified there are no loop backs either.

    Will be trying this tomorrow morning also. Cheers for the suggestion.
     
    Last edited by a moderator: Dec 6, 2011
  5. SpudBoy

    SpudBoy Member

    Joined:
    Jul 30, 2001
    Messages:
    5,281
    Location:
    Under the Bed.
    this - it looks very much like you have lots of broadcasts being sent.
     
  6. Gunna

    Gunna Member

    Joined:
    Dec 25, 2001
    Messages:
    7,044
    Location:
    Brisbane
    Got any old network printers on the network?

    I've seen 2 of these flood the network with broadcast traffic. Had to replace the NICs on them and the traffic stopped.

    I would try turning of each device one at a time. Power one on and see if the lights all flash at once. Power the 2nd on on and wait. Power the 3rd on etc.

    Then power them down, bring the 2nd switch on 1st, then the 3rd, 4th, then the intial 1st switch, do this to make sure it wasnt switch 1 in your 1st group causing the issue.

    See if you can find out which switch has the devices flooding the network.

    You can then narrow it down to machines and devices on a specific switch and trouble shoot from there.
     
  7. SpudBoy

    SpudBoy Member

    Joined:
    Jul 30, 2001
    Messages:
    5,281
    Location:
    Under the Bed.
    this can also be determined by using wireshark.

    if you see a high number of packets being sent to the broadcast address from specific source addresses its really easy to track it down.
     
  8. MrvNDMrtN

    MrvNDMrtN Member

    Joined:
    Dec 24, 2001
    Messages:
    1,355
    Location:
    SW Syd
    This is pretty normal in a flat network.

    Do you have any tools that show your switches running "flatout"?

    Using lights as an indicator is not the best way to determine how busy the network is.
     
  9. Lodion

    Lodion Member

    Joined:
    Dec 3, 2002
    Messages:
    18
    Location:
    Perth
    Lights on a switch all blinking at once is perfectly normal. Switches will broadcast for any unknown unicast MAC address. So the first time a device on the network is sent traffic, the first frame in the sequence is flooded throughout the network.

    When the receiving host receives it, it responds and the switches learn its MAC. The next from is sent unicast.

    As MAC address entries in a switches table are aged out, frames are flooded.

    Unless you are seeing high traffic levels or unexpected performance, it is nothing to worry about.

    Or you could have some misbehaving software on the network constantly sending broadcast traffic... NetBIOS, SMB/CIFS etc have a tendency to do this.

    edit: forgot to add, check the logs on your Cisco switches. Might shed some light on the issue. Otherwise wireshark is your best friend.
     
  10. g@z

    g@z Member

    Joined:
    Jul 27, 2001
    Messages:
    2,132
    Location:
    Melbourne
    I've seen something like this, but it was from a faulty module and the network had redundant links using spanning tree to prevent loops. If you power cycled one of the core switches everything would come up ok, the lights blinked like a normal network would blink it's lights for a minute then every light on every module on every switch just went hard on and stayed that way. It caused the business to send all their staff home for half a day as the network was unusable. A company that controlled tens to hundreds of millions of dollars - they were relying on default next business day warranty support.

    Anyway, checked the logs and it indicated an error from a specific module. Popped the module and bingo, back to normal. Module replaced the next day under warranty.

    Another method that helped me out back in the days of 12 port hubs was just to pull the links between each hub until the offending hub was isolated. Then each connection on that hub to identify the port. At the end of this port was a sales man demoing a network analyser with a 10 foot phone cable plugged into a LAN port.

    Anyway, I'm waffling. Try the simple isolation method or as stated, Wireshark should help :)

    Regards,
    g@z.
     
  11. taldoren

    taldoren Member

    Joined:
    Mar 27, 2005
    Messages:
    358
    Any VC units etc on the network ?

    Had similar issue before, turned out someone decided that Multicast was the best option for the VC units....:confused:
     
  12. Miff88

    Miff88 Member

    Joined:
    Nov 10, 2010
    Messages:
    427
    Location:
    Newcastle
    broadcasts by looks.

    According to cisco you should segment your network off at 250 devices, meaning VLANing.

    Since those switches are managed you can create your vlans on them. the next question is are any of them L3? i ask this because with a cisco layer 3 switch you can turn on IP Routing. That allows you to route between vlans without the need for a router on a stick topology.

    i have configured the network at one of my smaller (50-150 users) sites to be like this;

    **this can be configured on a core switch you allocate**
    MGT VLAN 10 /23 (servers, AP's, printers, etc) 10.20.0.1
    DATA VLAN 20/23 (all PC's) 10.20.2.1
    VOICE VLAN 30 /23 (Phones, CCME) 10.20.4.1

    Then you simple allocate which port you want in which VLAN with.
    pick the interface you want
    switch mode access
    switch acc mode vlan X

    That allows each broadcast domain a maximum of two networks to play in. The only issue with that is you require a router/L3 switch to make it happen.

    If you dont have a layer you will have to create a router on a stick network. how it will work, using my previous VLAN information;

    Pick a nice quick trunk port a router and create 3 sub interfaces. Each interface will have its own IP address which will be the default gateway of the VLAN.
    0/4.10 MGT VLAN 10.20.0.1
    0/4.20 MGT VLAN 10.20.2.1
    0/4.30 MGT VLAN 10.20.4.1
    Once that is done you will need to set it to a trunk port.
    switchport mode trunk
    then you will need to set it to dot1q so it will pass VLAN traffic
    switchport trunk encap(hit tab) dot1q
    Once that is complete you jump onto your core switch (the switch you connect to the router) and do the exact same trunk configuration on the port.
    switchport mode trunk
    switchport trunk encap dot1q

    Then, using the above example, allocate which port you wish to be in which VLAN.

    the next issue will you come into is DHCP. providing you have a nice DHCP server this is very very very easy. First step is to;
    create a new dhcp with the new subnet, lets pick 10.20.2.0 /23
    scope: 10.20.2.10 - 10.20.3.254. (i start at 10 incase we want reservations)
    DNS: current settings
    default gateway: 10.20.2.1 (this is the VLAN address you assigned the port.

    This will now issue DHCP requests to people who connect within that VLAN... but how does the switch know which VLAN to send it to and even which DHCP server?? this is a pretty important step, so please take note. On EACH vlan that connects to a DHCP pool you need to assign an IP Helper-Address to it. So the commands would be;
    conf t
    inter vl 20
    ip helper-address (YOUR CURRENT DHCP SERVER)
    exit
    wr mem


    this could be useless jibba, but after doing several networks with your issue it just screams broardcasts! D=
     
    Last edited: Dec 6, 2011
  13. evilasdeath

    evilasdeath Member

    Joined:
    Jul 24, 2004
    Messages:
    4,841
    If it an't broken don't fix it.

    Its pretty normal for switch ports to all flash at random intervals, given the rate at which packets can arrive and leave a switch the little light won't keep up.

    As long as they aren't blinking orange/green/orange/green which means collisions or errors your all fine.

    You would know if you had a loop, everything would run like a dog.
     
  14. Miff88

    Miff88 Member

    Joined:
    Nov 10, 2010
    Messages:
    427
    Location:
    Newcastle
    not ALL switch ports should be blinking at the same rate, only ones with known high data transfer rates.

    port colours can mean several things and not always errors. Cisco switches have a mode button on the front and ALOT of other brands only show duplex/speed with lights. gig = green & 10/100 = amber or full duplex = green & half = amber

    He said his network is slowing down, which means everything isn't all right and something needs fixing. just becuse you can still ping something with 1000ms doesn't mean it's working for some network/systems engineers. I would def be chucking wireshark onto an interface to see which traffic is going crazy.

    All multicast requests will hit every single port on your network. To elaberate, if port 1 on your first 24 port switch sends out a dhcp request it will hit every single port on that switch (same vlan). once it hits the port which connects to another switch (same vlan) it will hit every single port on that switch. once it hits the port that connects another switch (same vlan) it will hit every single port on that switch, rinse and repeat.

    so now we have that understood, imagen ALL protocols doing it and while still using the nework for other needs. kinda sucks.
     
    Last edited: Dec 6, 2011
  15. evilasdeath

    evilasdeath Member

    Joined:
    Jul 24, 2004
    Messages:
    4,841
    No he said everything is working fine otherwise.

    Broadcasts and multicast are a perfectly normal part of a switched network. Given that he has 6 switches. And he is running a flat network with 6 switches i'd take a gander to say he probably has ~100 hosts and servers and maybe even a router or 2, he might even be running OSPF between the routers over the same domain providing constant multi cast hellos.

    Most networks can be improved but unless your facing a WAN where the bandwidth is limited it doesn't really matter there are other things to worry about.

    Sure if you have the time to look into it and learn something go for it wireshark can be good fun. But i have seen networks in REALLY poor states and they still work because in a local switched LAN environment it takes a heck of a lot to drag performance down. Server performance is usually were things start to get bogged down.
     
  16. Lodion

    Lodion Member

    Joined:
    Dec 3, 2002
    Messages:
    18
    Location:
    Perth
    Multicast is not broadcast. In a correctly configured network, multicast traffic will only be sent to specific ports.

    DHCP requests are broadcasts (not multicast). The initial REQ from the client is broadcasted. The offer and ACK responses from the server are unicast... so do not get flooded out every port.

    As I posted earlier: traffic destined to MACs that a switch has not learnt will be flooded out all ports in that VLAN, with the exception of the source port. When the MAC is learnt, subsequent traffic is sent only to the correct port. This is unknown unicast traffic.

    Not sure why you're bringing protocols other than DHCP into this.... switch MAC address tables have no knowledge of the higher layers. Once you have a DHCP lease, if you send traffic to your DHCP server in the form of HTTP traffic, it will be switched normally and not flooded. When you send traffic (of any protocol) to a MAC the switch has not learnt, it will be flooded.
     
  17. evilasdeath

    evilasdeath Member

    Joined:
    Jul 24, 2004
    Messages:
    4,841
    In a correctly configured network, how many people do you know that understand multicast and configure and leave IGMP snooping enabled or dense/sparse mode. When most networks don't even care about it. Only those that use and are aware of multi-cast configure it correctly. Thats because the majority of multicast can be flooded like broadcast with no harm caused to your network.

    netbios, arp, routing protocols, all generate broadcast or multi-cast independent of what DHCP thinks.

    Some devices even do a gratuitous arp in order to check the address it has isn't used. Also broadcast.
     
  18. fileant

    fileant Member

    Joined:
    Feb 10, 2006
    Messages:
    567
    I tell my DHCP server to tell windows not to use NetBIOS

    dhcp-option=43,01:04:00:00:00:02

    Also if you give print servers (and all servers) a static IP and tell them to be static then it stops alot of poop.
    e.g. for dnsmasq you tell it to use and fill in "dnsmasq-hosts.conf" and then on each server set up the same IPs you assigned

    other handy options are
    bogus-priv
    filterwin2k
    dhcp-authoritative
    dhcp-option=vendor:MSFT,2,1i
     
    Last edited: Dec 7, 2011
  19. OP
    OP
    Piroteknik

    Piroteknik Member

    Joined:
    Jun 17, 2002
    Messages:
    379
    Location:
    Bat Country
    Ok guys,

    Thanks for all the suggestions - in the end it was the Cisco phone system causing the issues - Somehow it had been setup to broadcast the hold music to Everything, not just the Lan for the voip phones.

    I also noticed that someone mentioned that all lights blinking is not un common, and someone else agreed - this wasn't what made us believe there was an issue - this tipped us off that there MAY be an issue. This network had not been doing this in past. This only lead to investigation.

    Once we looked into it though - it was clear there was all magnitudes more data running arround the place than normal, and we were investigating this during the morning before staff arrived (This means approx 30 workstations not online)

    That was when we wondered what was going on.

    Cheers guys - at least i can sleep normal times now :)
     
  20. SpudBoy

    SpudBoy Member

    Joined:
    Jul 30, 2001
    Messages:
    5,281
    Location:
    Under the Bed.
    lol wut? :Paranoid:

    i'd love to the the reasoning behind that configuration.
     

Share This Page

Advertisement: