Network performance visualisation with Wireshark

Discussion in 'Networking, Telephony & Internet' started by Doc-of-FC, Apr 25, 2015.

  1. Doc-of-FC

    Doc-of-FC Member

    Joined:
    Aug 30, 2001
    Messages:
    3,347
    Location:
    Canberra
    As a network engineer, network performance is my bread and butter. So after experiencing slow download speeds from mirror.aarnet.edu.au I decided to dig a little deeper into the issue. I'll attempt to utilise tools and processes anyone can repeat to test their own internet links.

    Background:
    For a while now I've noticed the lovely aarnet mirror has been sub par in throughput performance when attempting to obtain updated binaries. This issue isn't with the AARNET mirror as parallel downloads from two Canberra ISP's reveal an 11x throughput difference between the service providers.

    Approach:
    I wont bore you with the fundamentals or quirks of TCP/IP or network improvements like FQ_codel, there are courses and plenty of online study material you can ingest to become more proficient in network engineering. What we want to do is investigate the 'feel' of a links performance, that snappy feeling when a page loads and why they may 'feel' slower.

    Let's start with a speedtest, TPG hosts a 15MB file on their website for link testing.

    So let's take a look at what exactly is happening when we download that file.

    You'll need to go and download Wireshark and install it.

    Next we'll need to pick a target file, for my first test I've used TPG's grass. now the server that hosts the file has an ip address, let's go get it so we can move on.
    Code:
    C:\>nslookup www.tpg.com.au
    Server:  xxxxxxx.xxxxxxx.xxx
    Address:  xx.xx.xx.xx
    
    Non-authoritative answer:
    Name:    www.tpg.com.au
    Addresses:  2407:8800:a000:1:216:3eff:fe67:9498
              2407:8800:a000:1:216:3eff:fe0c:3aca
              [color=red]203.26.27.38[/color]
    
    So when you open Wireshark, select your interface from the list in the capture pane, and click on capture options. Here you will need to put the ip of the server into the [Capture Filter] box, in the format of 'host 203.26.27.38' (not including '' and replacing the IP with the one of your choosing) and click start.

    Next go to your browser and do a non cached refresh of the url, at this point you may be thinking what does that mean, simply put let's download the file. This can be done by using CTRL +F5 in your browser for a large image or downloading a largish file from the server either using your browser or file downloader. Now at this point I have to warn you that multi part downloaders mask the effect of latency / QOS / congestion so for this test turn off those accelerators.

    Once your file is downloaded, let's go back to Wireshark and stop the capture (red square, fourth button along) From the Statistics menu we want to select Conversations and then the TCP tab. In this list you should see one entry for the test file you downloaded.

    Select the row then click on Graph B->A at this point you may need to move the graph over a little bit as it may appear over the graph control window. Within the graph control window you then want to go to the Graph Type tab and look at the Round-trip Time and Throughput graphs.

    A files throughput is directly related to the quality of the transmission path, the quality is a statistical representation of latency, packet loss and jitter; in a VOIP scenario this is represented as the MOS

    Now before we get to the pictures and shiny stuff, I need to mention QOS, for most people QOS means stop xyz person from killing my game performance or something of that ilk. Within a service providers network they'll use QOS to prioritise different kinds of traffic, mostly for stability but always for money as there's always someone willing to pay more than you.

    What this means is that there's the ability for an ISP to boost the priority of these famous speed checking servers above normal traffic, I'm not saying that this happens but it'd sure make an ISPs life easier when dealing with their helpdesk as your service is clearly fine and no issues exist, so it's in your head. [moral and ethical dilemmas aside]

    So, onto the pictures:
    [​IMG]
    [​IMG]

    The first image shows the throughput, alone this image shows nothing other than throughput distribution, it's what we'd call a baseline and from here we can now use it for comparison purposes against later attempts of the same test. Now a high performing network should draw a nice smooth line, though many things can interfere with this line, router CPU use, carrier network, inter carrier transmission path selection and distribution.

    The second image shows the consistency of the network, how quickly your requests are traversing it, TCP uses acknowledgement packets for confirmation purposes so the server knows it can send some more to you. Wireshark allows you to graph these values to get a visual representation of what's truly happening.

    Last night about 10PM I tested AARNET:
    [​IMG]
    [​IMG]

    Now what you draw from this information is directly related to you, your ISP and the network which may be outside of your ISP's control. These tests need to be done in controlled circumstances, other use of your service can skew results and cause misleading information.

    So how do we piece this all together, again this data is only a point in time snapshot of performance by itself, you need to build lots of data and piece it together to develop a trend. You need to poll multiple servers along the chain of routers and switches with the network, this is not always doable. you can use other tests like DNS server response times and plot this, although these requests may also be prioritised.

    For example: I've got a simple loop that grabs the same file from aarnet and stores it in the same place, nothing fancy and truthfully a small use of bandwidth to show a point.
    Code:
    speedtest@boxen:~# cat wgetter
    #!/bin/bash
    for i in {1..10000000}
    do
       wget -q http://mirror.aarnet.edu.au/pub/eclipse/mat/1.4/rcp/MemoryAnalyzer-1.4.0.20140604-linux.gtk.x86_64.zip -O ./mirror.aarnet.edu.au.speedtest
    done
    
    Coupled with PFSense rrdtool, this script allows me to show this:
    [​IMG]
    [​IMG]

    The maximum achievable throughput rate for a single TCP session using HTTP against one of Australia's largest and most robust file mirrors, from this it's apparent that the network path is under contention as my peak throughput varies greatly.

    For further learning, a great talk on wifi, TCP, bufferbloat and FQ_codel by the amazing Dave Taht:


    Props to AARNET for a great mirror and Wireshark for a great tool.
     
  2. evilasdeath

    evilasdeath Member

    Joined:
    Jul 24, 2004
    Messages:
    4,861
    lots of reads no comments,

    Network queuing is such a difficult topic, and peoples understandings on what the internet actually looks like on the inside are even fewer.

    It's interesting there is problems with aarnet from TPG, since aarnet is peer/hosted from within TPG.

    I know some providers have problems with aarnet because TPG peers with PIPE in QLD, and many of the providers out of pipe QLD only have 100mbit peers. Yes they still exist, and the havoc it causes to TCP when traffic goes small link big link small link across a path does cause problems.

    I've been looking into fq_codel myself i just need to find myself a router that i'm happy to brick or can recover to do testing on the different wrt solutions or go down the pfsense path haven't decided.

    But this plague of people that use speedtest as a testing method for there internet service is half the problem as well, it's a synthetic test to a synthetic location using a browser implemented method, ookla speedtest uses flash which can change between browser or flash version, all people care about is seeing a high number.
     
  3. Great_Guru

    Great_Guru Member

    Joined:
    Sep 5, 2001
    Messages:
    1,225
    Location:
    Australia
    Thanks for your post Doc-of-FC. I've seen you post a little in recent times and it's good to see. I've always liked what you bring to OCAU (confessed OCAU lurker)

    When is OCAU going to implement an "upvote/like" button for topics/posts. This kind of real world troubleshooting is what juniors need to soak up from experienced people in IT.

    And by junior I mean anyone who isn't a pro in the field being posted.

    Post as many wireshark tutorials are you like. It is such an under utilised tool.

    Speedtest *cringe*
     
  4. blankpaper

    blankpaper Member

    Joined:
    Feb 1, 2013
    Messages:
    1,013
    Nice post. Need more quality posts like this in this section.
     
  5. dcyloo

    dcyloo Member

    Joined:
    Apr 7, 2002
    Messages:
    5,062
    Location:
    Brisbane
    Thanks very much for this - I'll give it a go on my network and see what the reported, actual and QOS'ed Speedtest.net results end up comparing to be.
     
  6. cacaw

    cacaw Member

    Joined:
    Aug 22, 2010
    Messages:
    46
    Location:
    Australia
    Realistically, it is a bit rich to compare results from speedtest.net and real world results.

    Often times, the speedtest.net server resides within the same autonomous system as your service provider, and in essence, not truly representing the actual performance outside of your ISP.
     
    Last edited: Jun 6, 2015
  7. evilasdeath

    evilasdeath Member

    Joined:
    Jul 24, 2004
    Messages:
    4,861
    Speedtest.net has it's place because everyone loves looking at the numbers and it's simple.

    unfortunately due to the way the test is run and the way TCP works it is also misleading. Speedtest only represents the path between you and the speedtest server under the given test method. It does not mean you will get that performance to other sites in the area of that server.

    (you can also add to the fact that the browser you use can impact how speedtest performs)

    The way TCP works is actually a bit of a problem on the internet honestly, it works, SDN is slowly working towards a scenario where the application and the network communicate better, but that's a bit of a way off for LANs and a longer way off for the internet.

    The fact that TCP basically kills itself instead of running at a rate the network supports is a pain. TCP tries to use anything that's available and then when it drops 1 packet it shoots itself in the foot and then tries again.

    But hey it works, and it's the best we have at the moment.
     
  8. fR33z3

    fR33z3 Member

    Joined:
    Jul 16, 2001
    Messages:
    2,164
    Location:
    Perth
    great post doc. Always interesting reading another NE's approach to troubleshooting such problems.
     

Share This Page

Advertisement: