Internet performance (warning: somewhat geeky)

[Tokyo Tokei]

I thought I'd share my recent adventures down the rabbit hole of internet “performance”. As many work from home to some extent, it seems a good area.

Plus who doesn’t appreciate faster loading of the Friday Thread? Exactly.

Background: I haven’t changed my ISP for 15+ years as my house is limited to an ancient ADSL line, of max advertised capacity 160Mbps (down) and 16Mbps (up). But as I can't get fibre without seemingly funding it for the whole street, it is what it is and I have to make the most of it. I also still use my old Apple Airport (802.11ac, aka Wi-Fi 5) routers, which are super stable but not the latest and greatest, so bear that in mind.

This is far from a “state of the art” demo. But it may be more representative because of that.

I begin with my usual go-to, and many others it seems, Speedtest.net:



Well, not too bad! Quite close to the max advertised download speed, and the upload speed is fine - I rarely see it in the 10Mbps or over range. This is using wi-fi, from the iPad I am typing this on, and others in the house are online.

But…!

This hides some very pertinent things. It is based off a large data transfer from one site. Whereas typical household usage has many devices, all accessing the internet at various speeds. App updates, cloud photo syncing, game downloads, DNS requests, NTP updates, Teams meetings, Netflix, important threads about buns, and so on.

Regardless of download speed, some of these are latency sensitive. Such as voice and video services, and gaming.

If we look at a test that measures latency changes during high network activity, the picture is not rosy at all:


(From http://www.dslreports.com/speedtest)

Now bear in mind this is ADSL which has a degree of variability in latency anyway, but look at that middle “lag” bar: over 4 seconds of latency for some packets! Many are around 1 second or more.

That’s in just a one minute download test. It would be similar to when an app, OS or game updates, cloud files/photos synchronize, or an HD movie is streaming. Anything latency sensitive such as Teams/Webex/FaceTime/Gaming could potentially be impacted.

So…

I add some Quality of Service (QoS) to my router. This a huge subject. Hence the rabbit hole. There are many approaches and options. I choose a simple one: apply “fq-codel” to the bridge between my LAN and the internet.

This is a traffic shaper that attempts to fairly arbitrate multiple packet flows through an interface. So big transfers that may saturate the internet bandwidth are gently slowed down, and small flows are prioritized. Without QoS, any large transfer meeting the relative bottleneck of the ISP connection may buffer regardless of priority, and “hold up” packets from other sources.

I set it to shape to 100Mbps down, 10Mbps up (with a temporary “burst” allowable of up to 160Mbps down and 16Mbps up)

The result:



Note that the vertical scale on the chart has lost a couple of zeroes. Worst case latency is now under 60ms, and most under 30ms! Anything under 100ms should be unnoticeable on a live stream or call. For gaming, the lower the better.

A drastic improvement. Compared to Speedtest.net, this is telling me better information. Overall real world speed actually increased too, despite setting the QoS limits to less than my notional bandwidth.

Caveats include that this is over wifi infra, to an ADSL connection that is intrinsically variable latency, and my family is using the internet too. Results may vary. But it is representative of how our household uses the internet. I did run all the tests above on the same device (my old iPad, the one that that I'm typing this on) in the same location, within minutes of each other, with no changes other than enabling the QoS.

I think it’s a reasonable demo of “download speed isn’t everything” when it comes to network performance, at least as perceived by application response. I used my Mikrotik firewall/router to set this up, but some modern Wi Fi routers have QoS available as an option, which on this evidence suggests it would be well worth trying. But it’s a very deep rabbit hole indeed!

[Captain Morgan]

I think it’s a reasonable demo of “download speed isn’t everything” when it comes to network performance, at least as perceived by application response. I used my Mikrotik firewall/router to set this up, but some modern Wi Fi routers have QoS available as an option, which on this evidence suggests it would be well worth trying. But it’s a very deep rabbit hole indeed!

Interesting, a different path lead me to build a pfsense fw/router to give me the ability to use vlans and stateful fw rules to isolate IoT devices from trusted ones and to centralise the management of VPN’s and content filters.

I like you am limited to a adsl 2 line but at ~60/10 but generally it never feels constrained so I’m not sure if this is a rabbit hole I need to investigate or not.

Have you perceived any difference since you made these changes or is this log data evidenced only?

[Alansmithee]

I have 1000/100 and use a mesh network (orbi) but the best thing I did was simply to hardwire the office. One of the satellites is in there and that is hardwired and the desktops connected to it.

Otherwise - where things have Wifi6 they take advantage of that or simply don't need the speed for what they do...

[Tokyo Tokei]

Have you perceived any difference since you made these changes or is this log data evidenced only?

I also use the Mikrotik router to isolate IoT devices and provide Wireguard VPN. It's a very capable little device (RB750Gr3 or “hEX”).

For the internet performance, my total bandwidth is low by modern standards, but actually entirely fine for my household use:



I barely break 60Mbps, which is usually Netflix filling its cache for a few seconds, and generally usage averages much less. Anything that can be wired is wired, but that doesn’t help at all with buffer bloat. As mentioned, it’s not about bandwidth, but latency.

Other spikes are me playing with network performance tools.

That's with a house of 4 people and maybe 35 connected devices.

It was just the occasional audio or video hiccup on Teams calls or FaceTime. Could be when others are watching films or playing games, or something pulling down a software update. Or the worldwide internet collapse as people try to login to Timefactors at 2pm on a Sunday.

Too early to tell if that's fixed now I have applied QoS, but we shall see. There are so many variables, including of course what is the situation “on the other side”. I can only control my side, and needed some measurements to give me a view on whether I am tuning in the right direction.

Edit: The http://www.dslreports.com/speedtest site maintains a history of measurements (assuming you access it from the same device) so it is quite useful.

[Captain Morgan]

I looked at Mikrotik as an option but found the documentation and guides to be a degree harder to penetrate compared to pfsense

It does still amaze me the ability of some of the low cost fw/routers these days compared to generic domestic units.


I use the thinkbroadband to monitor inbound latency looks like there were network issues yesterday.

[Tokyo Tokei]

I looked at Mikrotik as an option but found the documentation and guides to be a degree harder to penetrate compared to pfsense

It does still amaze me the ability of some of the low cost fw/routers these days compared to generic domestic units.

I use the thinkbroadband to monitor inbound latency looks like there were network issues yesterday.

Mikrotik isn’t really for home users, but if you are familiar with pfsense it isn’t too hard. (I tried pfsense but was put off by the company behind it, but that’s a different story)

The latency graph above is useful but doesn’t say if the ping is under network load conditions. That’s when buffering causes the massive delays, in my case, from 30-60ms to ~4 seconds. That’s when interactive sessions, gaming, video or audio, suffer.

The DSLreports test is an edge case of course, deliberately loading the network while testing latency, but if you have more than one user accessing your internet gateway, it’s not unlikely. I only care now that I work at home a lot, and video/audio issues are more than a momentary annoyance.

[reecie]

I also use fast.com to check speeds in addition to Speedtest. If nothing else it is probably more representative for video streaming given Netflix provide it.

[scooobydont]

Here are couple ISP's ask us to use when testing leased lines:

https://speedtest.samknows.com/
https://www.nperf.com/en/

If there are issues, it's iperf testing they want after that.

[Tokyo Tokei]

If you want to test latency under load (as opposed to just bandwidth, which isn’t the same) I recommend:

http://www.dslreports.com/speedtest as in the original post
https://www.waveform.com/tools/bufferbloat (similar, uses http rather than websockets)
networkQuality (For Mac users, just type it in a Terminal window)

Well worth checking your router, if it is a modern one, to see what QoS settings it provides. Bandwidth != performance when it comes to interactive applications such as voice, video or gaming.

[Tokyo Tokei]

More obsessing and a bit of tuning later…

No QoS… (click either image for latency detail). Up to 3.5 seconds packet latency during download, scoring a miserable F Grade for buffer bloat:



QoS switched on (fq-codel)… latency under load down to 110ms, worst case, scoring a splendid A/A+…



Rabbit hole: http://www.dslreports.com/speedtest