New internet provider ☺

Note: all this is my personal experience, on my personal machines, and I don’t claim it to be the absolute truth. Also, I don’t directly call out names, although if you live in Switzerland it’s pretty easy to guess who the old provider is from some hints.

For a long time, I wanted to move away from my current (well, now past) provider, for a multitude of reasons. The main being that the company is very classic company, with classic support system, that doesn’t work very well - I had troubles with their billing system that left me out cold without internet for 15 days, but for the recent few years, they were mostly OK, and changing to a different provider would have meant me routing a really long Ethernet cable around the apartment, so I kept postponing it. Yes, self-inflicted pain, I know.

Until the entire work-from-home thing, when the usually stable connection start degrading in a very linear fashion day-by-day (this is a graph that basically reflects download bandwidth):

At first, I didn’t realise this, as even 100Mbps is still fast enough. But once the connection went below (I believe) 50Mbps, it became visible in day to day work. And since I work daily from home… yeah. Not fun.

So I started doing speedtest.net - and oh my, ~20Mbps was a good result, usually 12-14Mbps. On a wired connection. On a connection that officially is supposed to be 600Mbps down. The upload speed was spot on, so I didn’t think it was my router, but:

• rebooted everything; yes, including the cable modem.
• turned off firewall. and on. and off. of course, no change.
• changed the Ethernet port used on my firewall.
• changed firewall rules.
• rebooted again.

Nothing helped. Once in a blue moon, speedtest would give me 100Mbps, but like once every two days, then it would be back and showing 8Mbps. Eight! It ended up as even apt update was tediously slow, and kernel downloads took ages…

The official instructions for dealing with bad internet are a joke:

• turn off everything.
• change cable modem from bridged mode to router mode - which is not how I want the modem to work, so the test is meaningless; also, how can router mode be faster⁈
• disconnect everything else than the test computer.
• download a program from somewhere (Windows/Mac/iOS, with a browser version too), and run it. yay for open platform!

And the best part: “If you are not satisfied with the results, read our internet optimisation guide. If you are still not happy, use our community forums or our social platforms.”

Given that it was a decline over 3-weeks, that I don’t know of any computer component that would degrade this steadily but not throw any other errors, and that my upload speed was all good, I assumed it’s the provider. Maybe I was wrong, but I wanted to do this anyway for a long while, so I went through the “find how to route cable, check if other provider socket is good, order service, etc.” dance, and less than a week later, I had the other connection.

Now, of course, bandwidth works as expected:

Both download and upload are fine (the graph above is just download). Latency is also much better, towards many parts of the internet that matter. But what is shocking is the difference in jitter to some external hosts I care about. On the previous provider, a funny thing was that both outgoing and incoming pings had both more jitter and packet loss when done directly (IPv4 to IPv4) than when done over a VPN. This doesn’t make sense, since VPN is just overhead over IPv4, but the graphs show it, and what I think happens is that a VPN flow is “cached” in the provider’s routers, whereas a simple ping packet not. But, the fact that there’s enough jitter for a ping to a not-very-far host doesn’t make me happy.

Examples, outgoing:

And incoming:

Both incoming and outgoing show this weirdness - more packet loss and more jitter over VPN. Again, this is not a problem in practice, or not much, but makes me wonder what other shenanigans happen behind the scenes. You can also see clearly when the “work from home” traffic entered the picture and started significantly degrading my connection, even over the magically “better” VPN connection.

Switching to this week’s view shows the (in my opinion) dramatic improvement in consistency of the connection:

No more packet loss, no more jitter. You can also see my VPN being temporarily down during provider switchover because my firewall was not quite correct for a moment.

And the last drill down, at high resolution, one day before and one day after switchover. Red is VPN, blue is plain IPv4, yellow is the missing IPv6 connection :)

Incoming old:

and new:

Outgoing old:

and new:

This is what I expect, ping-over-VPN should of course be slower than plain ping. Note that incoming and outgoing have slightly different consistency, but that is fine for me :) The endpoints doing the two tests are different, so this is expected. Reading the legend on the graphs for the incoming connection (similar story for outgoing):

• before/after plain latency: 24.1ms→18.8ms, ~5.3ms gained, ~22% lower.
• before/after packet loss: ~7.0% → 0.0%→, infinitely better :)
• before/after latency standard deviation: ~2.1ms → 0.0ms.
• before/after direct-vs-VPN difference: inconsistent → consistent 0.4ms faster for direct ping.

So… to my previous provider: it can be done better. Or at least, allow people easier ways to submit performance issue problems.

For me, the moral of the story is that I should have switched a couple of years ago, instead of being lazy. And that I’m curious to see how IPv6 traffic will differ, if at all :)

Take care, everyone! And thanks for looking at these many graphs :)