Internet Upload Speed

[dennisgorelik]

About one week ago we noticed that upload from my computer to remote servers became quite slow.
Speed test indicates that while upload speed is good (80 - 90 Mbps), download speed is getting worse on remote servers.

My computer is in Jacksonville area and upload speed to Orlando servers is quite fast:
http://www.speedtest.net/my-result/5837858780
Orlando, FL
Ping: 21 ms
Download speed: 90.13 Mbps
Upload speed: 9.55 Mbps

That is close to 10 Mbps upload speed that Comcast claims to have for my account.

However the further away the server is - the longer is ping time and the worse is upload speed:
http://www.speedtest.net/my-result/5837865645
Dallas, TX
Ping: 47 ms
Download speed: 90.12 Mbps
Upload speed: 5.14 Mbps

http://www.speedtest.net/my-result/5837843259
Remote server: Seattle, WA
Ping: 83 ms
Download speed: 90.13 Mbps
Upload speed: 3.12 Mbps

http://www.speedtest.net/my-result/5837868925
London, UK
Ping: 115 ms
Download speed: 87.99 Mbps
Upload speed: 2.31 Mbps

http://www.speedtest.net/my-result/5837874972
Chelyabinsk, Russia
Ping: 188 ms
Download speed: 86.41 Mbps
Upload speed: 1.52 Mbps


Why does upload speed deteriorate proportionally to the ping to the remote server (but download speed stays the same)?
My hypothesis is that network protocol waits after every chunk of data for the response from the remote server (instead of continuing sending new chunks of data).
By why?
Comcast technician came today and found that there was some network issues with upload frequency. Lost packets during upload may initiate resynchronization waits that would drag upload speed down on connections with slower ping.

Comcast technician promised that within couple of days, Comcast network crew should fix the issues.
We'll see how that would change my upload speed.

Update (thanks to anspa recommendation):
PingPlotter shows a lot of Packet Loss with my Internet connection:

Comments

[yatur.livejournal.com]

> My hypothesis is that network protocol waits after every chunk of data for the response from the remote server

Well, it depends on how exactly they measure upload speed. Regular TCP protocol was specifically designed NOT to wait for acknowledge before transmitting the next chunk (google 'TCP Window').

[dennisgorelik]

Our own attempts to upload large file show similar speed results.

What does speedtest.net shows on your computer?
Does the upload speed depend on the server (ping) that you use for testing upload?

> Regular TCP protocol was specifically designed NOT to wait for acknowledge

Exactly.
That's why I think it is either Comcast wide artificial limitation introduced into their network protocols, or these delays are caused by lost packets in the network cable frequency that carries upload data.

[sab123.livejournal.com]

Possibly they have a packet propagation delay that overwhelms the TCP window size. For the local connections it might be just below the window size, but then when mode network delay is added on top it overwhelms the window size. You might be able to tune it in the OS settings to test this hypothesis (or maybe just run tcpdump and see).

[dennisgorelik]

1) How do I run tcpdump?

2) I actually tried to add DefaultReceiveWindow and DefaultSendWindow to registry:
http://www.miersengineering.com/2015/03/480/

It increased upload speed a little for connections with "nearby" servers.
But upload speed to remote servers is still slow.

[sab123.livejournal.com]

1) step 1: install Linux or BSD :-) May also need the developer packages, nowadays they try to hide all the good stuff :-( Then run "man tcpdump" for help.

On Windows the similar thing is called "Network Monitor" if I remember right, might be downloadable from MSDN. Or some 3rd-party tool like Wireshark.

Basically look at the dump of the TCP packets and see if the send window gets all used up. You'll also see the retransmits if the packets get lost. A lost packet resets the window size for this connection down to 0 if I remember right. but the modern TCP implementations might be smarter. Linux actually has the pluggable modules for retransmit policies and you can tune them in a very wide range.

2) The send window is the one you need for upload. I'm not sure about the exact settings but it might be also limited by the configuration of the network card driver. Try looking in the NIC device properties.

[dennisgorelik]

1) anspa recommended PingPlotter.
I updated this posting with the graph from PingPlotter.
Is that kind of data that you are suggesting to extract using tcpdump/Network Monitor?

2) What "NIC device" do you mean? Cable modem?

[anspa.livejournal.com]

Anspa also recommends: go to dslreports.com and order a bunch of free tests against your external IP, like going to Tools and choosing Line quality test. That would show you how hosts from around the globe TCPing into your cable modem. It may suggest the Windows parameter adjustments etc. But I think your problem is a temporary backbone router that is at fault here and it should be addressed automagically within 1-2 days as Comcast suggested.

[sab123.livejournal.com]

1) No, a different kind of data. Basically, you can get the actual window size used for sending on each packet (and also see the round-trip time between the packet and the ACK for it), and the packet retransmits that cause the window to collapse. Well, you'll only see the window size advertised from the receiving side directly but you can compute the send size by subtracting the last ack's sequence from the last data's sequence.

BTW, the easy way to see the retransmit is to use the plain old command-line FTP with the hash printing enabled (it will print "#" for each block of data sent). You'll see these hashes printed fast when the data flows nicely, and hiccuping when a packet gets lost.

But PingPlotter's data is useful too. Traceroute ("tracert" on Windows) would probably give the approximately same information. It looks like Comcast has some kind of issues where their network connects to another provider, with the delays and packet loss.

2) The network card in your computer. BTW, the other side also has its settings for the maximal windows sizes, and the smallest of the two gets actually used. So your increasing the setting on your side might not help at all if the other side already has the lower limits.

[anspa.livejournal.com]

Am truly sorry, in my earlier comments (now deleted) I mixed up tcpdump and Dr.TCP tool which dslreports provides with its Tweak test to adjust TCP/IP parameters in Windows registry. Tcpdump is a command line tool that would dump packets. For Windows we use Wireshark for that purpose, not really tcpdump, Wireshark is much more convenient for Windows, though any dump would be irrelevant for the problem you have right now. Forgive me, Linux people here. =)

[lamantyn.livejournal.com]

This Spring I had a similar issue with upload speed with Verizon FIOS.
Tech was puzzled, theorized about issues with some equipment on Verizon side.

At some point I tried another cable socket (on another wall) and magically everything became OK. Seems a bad coaxial splitter.

[dennisgorelik]

So the hypothesis, explaining the issue is:
1) Bad coaxial splitter caused multiple data packets loss.
2) Multiple data packets loss caused smaller size of the congestion window.
3) Smaller size of the congestion window caused slower upload speed.

Does this hypothesis sound about right?

Hopefully I have similar issue with the network between cable box on my front yard and Comcast modems (connection between my home and Internet box on my frontyard seems to be good, according to the Comcast tech).
If it is the case, when they would fix their network issues I would get better upload speed.

[anspa.livejournal.com]

Install PingPlotter. That will show you exactly what network node is at fault, also drawing a nice graph of pings.. You would see lost packets in real time.

PingPlotter

[dennisgorelik]

PingPlotter shows a lot of Packet Loss with my Internet connection:


Do I read it correctly that about 30% of packets are lost?

Re: PingPlotter

[anspa.livejournal.com]

Yup, also you see that hopes #8 and #11 are suspicious and you may blame them as your root trouble cause. Share this with Comcast techs. You may also try to stress-test it by changing parameters to send 100 packets per 1 second. That would show even more packet loss (or outline a specific hop that limits your bandwidth). So far it looks like it is not even Comcast infrastructure that is at fault here. It is one of the internet backbone providers. I've never seen networklayer.com before, but it looks as Layer3 kind of player.

Re: PingPlotter

[dennisgorelik]

Hop #8 is "as36351-1-c.nota.fl.ibone.comcast.net"
I agree that it looks suspicious and is probably the blame.

#11 is "po1.fcr04.sr05.dal01.networklayer.com" which is a new name for softlayer (my guess is that IBM finally digested SoftLayer and renamed softlayer.com domain to networklayer.com).
I do not think that hosting provider of my remote server is to blame here:
networklayer.com is getting packet loss simply because it is behind malfunctioning "as36351-1-c.nota.fl.ibone.comcast.net" hop (of my ISP).

Hops at the end of the network chain must work through the hops at the beginning of the chain, right?
So TCP exchange with remote hops collects all errors from nearby hops too.