Unacceptable Packet Loss and Latency Issues (San Francisco)

[sethdhalpern]

Hello,

I'm using 1Gb fiber in the Castro neighborhood of San Francisco. I've been dealing with latency and rubber banding in-game issues that seem to relate to Sonic's infrastructure (traceroute below). This is affecting all of my devices (Mac/pc/xbox). My xbox warns me about "unstable packet loss" regularly, and I am frequently booted from PVP games on the Xbox like Halo. My iPad multiplayer games are also affected, with rubber banding and latency spikes to 250+ms in-game. The trace route has three lines that are problematic. I have bolded them below. This definitely seems like an issue with Sonic's physical infrastructure or server software. What can be done to stop this issue from reoccurring? Has anyone else experienced this issue, or identified problems with the following Sonic subdomain/services?

0.ae1.cr4.colaca01.sonic.net
0.ae1.cr6.rcmdca11.sonic.net
0.ae1.cr4.lsatca11.sonic.net

traceroute to riotgames.com (23.203.211.122), 64 hops max, 52 byte packets
1 192.168.1.1 (192.168.1.1) 4.344 ms 3.768 ms 2.303 ms
2 lo0.bras1.snfcca05.sonic.net (50.0.79.115) 3.587 ms 3.224 ms 3.080 ms
3 300.irb.cr1.snfcca05.sonic.net (198.27.186.138) 4.736 ms 11.066 ms 8.207 ms
4 0.ae14.cr2.colaca01.sonic.net (198.27.244.57) 22.835 ms 96.174 ms 89.981 ms
5 0.ae2.cr1.snrfca01.sonic.net (157.131.209.169) 11.726 ms
0.ae1.cr1.colaca01.sonic.net (157.131.209.65) 9.752 ms
0.ae2.cr1.snrfca01.sonic.net (157.131.209.169) 11.360 ms
6 * 0.ae1.cr4.colaca01.sonic.net (157.131.243.30) 4326.400 ms
0.ae1.cr1.rcmdca11.sonic.net (157.131.209.137) 14.024 ms
7 0.ae1.cr6.rcmdca11.sonic.net (23.93.54.154) 3776.239 ms * *
8 0.ae1.cr4.lsatca11.sonic.net (135.180.179.194) 4468.302 ms 4394.620 ms 4404.787 ms
9 * * *
10 100.ae1.nrd1.equinix-sj.sonic.net (75.101.33.185) 6.983 ms 5.084 ms 5.327 ms
11 * akamai.0.ae4.nrd1.equinix-sj.sonic.net (157.131.2.38) 7.784 ms *
12 * ae6.r01.border.sjc01.sdn.netarch.akamai.com (23.203.158.17) 10.786 ms *
13 * * *
14 akamai.0.ae4.nrd1.equinix-sj.sonic.net (157.131.2.38) 8.181 ms 5.619 ms 5.483 ms
15 ae6.r01.border.sjc01.sdn.netarch.akamai.com (23.203.158.17) 11.097 ms * 10.783 ms
16 * * *
17 * * *
18 * * *

[sethdhalpern]

Here is a follow-up trace route to a different server, with problematic node-specific results bolded/underlined:

traceroute to apple.com (17.253.144.10), 64 hops max, 52 byte packets
1 192.168.1.1 (192.168.1.1) 2.943 ms 2.105 ms 1.782 ms
2 lo0.bras1.snfcca05.sonic.net (50.0.79.115) 3.885 ms 3.615 ms 6.570 ms
3 300.irb.cr1.snfcca05.sonic.net (198.27.186.138) 36.728 ms
301.irb.cr2.snfcca05.sonic.net (198.27.186.210) 6.351 ms
300.irb.cr1.snfcca05.sonic.net (198.27.186.138) 88.047 ms
4 0.ae14.cr2.colaca01.sonic.net (198.27.244.57) 421.668 ms 19.473 ms
0.ae14.cr2.snrfca01.sonic.net (198.27.244.49) 33.627 ms
5 0.ae2.cr1.snrfca01.sonic.net (157.131.209.169) 308.015 ms
0.ae1.cr1.colaca01.sonic.net (157.131.209.65) 46.812 ms
0.ae2.cr1.snrfca01.sonic.net (157.131.209.169) 583.238 ms
6 0.ae1.cr4.colaca01.sonic.net (157.131.243.30) 4326.505 ms 4158.516 ms
0.ae1.cr1.rcmdca11.sonic.net (157.131.209.137) 39.701 ms
7 * * *
8 * 0.ae1.cr4.lsatca11.sonic.net (135.180.179.194) 4456.909 ms *
9 * * 0.ae0.cr4.snjsca11.sonic.net (157.131.209.149) 4330.691 ms
10 100.ae1.nrd1.equinix-sj.sonic.net (75.101.33.185) 6.153 ms * 8.183 ms
11 * apple.0.ae7.nrd1.equinix-sj.sonic.net (157.131.2.46) 40.983 ms 20.924 ms
12 apple.it (17.253.144.10) 5.765 ms !Z * 8.385 ms !Z

[joeyyung911]

I got the same result. For now, I would suggest using a different DNS server, that seems to avoid the problem.

[msiegen]

FYI, high latency/loss on the intermediate hops of a traceroute are normal and not indicative of a problem. This happens because responding to a trace requires greater resources than forwarding a packet, so routers often deprioritize it. The results are only valid if all subsequent hops also exhibit the same (or greater) loss.

It also looks like your first trace never reached as far as riotgames.com. That might indicate a problem, but I've also seen it happen with the traceroute built in to many OSes. If you can run MTR that will give more confidence.

From Santa Rosa, mtr shows the following. The final destination looks healthy, so I'm clearly not able to reproduce the experience that you're getting. Do we know if riotgames.com is the correct host to trace though? Often a company's web site will be on a different network than their game servers, so we should try to identify and trace the latter.

Code: Select all

# mtr -wbzc 100 23.203.211.122
Start: 2023-11-22T18:31:47-0800
HOST: wv2                                                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    10.31.16.81                                                             0.0%   100    0.3   0.3   0.1   0.5   0.1
  2. AS7065   lo0.bng1.snrsca01.sonic.net (50.1.36.48)                                0.0%   100    0.9   1.9   0.7  39.2   3.8
  3. AS46375  301.irb.cr2.snrsca01.sonic.net (192.184.184.98)                         0.0%   100    7.3   6.0   1.2  35.8   3.8
  4. AS46375  0.ae2.cr8.snrsca01.sonic.net (192.184.185.165)                          0.0%   100    9.3   6.1   1.0  30.4   3.6
  5. AS46375  0.ae0.cr7.snrsca01.sonic.net (198.27.244.113)                           0.0%   100    4.9   6.3   1.8  19.1   3.0
  6. AS46375  0.ae11.cr5.snrsca01.sonic.net (192.184.185.154)                         0.0%   100   12.4   9.5   1.2 108.9  14.3
  7. AS46375  0.ae0.cr1.ptlmca01.sonic.net (142.254.59.42)                            0.0%   100    6.4   8.2   1.9  84.5   9.9
  8. AS46375  0.ae0.cr2.ptlmca01.sonic.net (192.184.185.162)                          0.0%   100   10.2   7.5   2.3  54.9   7.9
  9. AS46375  0.ae1.cr3.snrfca01.sonic.net (23.93.52.254)                             0.0%   100   55.1  10.7   2.9  73.2  12.8
 10. AS46375  0.ae2.cr4.snrfca01.sonic.net (157.131.242.229)                          0.0%   100    3.6   8.1   2.2  44.4   5.4
 11. AS46375  0.ae4.cr1.snrfca01.sonic.net (135.180.179.197)                          0.0%   100    9.0  25.1   2.2 136.5  22.8
 12. AS46375  0.ae1.cr1.rcmdca11.sonic.net (157.131.209.137)                          0.0%   100   35.7  63.0   6.1 250.7  58.4
 13. AS46375  0.ae1.cr6.rcmdca11.sonic.net (23.93.54.154)                             3.0%   100  3417. 2989.  78.4 6772. 1758.6
 14. AS???    ???                                                                    100.0   100    0.0   0.0   0.0   0.0   0.0
 15. AS46375  0.ae0.cr1.equinix-sj.sonic.net (75.101.36.253)                         69.0%   100  8908. 8885. 8516. 9295. 177.0
 16. AS46375  100.ae1.nrd1.equinix-sj.sonic.net (75.101.33.185)                       0.0%   100    4.0   8.6   3.0  72.6  12.4
 17. AS7065   akamai.0.ae4.nrd1.equinix-sj.sonic.net (157.131.2.38)                   0.0%   100    4.2   3.9   3.0   4.4   0.2
 18. AS20940  ae6.r01.border.sjc01.sdn.netarch.akamai.com (23.203.158.17)             0.0%   100   52.0  63.7   4.6 432.6  82.3
 19. AS???    ???                                                                    100.0   100    0.0   0.0   0.0   0.0   0.0
 20. AS???    ???                                                                    100.0   100    0.0   0.0   0.0   0.0   0.0
 21. AS16625  a23-203-211-122.deploy.static.akamaitechnologies.com (23.203.211.122)   0.0%   100    4.2   4.2   3.4   4.7   0.3

[sethdhalpern]

Thanks for the reply. I will try again using the program you linked. The main issue: periods of latency and rubber banding, and disconnects, during online gaming- across multiple client devices incl. Xbox plugged into ethernet, iPad etc. The generic speed/ping test results are fine, except I’ve noticed packet loss in the 6% - 7% range with some regularity, and I suspect it’s the cause of my real-time online gaming woes acrosss my multiple devices.