Page 1 of 2

odd latency issue

Posted: Mon May 24, 2021 7:53 pm
by rebel316
For the past few days in the evening, I've been seeing unusual latency spikes. I am almost positive it's not anything on my end as I have observed latency within a couple of games I've been playing as well as livestreams. For instance, one of the games I play, every time I switch/move to a new area, the game will put me into a new server/connect me to a different IP. The ping on these vary, but usually when there is no issues I am at around 7-10ms. I am also running Net Uptime Monitor which constantly pings a couple of DNS servers and this is what it showed last night when it happened for a few hours:

https://imgur.com/a/tGz2a9N

https://imgur.com/a/VFpO7Gk

As I'm writing this post now, the latency has started creeping up again this evening.

The latency as you can see is way higher than usual (when there isn't this issue, everything is around 3-5 ms). Sometimes it takes longer than 5 seconds for a ping response, is why it logs all the failures in the 2nd screenshot. This doesn't disrupt my service/disconnect me, but I do notice lag and my speedtest results slow down drastically. I usually get about 930 down and 930 up, but at it's worst I've seen it result in 6 down 20 up.

This doesn't interrupt my service, but I would like to know what's happening.

Re: odd latency issue

Posted: Mon May 24, 2021 8:58 pm
by joeyyung911
Have you tried connecting the computer directly to the ONT, to isolate the router out of the equation?

Re: odd latency issue

Posted: Mon May 24, 2021 9:38 pm
by rebel316
I have not done that, but the strangest thing is it resolves itself after a short while and is only apparent during the evening. I'm not sure how the router would cause this issue at what seems to be a rather consistent time frame.

I have also power cycled yesterday to see if that would fix the problem and factory reset the router not too long ago as well.

Re: odd latency issue

Posted: Mon May 24, 2021 10:44 pm
by rebel316
If it also helps, I ran a tracert and here's the output:

Tracing route to dns.google [8.8.8.8]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms dsldevice.attlocal.net [192.168.1.254]
2 1 ms 1 ms 1 ms 107-199-200-1.lightspeed.sntcca.sbcglobal.net [107.199.200.1]
3 2 ms 2 ms 1 ms 71.148.135.252
4 82 ms 79 ms 69 ms 12.122.149.186
5 70 ms 73 ms 82 ms sffca22crs.ip.att.net [12.122.3.70]
6 74 ms 75 ms 89 ms 12.122.163.61
7 76 ms 71 ms 71 ms 12.255.10.234
8 65 ms 61 ms 60 ms 108.170.230.54
9 82 ms 73 ms 72 ms 72.14.235.3
10 77 ms 81 ms 67 ms dns.google [8.8.8.8]

I'm not too sure how to read this, but the 4th hop appears to be where the problem starts.

Here's another tracert to a server IP from one of the game servers I'm on, which I'm experiencing no noticeable increase in latency on, again it looks like hop #4 has some sort of problem:

Tracing route to 64.90.2ca9.ip4.static.sl-reverse.com [169.44.144.100]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms dsldevice.attlocal.net [192.168.1.254]
2 1 ms 1 ms 1 ms 107-199-200-1.lightspeed.sntcca.sbcglobal.net [107.199.200.1]
3 1 ms 1 ms 1 ms 71.148.135.252
4 54 ms 71 ms 81 ms 12.122.149.186
5 30 ms 2 ms 2 ms 12.122.149.57
6 4 ms 3 ms 9 ms 12.252.87.30
7 * * * Request timed out.
8 4 ms 4 ms 4 ms e5.11.6132.ip4.static.sl-reverse.com [50.97.17.229]
9 4 ms 4 ms 4 ms po1.fcr02a.sjc03.networklayer.com [169.45.118.153]
10 4 ms 4 ms 4 ms 64.90.2ca9.ip4.static.sl-reverse.com [169.44.144.100]

and it just resolved and everything is back down to normal ms as of about 11 PM

Re: odd latency issue

Posted: Tue May 25, 2021 10:25 am
by briancw
Hi rebel316,

That tracert is very helpful. So basically from what it looks like is there was some issue with AT&T's peering network and they appear to have issues with how they were routing traffic. This can sometimes happen and it appears they identified and fixed the issue. The details of what happened are mostly obscured to us as the effected servers are all inside AT&T's network but it seems like a pretty standard routing issue happened here.

Best wishes,

Re: odd latency issue

Posted: Tue May 25, 2021 10:34 am
by dane
Everybody with an interest in network performance diagnosis should read this:

https://archive.nanog.org/meetings/nano ... 47_Sun.pdf

Re: odd latency issue

Posted: Tue May 25, 2021 12:10 pm
by ngufra
dane wrote:Everybody with an interest in network performance diagnosis should read this:

https://archive.nanog.org/meetings/nano ... 47_Sun.pdf
Nicely presented.

It looks like a slide deck for sharkfest!

Re: odd latency issue

Posted: Tue May 25, 2021 8:00 pm
by rebel316
Hi again, this is the 4th evening in a row this is happening. Latency creeping back up to the 50-90s. Tracert's as follows, one to google, and one to a server in a game which I know is not lagging since I'm connected to it at 7ms :

Tracing route to dns.google [8.8.8.8]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms dsldevice.attlocal.net [192.168.1.254]
2 1 ms 1 ms 1 ms 107-199-200-1.lightspeed.sntcca.sbcglobal.net [107.199.200.1]
3 2 ms 1 ms 1 ms 71.148.135.252
4 98 ms 115 ms 90 ms 12.122.149.186
5 95 ms 99 ms 121 ms sffca22crs.ip.att.net [12.122.3.70]
6 92 ms 86 ms 75 ms 12.122.163.61
7 81 ms 76 ms 61 ms 12.255.10.234
8 61 ms 46 ms 40 ms 108.170.230.54
9 83 ms 73 ms 94 ms 72.14.235.3
10 61 ms 51 ms 53 ms dns.google [8.8.8.8]

Tracing route to af.8c.2ca9.ip4.static.sl-reverse.com [169.44.140.175]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms dsldevice.attlocal.net [192.168.1.254]
2 2 ms 1 ms 1 ms 107-199-200-1.lightspeed.sntcca.sbcglobal.net [107.199.200.1]
3 1 ms 1 ms 2 ms 71.148.135.252
4 74 ms 79 ms 84 ms 12.122.149.186
5 36 ms 105 ms 85 ms 12.122.149.57
6 4 ms 5 ms 4 ms 12.252.87.30
7 * 5 ms * ae6.cbs02.eq01.sjc02.networklayer.com [50.97.17.78]
8 5 ms 5 ms 5 ms e7.11.6132.ip4.static.sl-reverse.com [50.97.17.231]
9 5 ms 4 ms 4 ms po2.fcr02b.sjc03.networklayer.com [169.45.118.159]
10 4 ms 4 ms 5 ms af.8c.2ca9.ip4.static.sl-reverse.com [169.44.140.175]

Can this be brought up to AT&T somehow by Sonic? This can't be coincidence that this has been happening 4 nights in a row and it resolves itself after a few hours...

update: and of course like clockwork it's resolved itself around 11 PM PST:

Tracing route to dns.google [8.8.8.8]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms dsldevice.attlocal.net [192.168.1.254]
2 1 ms 1 ms 1 ms 107-199-200-1.lightspeed.sntcca.sbcglobal.net [107.199.200.1]
3 2 ms 1 ms 1 ms 71.148.135.252
4 24 ms 21 ms 21 ms 12.122.149.186
5 6 ms 8 ms 6 ms sffca22crs.ip.att.net [12.122.3.70]
6 4 ms 4 ms 4 ms 12.122.163.61
7 4 ms 4 ms 5 ms 12.255.10.234
8 9 ms 8 ms 7 ms 108.170.230.54
9 4 ms 5 ms 4 ms 72.14.235.3
10 6 ms 5 ms 4 ms dns.google [8.8.8.8]

Re: odd latency issue

Posted: Wed May 26, 2021 7:29 pm
by rebel316
Here we go again with the latency creep, 5th night in a row, starting around 7:30pm:

Tracing route to dns.google [8.8.8.8]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms dsldevice.attlocal.net [192.168.1.254]
2 1 ms 1 ms 1 ms 107-199-200-1.lightspeed.sntcca.sbcglobal.net [107.199.200.1]
3 1 ms 2 ms 2 ms 71.148.135.252
4 48 ms 40 ms 31 ms 12.122.149.186
5 33 ms 36 ms 62 ms sffca22crs.ip.att.net [12.122.3.70]
6 53 ms 64 ms 66 ms 12.122.163.61
7 33 ms 31 ms 32 ms 12.255.10.234
8 41 ms 36 ms 36 ms 108.170.230.54
9 47 ms 60 ms 68 ms 72.14.235.3
10 51 ms 53 ms 48 ms dns.google [8.8.8.8]

Re: odd latency issue

Posted: Thu May 27, 2021 8:57 am
by jhaines67
Thank You for keeping tabs on this.
I'm not as technical as you, but I have had Sonic DSL through ATT for many years and haven't changed anything in my home network and recently it seems very unstable at certain times like it's hard to resolve DNS requests (or something else strange).
Since it's so sporadic and hard to troubleshoot all I can do is reboot everything but it comes and goes so often I don't know what else to do.
-Jen