why high latency / so many hops

[gatsbylee]

Hi,

I uses some Apps or Service from outside of US.
( even including YouTube content from loaded from outside of US )

When I used xfnity, I didn't see much latency issue or page loading issue.

I started wondering if this issue is related to the backbone Sonic Internet uses.

And, why are there so many hops in Sonic Internal switches?
one of hop even takes more than 1sec.

Is there a way to reduce the number of hops or latency?

My wife started complaining that it's worse than just using xfinity 50M Internet.

Thank you.
Gatsby

```
traceroute kensington.co.kr
traceroute: Warning: kensington.co.kr has multiple addresses; using 65.8.158.46
traceroute to kensington.co.kr (65.8.158.46), 64 hops max, 52 byte packets
1 192.168.1.1 (192.168.1.1) 0.525 ms 0.198 ms 0.185 ms
2 lo0.bras1.snmtca11.sonic.net (157.131.132.100) 1.426 ms 1.337 ms 0.644 ms
3 300.irb.cr1.snmtca11.sonic.net (157.131.211.202) 13.731 ms
301.irb.cr2.snmtca11.sonic.net (157.131.211.210) 14.981 ms
300.irb.cr1.snmtca11.sonic.net (157.131.211.202) 17.092 ms
4 0.ae4.cr2.lsatca11.sonic.net (157.131.211.69) 38.036 ms 23.291 ms
0.ae0.cr1.snmtca11.sonic.net (157.131.211.61) 31.118 ms
5 0.ae2.cr1.colaca01.sonic.net (157.131.209.69) 12.152 ms
0.ae2.cr1.lsatca11.sonic.net (157.131.209.161) 10.542 ms
0.ae2.cr1.colaca01.sonic.net (157.131.209.69) 4.814 ms
6 0.ae0.cr1.snrfca01.sonic.net (157.131.209.82) 1397.077 ms 1102.650 ms
0.ae0.cr1.lsatca11.sonic.net (157.131.209.86) 9.620 ms
7 0.ae4.cr1.hywrca01.sonic.net (157.131.209.77) 3578.004 ms 2363.012 ms 2196.572 ms
8 * * *
9 100.ae1.nrd1.equinix-sj.sonic.net (75.101.33.185) 3.706 ms * 3.875 ms
10 99.83.69.168 (99.83.69.168) 3.720 ms 5.590 ms
100.ae1.nrd1.equinix-sj.sonic.net (75.101.33.185) 4.318 ms
11 150.222.97.249 (150.222.97.249) 4.521 ms
150.222.97.255 (150.222.97.255) 5.476 ms
99.83.69.168 (99.83.69.168) 4.627 ms
12 150.222.97.5 (150.222.97.5) 4.030 ms
150.222.97.83 (150.222.97.83) 3.507 ms 3.817 ms
13 54.240.242.19 (54.240.242.19) 4.262 ms
15.230.36.154 (15.230.36.154) 3.608 ms
15.230.36.162 (15.230.36.162) 4.348 ms
14 150.222.30.94 (150.222.30.94) 4.220 ms
15.230.36.166 (15.230.36.166) 5.285 ms
150.222.30.82 (150.222.30.82) 3.844 ms
15 15.230.36.191 (15.230.36.191) 4.043 ms
15.230.36.185 (15.230.36.185) 4.171 ms
15.230.36.177 (15.230.36.177) 3.881 ms
16 * 15.230.36.181 (15.230.36.181) 4.264 ms
15.230.36.191 (15.230.36.191) 4.803 ms
17 * * *
18 * * *
19 * * *
20 * * *
21 * server-65-8-158-46.sfo53.r.cloudfront.net (65.8.158.46) 3.717 ms 3.599 ms
```

[dane]

The individual ping response times of routing devices will vary, it’s the total round trip to the end that is actually accurate. In this case, just over three milliseconds to and from the final hop.

Traceroute is a bit confusing this way. One hop‘s router shows high latency, the next not…how can that be? Because the CPU is used to respond to pings, not the routing engine. Traceroute latency info is only useful in finding congestion for example if it looks like this:

Hop1: 2ms
Hop2: 3ms
Hop3: 4ms
Hop4: 100ms
Hop5: 105ms
Hop6: 110ms

From this, one could infer that the link between the routers at Hop3 and Hop4 is adding latency. (But even that can be wrong, more on that below.) This could also sometimes be seen when connections were made between countries, via undersea cables of significant length. Everything after that point is a step higher in latency.

But on the other hand, if you see something like this:

Hop1: 2ms
Hop2: 3ms
Hop3: 4ms
Hop4: 100ms
Hop5: 6ms
Hop6: 7ms

This would just mean that the Hop4 router isn’t fast in replying to ping, but it IS routing traffic without delay, which can be seen at the subsequent hops.

This misunderstanding of traceroute is a common one, and leads to a lot of confusion for users.

Another common issue is asymmetrical paths. You can easily look at the first example and assume that the Hop3 to Hop4 is the link with the issue. But the return path of the traffic may be different, and that can also be the issue. And that can’t be seen or diagnosed without a traceroute from BOTH sides of the link.

Not sure if this helps with your issue related to performance, but the traceroute isn’t showing any trouble.

[dane]

The thing that does look fishy here is the back and forth to and from the same POPs. Checking on that.

[gatsbylee]

Hi Dane,

Thank you for your response.
I got what you mean.

And, thank you for taking a look.

Gatsby

[bgile]

Hi Gatsby,

What you are seeing in that traceroute is "multipath load balancing behavior". Traceroute is using UDP ports when sending out its probes as part of the TTL sequence and they increment the port number. This will show equal paths through our backbone because we have equal-cost-multipath-routing or ECMP from our fiber locations. The router will treat each port as its own flow and that flow will get hashed equally across our backbone from the fiber headend so really you are seeing 2 equal paths happen on one traceroute.

It is very confusing to look at I agree. MTR uses ICMP which has no port and so in most cases it looks more straightforward. As a side note, I did see that some distributions on Linux will have different traceroute packages and in some cases how the UDP port is incremented can be handled differently.

That said to address the latency you saw, our backbone does not prioritize... so it will always appear to perform poorly. The important thing to look at is the latency to the last hop, and find the position in the traceroute where hops are consistently slower.

[gatsbylee]

Hi Gatsby,

What you are seeing in that traceroute is "multipath load balancing behavior". Traceroute is using UDP ports when sending out its probes as part of the TTL sequence and they increment the port number. This will show equal paths through our backbone because we have equal-cost-multipath-routing or ECMP from our fiber locations. The router will treat each port as its own flow and that flow will get hashed equally across our backbone from the fiber headend so really you are seeing 2 equal paths happen on one traceroute.

It is very confusing to look at I agree. MTR uses ICMP which has no port and so in most cases it looks more straightforward. As a side note, I did see that some distributions on Linux will have different traceroute packages and in some cases how the UDP port is incremented can be handled differently.

That said to address the latency you saw, our backbone does not prioritize... so it will always appear to perform poorly. The important thing to look at is the latency to the last hop, and find the position in the traceroute where hops are consistently slower.

I hesitated to leave a reply to your comment because my comment would make this post go up to the top of the posts in this board. ( If I noticed it correctly )
But, I wanted to leave "thank you" comment to you

Thank you for the detailed explanation
Gatsby