The trace doesn't change between before and druing the drops, eg here's one:
Code: Select all
traceroute to 198.27.190.XXX (198.27.190.XXX), 30 hops max, 60 byte packets
1 * * *
2 107.191.59.33 (107.191.59.33) 3.621 ms 6.600 ms 9.574 ms
3 * * *
4 * * *
5 ae11-551.cr5-lax2.ip4.gtt.net (173.205.58.193) 0.320 ms 0.319 ms 0.257 ms
6 ae6.cr6-sjc1.ip4.gtt.net (89.149.180.78) 21.300 ms 20.721 ms 20.663 ms
7 as7065.xe-3-0-3.ar2.sjc1.us.as4436.gtt.net (69.22.130.110) 7.127 ms 8.299 ms 25.966 ms
8 100.ae1.cr1.equinix-sj.sonic.net (75.101.33.186) 105.655 ms 104.624 ms 102.205 ms
9 0.ae0.cr1.hywrca01.sonic.net (75.101.36.254) 87.581 ms 87.573 ms 87.563 ms
10 0.ae0.cr1.rcmdca11.sonic.net (157.131.209.74) 75.563 ms 216.230 ms 216.188 ms
11 0.ae1.cr1.snrfca01.sonic.net (157.131.209.138) 9.062 ms 9.069 ms 9.053 ms
12 0.ae2.cr2.snrfca01.sonic.net (157.131.209.170) 27.610 ms 93.338 ms 93.332 ms
13 0.ae0.cr1.snfcca05.sonic.net (198.27.244.50) 23.680 ms 107.220 ms 107.193 ms
14 300.ae0.bras1.snfcca05.sonic.net (198.27.186.140) 9.168 ms 9.164 ms 9.165 ms
15 198-27-190-XXX.fiber.dynamic.sonic.net (198.27.190.XXX) 10.077 ms 10.065 ms 9.796 ms
and here's two outbounds (one before, one during the inbound drops):
Code: Select all
traceroute to vt-lax-prox.as397444.net (144.202.126.211), 30 hops max, 60 byte packets
1 lo0.bras1.snfcca05.sonic.net (50.0.79.115) 1.373 ms 1.213 ms 0.924 ms
2 301.irb.cr2.snfcca05.sonic.net (198.27.186.210) 21.177 ms 21.043 ms 20.777 ms
3 * * *
4 0.ae2.cr1.snrfca01.sonic.net (157.131.209.169) 2.628 ms 2.535 ms 0.ae1.cr1.colaca01.sonic.net (157.131.209.65) 7.082 ms
5 0.ae0.cr1.lsatca11.sonic.net (157.131.209.86) 9.084 ms 8.970 ms 8.901 ms
6 * * 0.ae1.cr1.snjsca11.sonic.net (157.131.209.149) 7.329 ms
7 * * *
8 * * *
9 ae17.cr6-sjc1.ip4.gtt.net (69.22.130.109) 4.598 ms 3.181 ms 100.ae1.nrd1.equinix-sj.sonic.net (75.101.33.185) 2.770 ms
10 ae16.cr5-lax2.ip4.gtt.net (89.149.182.174) 10.110 ms 9.093 ms ae17.cr6-sjc1.ip4.gtt.net (69.22.130.109) 3.746 ms
11 ae16.cr5-lax2.ip4.gtt.net (89.149.182.174) 11.261 ms as20473-gw.lax20.ip4.gtt.net (173.205.58.194) 10.203 ms 10.036 ms
12 as20473-gw.lax20.ip4.gtt.net (173.205.58.194) 9.857 ms 9.955 ms 11.029 ms
13 * * *
14 * * *
15 * vt-lax-prox.as397444.net (144.202.126.211) 9.879 ms 9.428 ms
Code: Select all
traceroute to vt-lax-prox.as397444.net (144.202.126.211), 30 hops max, 60 byte packets
1 lo0.bras1.snfcca05.sonic.net (50.0.79.115) 0.889 ms 0.367 ms 0.906 ms
2 301.irb.cr2.snfcca05.sonic.net (198.27.186.210) 20.114 ms 300.irb.cr1.snfcca05.sonic.net (198.27.186.138) 12.889 ms 12.304 ms
3 0.ae14.cr2.snrfca01.sonic.net (198.27.244.49) 24.841 ms 0.ae14.cr2.colaca01.sonic.net (198.27.244.57) 41.855 ms 41.406 ms
4 0.ae1.cr1.colaca01.sonic.net (157.131.209.65) 39.654 ms 39.596 ms 39.586 ms
5 0.ae1.cr1.rcmdca11.sonic.net (157.131.209.137) 13.696 ms 13.033 ms 0.ae0.cr1.snrfca01.sonic.net (157.131.209.82) 8.229 ms
6 0.ae3.cr1.hywrca01.sonic.net (157.131.209.73) 6.947 ms 3.032 ms 2.978 ms
7 0.ae3.cr1.hywrca01.sonic.net (157.131.209.73) 11.082 ms 8.113 ms 7.960 ms
8 100.ae1.nrd1.equinix-sj.sonic.net (75.101.33.185) 8.542 ms 0.ae0.cr1.equinix-sj.sonic.net (75.101.36.253) 9.796 ms 14.513 ms
9 ae17.cr6-sjc1.ip4.gtt.net (69.22.130.109) 2.862 ms 100.ae1.nrd1.equinix-sj.sonic.net (75.101.33.185) 3.108 ms 3.031 ms
10 ae17.cr6-sjc1.ip4.gtt.net (69.22.130.109) 2.910 ms 3.099 ms 2.942 ms
11 as20473-gw.lax20.ip4.gtt.net (173.205.58.194) 11.141 ms 11.033 ms ae16.cr5-lax2.ip4.gtt.net (89.149.182.174) 9.386 ms
12 * * as20473-gw.lax20.ip4.gtt.net (173.205.58.194) 10.308 ms
13 * * *
14 * * *
15 vt-lax-prox.as397444.net (144.202.126.211) 9.715 ms * *
Both of the following pings were sent inside the tunnel at the same time (note the drops between 158 and 189 on the pings using 1448 bytes, which blows up to >1500 bytes outside but no drops on the smaller packets). The traffic was generated with iperf -l 1410 -V -u -c IP_INSIDE_VPN -b 200M, but the issue doesn't occur if the bandwidth is less than around 100M. I can provide pcaps as well, but there's nothing surprising there.
Code: Select all
1448 bytes from IP_INSIDE_VPN: icmp_seq=154 ttl=64 time=10.7 ms
1448 bytes from IP_INSIDE_VPN: icmp_seq=155 ttl=64 time=10.3 ms
1448 bytes from IP_INSIDE_VPN: icmp_seq=156 ttl=64 time=27.5 ms
1448 bytes from IP_INSIDE_VPN: icmp_seq=157 ttl=64 time=11.0 ms
1448 bytes from IP_INSIDE_VPN: icmp_seq=158 ttl=64 time=10.9 ms
1448 bytes from IP_INSIDE_VPN: icmp_seq=189 ttl=64 time=131 ms
1448 bytes from IP_INSIDE_VPN: icmp_seq=190 ttl=64 time=10.3 ms
1448 bytes from IP_INSIDE_VPN: icmp_seq=191 ttl=64 time=10.5 ms
1448 bytes from IP_INSIDE_VPN: icmp_seq=192 ttl=64 time=10.2 ms
1448 bytes from IP_INSIDE_VPN: icmp_seq=193 ttl=64 time=10.5 ms
1448 bytes from IP_INSIDE_VPN: icmp_seq=194 ttl=64 time=11.8 ms
1448 bytes from IP_INSIDE_VPN: icmp_seq=195 ttl=64 time=11.3 ms
1448 bytes from IP_INSIDE_VPN: icmp_seq=196 ttl=64 time=10.2 ms
Code: Select all
64 bytes from IP_INSIDE_VPN: icmp_seq=12 ttl=64 time=10.3 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=13 ttl=64 time=9.90 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=14 ttl=64 time=35.5 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=15 ttl=64 time=9.54 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=16 ttl=64 time=10.8 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=17 ttl=64 time=10.6 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=18 ttl=64 time=10.1 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=19 ttl=64 time=10.5 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=20 ttl=64 time=10.4 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=21 ttl=64 time=9.79 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=22 ttl=64 time=9.64 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=23 ttl=64 time=10.6 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=24 ttl=64 time=10.1 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=25 ttl=64 time=10.3 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=26 ttl=64 time=9.58 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=27 ttl=64 time=10.2 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=28 ttl=64 time=9.87 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=29 ttl=64 time=9.81 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=30 ttl=64 time=9.45 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=31 ttl=64 time=10.0 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=32 ttl=64 time=10.3 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=33 ttl=64 time=9.59 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=34 ttl=64 time=9.96 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=35 ttl=64 time=9.62 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=36 ttl=64 time=9.74 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=37 ttl=64 time=10.8 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=38 ttl=64 time=9.88 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=39 ttl=64 time=10.6 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=40 ttl=64 time=10.3 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=41 ttl=64 time=10.6 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=42 ttl=64 time=9.93 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=43 ttl=64 time=9.81 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=44 ttl=64 time=9.63 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=45 ttl=64 time=9.59 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=46 ttl=64 time=10.5 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=47 ttl=64 time=10.1 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=48 ttl=64 time=11.1 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=49 ttl=64 time=10.5 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=50 ttl=64 time=10.4 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=51 ttl=64 time=11.0 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=52 ttl=64 time=10.4 ms
64 bytes from IP_INSIDE_VPN: icmp_seq=53 ttl=64 time=9.53 ms