I experience this exact same issue. I spoke with Sonic support about it and they advised I post here. I'd like to get a Sonic NetOps person engaged if possible -- I'm more than happy to help diagnose whatever way I can. This is an important issue for me and for others who are trying to administer or use AWS services in us-east-1.
This issue doesn't occur 24/7. It seems to occur more often in the evenings Pacific Time, although I just experienced it just now (morning Pacific Time) as well.
This is not just a TLS issue. I can reproduce it with plaintext HTTP as well. When it's happening, try a bunch of rapid-fire requests as follows:
Code: Select all
time curl -v -IXGET http://us-east-1.console.aws.amazon.com
You'll see that they fail intermittently.
I believe I've eliminated DNS as a culprit. This hostname (us-east-1.console.aws.amazon.com) currently resolves to 3 IPv4 addresses, regardless of which DNS server I use, and all 3 exhibit the same behavior when I force cURL to use them.
I've tested from multiple networks throughout the Bay Area. I've found at least one other network where this happens (Monkeybrains). I've confirmed that the issue occurs on multiple Sonic fiber connections. I've confirmed that the issue does *not* happen on Comcast, nor does it happen on Level3 or Cogent enterprise fiber circuits.
I've also confirmed that the issue does *not* happen when connected to Sonic VPN. This one is strange to me, since presumably it's all the same routing out of Sonic whether it's a residential or VPN connection, but I've tested extensively. VPN "fixes" the issue.
Sonic support asked me for some traceroutes from broken and working networks. I've collected a bunch here.https://fileshare.pmcc.net/RUjupsDu/BROKEN_sonic_residential_fiber.txthttps://fileshare.pmcc.net/xyqtjdzw/BROKEN_sonic_small_business_fiber.txthttps://fileshare.pmcc.net/hAsGWBJe/BROKEN_monkeybrains_residential.txthttps://fileshare.pmcc.net/eQrveVlu/WORKING_comcast_residential.txthttps://fileshare.pmcc.net/xjwBedkF/WORKING_level3_enterprise_fiber.txthttps://fileshare.pmcc.net/BasmeWjb/WORKING_cogent_enterprise_fiber.txthttps://fileshare.pmcc.net/RZXCdInL/WORKING_sonic_vpn.txt
The only commonality in upstream providers I've seen is that both Sonic and Monkeybrains use Cogent as their upstream for getting to AWS us-east-1, at least for ICMP. However this doesn't tell the full story, because 1.) It doesn't happen from Sonic VPN, which also uses Cogent as an upstream, and 2.) It doesn't happen from an actual Cogent enterprise fiber circuit in downtown SF.
Let me know what else I can do to help troubleshoot this!