HTTPS/TLS 1.2 Handshake Issues

[mphilpot]

In the last 2 weeks I've been experiencing hung connections with https protocols. I've been able to track things down a bit. If I run

curl -v 'https://pbs.twimg.com/card_img/13337790 ... ame=medium'

as an example, about 2 of every 10 attempts or so hangs with the following output:

Code: Select all

*   Trying 2606:2800:220:13d:2176:94a:948:148e...
* TCP_NODELAY set
* Connected to pbs.twimg.com (2606:2800:220:13d:2176:94a:948:148e) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):

# Paused here for about 30 seconds

* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to pbs.twimg.com:443
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to pbs.twimg.com:443

Running it repeatedly, you occasionally see it connecting to different servers, but I see it fail and then work on the same server. Running the same set of requests on a different network sees no issues. Repeatedly running traceroute to the same domain see's no issues.

It's definitely not limited to the pbs.twimg.com domain as I see the behavior when connecting to other domains (browser seems to hang at TLS handshake step).

The traceroute for the above domain currently looks like:

Code: Select all

traceroute to cs672.wac.edgecastcdn.net (192.229.173.16), 64 hops max, 52 byte packets
 1  192.168.1.1 (192.168.1.1)  1.075 ms  0.741 ms  1.058 ms
 2  107-129-92-1.lightspeed.sntcca.sbcglobal.net (107.129.92.1)  1.913 ms  1.792 ms  1.818 ms
 3  71.148.149.186 (71.148.149.186)  2.358 ms  1.669 ms  1.719 ms
 4  71.145.1.52 (71.145.1.52)  2.728 ms  2.682 ms  2.923 ms
 5  12.83.39.209 (12.83.39.209)  3.482 ms
    12.83.39.213 (12.83.39.213)  3.221 ms
    12.83.39.209 (12.83.39.209)  4.419 ms
 6  12.242.117.22 (12.242.117.22)  5.647 ms  4.192 ms  4.866 ms
 7  att-gw.sfo.edgecast.com (192.205.32.238)  6.273 ms
    att-gw.sfo.edgecast.net (192.205.32.110)  6.586 ms  8.557 ms
 8  ae-91.core1.sjc.edgecastcdn.net (152.195.85.133)  5.214 ms
    ae-90.core1.sjc.edgecastcdn.net (152.195.84.133)  5.706 ms
    ae-91.core1.sjc.edgecastcdn.net (152.195.85.133)  5.090 ms
 9  192.229.173.16 (192.229.173.16)  6.226 ms  5.924 ms  6.050 ms
10  192.229.173.16 (192.229.173.16)  5.992 ms  5.782 ms  6.132 ms

[tigertech]

I'm using pure Sonic (not resold AT&T), and this problem doesn't happen to me.

However, I note this is an IPv6 target. Does the same thing happen if you force curl to use IPv4, with

Code: Select all

curl -4 -v 'https://pbs.twimg.com/card_img/1333779022996779009/hiPef8m1?format=jpg&name=medium

? If so, maybe try disabling IPv6 to see if it solves the other problems.

[mphilpot]

Just tried with the IPv4 flag and I still see the same behavior.

Curious though, I seemed to get even more failure modes. Sometimes I would get a Server hello response before timing out, sometimes a `bad signature` crypto error.

It still seems about 2 or 3 of every 10 requests goes into some sort of failure mode:

1. Client hello -> hang
2. Client hello -> Server hello -> hang
3. Client hello -> Server hello -> ... -> decrypt error

Mode 1 is definitely the most frequent.

Also want to clarify I'm seeing TLS issues with other domains (not just Twitter's img CDN).

[klui]

I'm on resold fiber. It started happening maybe 2-3 weeks ago and more on Wikipedia than most other sites with an SSL_ERR_BAD_MAC_READ from FF nightly. Sometimes I would get a hang during TLS handshakes on other sites. Typically a reload will fix it.

My take is it's some systemic issue with some websites or some TLS stack somewhere. It happens not only with FF but also with CLI tools so it's not a FF issue.

Doesn't happen too often, I'd get maybe at most a dozen times per day and I'm on the computer for over 12 hours each day. Seems to have gotten better since this weekend. There was 2 days in late November where I had duplicate WAN IP errors from my firewall. A reboot of the firewall fixed that but I still got those errors at times. This issue probably exacerbated the problem.