Slow / timed-out access to Facebook

[forest]

Kelsey, don't you think it's odd that facebooks's DNS servers would tell Sonic's DNS servers one thing and Sonic's customers another?

As for flushing DNS records, there's a low (900 seconds) TTL so it isn't going to make any difference.

Thanks for checking.

[kgc]

Kelsey, don't you think it's odd that facebooks's DNS servers would tell Sonic's DNS servers one thing and Sonic's customers another?

No, not particularly. I won't hazard a guess how Facebook manages their DNS responses to support geographic redundancy, load-balancing and other policies.

[forest]

No, not particularly. I won't hazard a guess how Facebook manages their DNS responses to support geographic redundancy, load-balancing and other policies.

Yeah, I suppose if any site is large enough to warrant complicated DNS, Facebook would qualify.

To anyone interested in submitting a problem report to Facebook, here are their instructions:
https://www.facebook.com/help/326603310765065/

[tbessie]

I'm seeing this too, have been for at least 2 weeks, but I think around a month.

Using all browsers, different systems, etc.

- Tim

[dsporter]

Kelsey, don't you think it's odd that facebooks's DNS servers would tell Sonic's DNS servers one thing and Sonic's customers another?

I work on global load balancing at Facebook; this is expected behavior. If people are still experiencing problems, please post more details on what specifically is wrong and we'll have a look. The more data you can provide (e.g. pcap, traceroute) the better. Thanks, and my apologies.

[forest]

If people are still experiencing problems, please post more details on what specifically is wrong and we'll have a look. The more data you can provide (e.g. pcap, traceroute) the better. Thanks, and my apologies.

At the risk of repeating myself...

While loading additional content for a page, such as when scrolling down in the news feed, requests to www.facebook.com are taking a long time to complete. These are requests for relative URLs (e.g. /ajax/pagelet/generic.php/LitesandMoreStoriesPagelet) while the browser's address bar says "https://www.facebook.com/". In the Network tab of Chrome's developer tools, their status remains "(pending)". Sometimes they take tens of seconds to complete, sometimes minutes, and sometimes they time out.

The problem shows up often (but not always) and disappears entirely when I add this entry to my /etc/hosts file:

Code: Select all

173.252.74.22   star.c10r.facebook.com   www.facebook.com

It also disappears when folks use Google's DNS servers instead of our ISP's. Here's how our ISP's DNS server is resolving the domain:

Code: Select all

www.facebook.com.	3509	IN	CNAME	star.c10r.facebook.com.
star.c10r.facebook.com.	27	IN	A	185.60.216.7

The effect of this problem is that facebook appears broken. Pages only partially load, comments fail to post, chat says "unable to connect", etc.

[oddhack]

If people are still experiencing problems, please post more details on what specifically is wrong and we'll have a look. The more data you can provide (e.g. pcap, traceroute) the better. Thanks, and my apologies.

At the risk of repeating myself...

Yeah, the "problem" is clearly coming from deep inside the interaction between FB's browser client code and the servers that code is talking to for incremental updates. I've already described the user-level problems that result, as has forest, and there's no realistic way as users of the site/browser we can diagnose it any further. Not worth much effort when I can just replace the Sonic DNS servers with ones that are observed to fix the problem behavior. Maybe I'll switch back in a month or two and see if things have improved.

[dsporter]

The problem should be mitigated now. Thanks for all of your help, and apologies this took so long for us to figure out.

[jordanschreiber]

I know you say the problem has been mitigated, but I see no evidence of that. I just got Sonic today and I can't access Facebook at all. The app won't update my newsfeed on either my iPhone or my iPad, and using Safari I can't access any website that includes Facebook.com in its URL. Whether I'm using a browser or the IOS app, it eventually times out.

[forest]

I just got Sonic today and I can't access Facebook at all.

That might simply be bad timing. They fixed the problem described in this thread, but some time within the past hour or two, facebook stopped responding entirely (even from the non-sonic network I just tried). Looks like this is a new problem.