AT&T troubles again

[plwww]

Thank you Brian. I appreciate Sonic staying on this.

[plwww]

Any update on this? Just want to make sure I haven't missed any calls or anything. Unfortunately, we're struggling to stay connected above 400kbps today.

[briancw]

Hi plwww,

AT&T has reported that per remote testing your service should be up and running with no errors and have confirmed that you are in fact provisioned for 6mbps. When we were speaking with AT&T about this the AT&T rep was able to see 5 active devices connected over ethernet, 9 active over WiFi, and 6 inactive devices. There is a possibility that the connection is getting saturated.

What we would want to do is to disconnect everything from the modem except a single device over ethernet and run a speedtest over wireless to see what speeds you can get out of it with a single device connected. AT&T at times will push back on dispatches which they are doing a bit here. So we'll want to address their pushback which in this case is that they are theorizing the line might be saturated.

Give that a shot if you can and reach back out and we'll hit them again.

[plwww]

First, have I been waiting 9 days because AT&T denied there was an issue?...I feel like I'm starting over on this. Let me assure you the state of our line has changed little from the beginning(aside from them not fixing it and capping our speeds instead). Please re-read this whole thread above before continuing, to make sure all the context is fresh, as there's no reason we should be at a point where AT&T claims there's no issue with the line.

The DSL is losing sync. It is not merely too many devices using too slow a connection. The modem statistics during periods of instability show heavy FEC and CRC errors, errored seconds and unavailable seconds, and the line completely drops. In no reasonable world is this due to too many devices connected behind the modem. In fact, it will often come up and down during even light use. Most of those devices spend most of their time idle, and the wireless devices are mostly smart-plugs. Currently, after ~18 hours of it not keeping a sync faster than 1mb, it's back to 3mb(still capped at 3mb, never syncing higher).

Without disconnecting anything, right now I'm getting a speedtest of 2.7mbps down, 0.5mbps up: matching the current sync speed. It's sync speed(and "Max downstream available" value), had both been fluctuating between 200kbps, and 1.2mbps since yesterday evening at least, with very unstable service. It has stabilized again(briefly, I assume), at 3006kbps down, with "Max Downstream Attainable" bouncing between 5000kbps and 7600kbps.

AT&T's suggestion is a joke, so I'm not going to bother unplugging all my smart-plugs. I'll meet you halfway and disconnect ethernet though, which takes out several devices: results are 2.71 down, 0.46 up...again matching current sync(of 3mb, not 6).

....and big surprise: Line dropped again as typing this. Reconnected at a whopping 200kbps(yes, that's two-hundred kbps). I tried to run a speedtest for completeness, but it is failing to finish. After I load the page at sonic.com/speedtest and hit go, it measures the latency(37), then errors with, "could not connect". This is busy work, as a speedtest is pointless when the DSL connection itself is dropping.

After a few minute delay, and hitting the modem's button Diagnostics->resets->"reset connection", it is back up to 3mbps...surely temporary.

[plwww]

Here's some screenshots, edited for privacy:

Screenshot from 2021-02-17 17-02-37_edited.png (176.36 KiB) Viewed 1480 times

Screenshot from 2021-02-17 17-42-17_edited.png (176.38 KiB) Viewed 1480 times

I've also been saving the modem's Broadband Statistics page every 5 minutes, so long as my desktop is on. If you want those to help prove an issue with AT&T, let me know and I'll zip them up for you. Since they're sampling at 5 minutes they miss many short disconnects, but the values do show instability issues that AT&T apparently denies is happening.

[plwww]

The fact that we are months into this and AT&T is now denying there's even a problem, left me more unsettled the longer I thought about it. I ripped the values from the past few weeks of the Broadband Status page into csv, and graphed a few to further highlight the issues. First up is the graph of "DSL Initialization Timeout (DSLT) for the Current Day". Since these values are sampled 5 minutes apart, every rise in value from one dot to the next means the DSL line dropped at least once in that 5 minute period. You can see numerous long lines of vertical rises, meaning we may go hours at times with at least one drop every 5-30 minutes(typically for a minimum of 60 seconds to reconnect). Even the flatter sections typically show a drop every few hours. Saturation has nothing to do with this: the line is dropping.

20210201-20210218DSLT.jpg (55.95 KiB) Viewed 1452 times

Here is the graph for CRC Error count for the Current Day, which mimics the DSLT graph. If we see CRC errors, we almost always see a full drop.

20210201-20210218CRC(1).jpg (64.16 KiB) Viewed 1452 times

And then here's the graph for Severely Errored Seconds(SESL) and Unavailable Seconds(UASL). The y-axis is log-scale. Looking just at Unavailable Seconds(the time from the modem losing sync to active again), you can see most days we're seeing on the order of 15minutes of downtime, with some days having cumulative downtime of an hour or more.

20210201-20210218SESL-UASL.jpg (74.95 KiB) Viewed 1452 times

[plwww]

And as yet another example, the line has been down hard for at least 15 minutes now. Here's a screenshot from around 10 minutes in to the disconnect(I've also noticed this newer modem resets after failing to connect for some time, which is rather unfortunate as it clears the statistics). It will try to come back up, say it's connected at a few hundred kbps, then drop again before becoming active.

Screenshot from 2021-02-18 20-40-58_dsl_down_over_10_minutes_edited.png (160.23 KiB) Viewed 1423 times

[briancw]

Hi plwww,

I am very sorry this keeps dragging on. We have a ticket in with AT&T again and someone is calling over to get us a dispatch. Our team is calling out to AT&T today. You should be hearing from us a bit later today.

[plwww]

Well, they came out Saturday and might have partly solved the issue: it has been stable the last day and a half, but we're still stuck at 3mbps. The first AT&T technician here worked by the box to the house a while; eventually said the line looked great, and "I don't see any disconnects in the history"(which was surprising, since it had been up and down several times that morning alone); but there were a few errors so a line tech with a bucket truck would be out to look at it later(keep in mind the last time he was here, he said the same thing about the line but "couldn't escalate it to a bucket truck without a failed test"). That next guy looked at it a while, then said he's monitoring it and "It looks good, if it stays like this we're ok". After explaining the sporadic issue to him again, and getting a bit annoyed since he was clearly not going to fix anything, I tried showing him the data proving there are disconnects. He said, "No-one doesn't believe you, I can see its had some disconnects recently"..."really? That's good, the guy out earlier today said he saw no disconnects in the history": I have no faith in the job AT&T is doing or what they're telling me or Sonic. I asked him to let me know when he was done working on it, and left him to be; eventually he dropped us offline, ran up the road, came back and worked on the pole by our house again. The line came back up with a higher Max Downstream Attainable rate(8400), which was encouraging. But internet dropped at the house across the street right after he went up the pole(it's a relative's house, and served from the same pole, so I had been monitoring it as well for just that possibility). I let him know, he seemed surprised but said he'd go back up and check. He brought it back up then left without talking to me again, so I was unable to remind him about raising the line back to 6mb.

So far it hasn't dropped in the last day and a half, and has only seen 2 CRC errors in that time. The Max Downstream Attainable has stayed in the range 8300-8800kbps; Since we went through this a few weeks ago, only to fail after 2 days of stability, I'm reluctant to say it is resolved; but I am hopeful it will remain stable. We still need them to restore our speeds though.

[macguyver]

Reading through your trials with AT&T and just out of curiosity, which modem are you currently using? Sounds like you swapped the modem recently, so was it a different model before?

Your comment about the relative's house next to/across from you stood out to me. Are you mid-way on a longer street that has service down to the end, or are you near the end of the AT&T run?

Our Legacy DSL (6 mbps) went dead last month (the Pace 4111N modem could no longer sync so Sonic sent a replacement, and there were possible issues with the "DPG" [ie. Remote Terminal] for this area as multiple customers were affected). While troubleshooting our signal noise, the AT&T tech noted that we were on an old copper run that had no customers beyond our drop, so he terminated the line at our location to be more point-to-point vs vampire tap. This improved our signal reliability and while our speed has not changed, so far it appears streaming media has fewer drop-out/buffering issues.

This is wild speculation, but if your line pair is continuing down the street but not serving any customers perhaps they can try terminating it at your pole. How stable is your relative's line? In crazy-land thinking, it would be interesting if those two lines could be bonded and both of you somehow share the "double line". Depending how far your are, sometimes you can get a high gain wifi antenna to bridge the two buildings.

OK, put the following under general complaints/whining:

I am hearing familiar themes in your scenario with AT&T: Inability to trace/address issues while on site, Swapping copper pairs with other customers, Additional tasks that can only be done with different technicians/equipment, etc. While there are many skilled and willing technicians within AT&T, the company does not "support" their efforts in any number of ways. Sometimes repairs are achieved by cannibalizing other customers' existing resources, pushing problems onto others. The internal organization and communication at AT&T is simply abysmal.

I have numerous examples, but let me just mention this one: After moving a client to Sonic, I had to contact roughly 12 departments at AT&T across the country, at more phone numbers than I care to count, just to get their old sbcglobal.net email separated from their "business" account so they could cancel the $180+ monthly bill. It required a few full days on the phone to complete this task. Days... not hours.