Pace 5268AC - DMZplus Bug | Workaround provided, input welcome

Internet access discussion, including Fusion, IP Broadband, and Gigabit Fiber!
13 posts Page 1 of 2
by ftffttn » Thu Feb 28, 2019 10:42 pm
Hi all,

Several weeks ago, I noticed that SSH connections to a remote AWS host would freeze or hang intermittently whenever a moderate amount of text output was received. By "moderate", I mean a normal system update command like apt update or viewing a log file via cat <log file>, or using scp to transfer config files.

Basically, this issue had all of the earmarkings of an MTU issue. I spent a good amount of time diagnosing the problem, which was earmarked by the following symptoms:
    1. Used tcpdump on the remote host and determined that the remote host wouldn't get an ACK from my computer, resulting in the host retrying the packet.
    2. Tcpdump on my computer indicated that the ACK was sent
    3. My computer would retry the ACK
    4. My computer would receive packet retries from the remote host
    5. The remote host wouldn't receive any of the ACK retries from my computer

What made this especially frustrating was that no hardware on my end changed in the 2 years since I've been a Sonic member (not the router, computers, or the Pace 5268AC). Even worse, the "DMZplus" setting had been enabled for my router the entire time, and you'd think that this would be the last place a problem would originate.

The Fix
After much trial and error, I did a factory reset on the modem, disabled "DMZplus", and reinitiated DHCP for the 5268AC and SSH works properly again. I can cat 6MB+ log files with ease. However, if I reenable "DMZplus" for my router and reset DHCP, the SSH problem returns. There is a catch, however. With "DMZplus" mode off, my router has to use the 5268AC's firewall, which severely limits the usefulness of the service...

In short, the issue appears to have been caused by a firmware glitch on the 5268AC. This problem began around the time I received the "11.1.0.531418-att" firmware update. Hopefully, this helps someone else who is experiencing the same problem. In the meantime, I'd love to find a way to bypass the 5268AC, switch devices, or bring my own FTTN compatible modem, like the BGW210-700. Heck, I'd be OK with continuing to pay the modem rental fee if I could use a device that can actually bridge.

Any Input?
I've spoken with a number of friendly Sonic technical representatives, but they tend to default to the "yeah, this is your problem since you're running your own router." However, it's not unforeseeable that customers would want to run their own network hardware in lieu of what AT&T provides.
by chirano » Fri Mar 01, 2019 12:46 pm
Your diagnosis that the firmware is the culprit is probably right. On DSLReports, AT&T customers report that this version of the firmware causes a significant performance hit when DMZPlus is used. My connection speed has dropped to about 40 Mbps with DMZPlus enabled compared to 50 Mbps when it's off.

AT&T is presumably aware of the issue, but it seems the company doesn't care about it enough to roll back the firmware or issue a bug fix. The only "fix" currently appears to be to get AT&T to replace the Pace with a non-Pace gateway.
by ftffttn » Sun Mar 03, 2019 12:32 am
@Chirano, thanks for the reply

Firmware is definitely the issue. I downgraded my firmware to an older version and SSH works as expected with DMZplus enabled. One of AT&T's reps even mentioned a work around after acknowledging (a first!) this bug:

https://forums.att.com/t5/AT-T-Fiber-Eq ... 5153#M7940

For what it's worth, if this were due to something on my end, I'd consider it a learning experience. However, this is a problem with AT&T's hardware. AT&T insists on exclusively controlling all aspects of their hardware, yet they can't get it right. They've been charging us for 3 months since the bug was deployed and have yet to deploy a fix. Worse yet, we're obligated to deal with it because their only competition is Comcast. My network's finally working as expected after I got involved and wasted a lot of time troubleshooting what would otherwise be considered an MTU glitch. And to add insult to injury, AT&T's previous firmware leaves the 5ghz wifi radio on, with no way to disable it.
by ftffttn » Mon Jun 17, 2019 10:47 pm
The most recent update (11.2.1.531810) was released by AT&T several days ago. While SSHing in to my box from work, I noticed that pip3 and apt-get were freezing up. Strange. I confirmed that the latest firmware update had been automatically installed on my 5268AC, did a factory reset, and reconfigured DMZ+, but the glitches (described above) that I experienced were not resolved. What a pain.

I downgraded to 10.5.3.527283 and everything works as expected; however, this shouldn't be considered a long-term or stable fix. Unfortunately, Sonic can't provide a workaround or fix for this AT&T induced problem, and Fiber isn't coming to my neighborhood anytime soon, so I'll be evaluating alternate ISPs.
by willtull » Wed Jul 31, 2019 3:15 pm
This definitely sounds like the PACE firmware issue which causes problems with DMZPlus enabled. Neither Sonic nor AT&T were willing or able to solve this despite many people reporting the issue on the AT&T forums. I finally bought my own Arris BGW210 which immediately fixed the issues.
by bbrendon » Sun Dec 22, 2019 12:23 pm
I was banging my head on this for about a day trying to figure this out. Finally narrowed it down enough and typed in the right keywords. That go me here. I ended up calling Sonic support and they said they would see about getting a different modem for me from ATT but couldn't guarantee anything.

The other option is to bypass the modem but that's a PITA itself.

My box currently reports firmware 11.4.1.532484-att
by gtwrek » Thu Jan 02, 2020 8:53 am
For what it's worth, my Pace got upgraded to 11.4.something, and my ssh connections are now dog slow/unusable.
And now, it appears that I can no longer use my previous workaround of backrev the Pace to 10.5.something. I tried that, and it appears the Pace no longer is recognizing the backrev firmware as valid.

I'm not sure what the best solution. Disabling DMZplus, and Double-NATing will break things for me.

There's some esoteric solutions out there involving just using the Pace for authentication, then turning it off.

Other options seems to be forcing a truck roll from ATT to acquire an BGW210 router instead. This must be a truck roll as online support can't guarantee which router will be delivered, and experience has shown ATT will just keep shipping replacement Pace (even if they say they're shipping an BGW210).

I'm not sure which option I'm going to use, but as it is the Fiber link for me is almost unusable...
by mwolczko » Mon Mar 02, 2020 3:08 pm
Thanks to all contributions to this thread.

A few weeks ago I switched over to DMZplus* and everything seemed fine except wi-fi calling stopped working and now I've found that scp of all but small files doesn't work. If I throttle scp down to a few KB/s it seems to run indefinitely, but of course this is ridiculous - I have gigabit fiber, and Speedtest shows 950/950 most of the time, so I'm throttling to 1/250000 of the raw bandwidth.

The firmware is currently at 11.5.1.532678. Any pointers on a different version that might work better, and how to obtain and install it, would be gratefully received. Does Sonic monitor these threads? If so, I hope they're bumping this problem up to AT&T.
Does AT&T announce new releases anywhere? I'm still working through the info on DSLreports and elsewhere...

Mario

* The reason for the switch was that I was having lots of problems with AT&T's DNS, the Pace has those servers wired in, and DHCP cannot be disabled or told to use any other. DMZplus nicely took care of those issues.
by gtwrek » Mon Mar 02, 2020 3:48 pm
Yep, the scp (and in generall ssh) issues you are seeing are exactly the problem I was having. A 1Gbit fiber link running slower than an V.32bis modem..

FWIW, I resolved the problem by completely bypassing the PACE. I use it on power up to do the 802.x authentication required, then completely turn off the PACE and bypass it. Way into unsupported territory - but it's been working fine for me for months. (As long as the power isn't interrupted to the ONT, it appears that you never need to re-authenticate...) I dislike such hacky solutions (spouse acceptance factor is low - power outages require fiddling around to bring the network back up), but ATT has really given us no choice.
by Sonic Guest » Tue Mar 03, 2020 1:51 am
Call Sonic and tell them to inform AT&T you want another RG. Specify the BGW-210. That way you won't have to buy one from eBay.
13 posts Page 1 of 2

Who is online

In total there are 11 users online :: 2 registered, 0 hidden and 9 guests (based on users active over the past 5 minutes)
Most users ever online was 487 on Tue May 05, 2020 2:07 pm

Users browsing this forum: Bing [Bot], Google [Bot] and 9 guests