Upstream bandwidth saturation

Internet access discussion, including Fusion, IP Broadband, and Gigabit Fiber!
11 posts Page 1 of 2
by dhwalker » Wed Aug 30, 2017 2:55 pm
I've had intermittent short (several seconds to a few minutes) interruptions to my Internet connectivity (FTTN X2 - Pace 5268AC) over the past few months. During the interruptions, it appears there's no Internet connectivity; IMAP servers do not respond, new pages do not open in a browser, pings to ping.sonic.net are not answered, etc. The DSL links, however, remain connected.

The problem was recently recently diagnosed, with the help of Sonic support, to be due to the periodic CrashPlan backups my five home systems run. I've since limited the bandwidth used by the CrashPlan client software, and things are much better, but any large upload causes another interruption.

Do others see this? Do you have strategies to mitigate it? Sonic, this feels like a buffer bloat related issue in the 5268AC; is anyone working on a fix?

David
by pockyken007 » Wed Aug 30, 2017 3:05 pm
dhwalker wrote:I've had intermittent short (several seconds to a few minutes) interruptions to my Internet connectivity (FTTN X2 - Pace 5268AC) over the past few months. During the interruptions, it appears there's no Internet connectivity; IMAP servers do not respond, new pages do not open in a browser, pings to ping.sonic.net are not answered, etc. The DSL links, however, remain connected.

The problem was recently recently diagnosed, with the help of Sonic support, to be due to the periodic CrashPlan backups my five home systems run. I've since limited the bandwidth used by the CrashPlan client software, and things are much better, but any large upload causes another interruption.

Do others see this? Do you have strategies to mitigate it? Sonic, this feels like a buffer bloat related issue in the 5268AC; is anyone working on a fix?

David

Last time I experienced this was on DSL line from sonic ages ago , when I was running a lot of uploads when the upload would reach the upper limit of what my pipes were able to push the internet would pretty much be unusable ... now that I am on fiber I haven't seen this issue ( but then it's really hard to max out those pipes )
by miken » Wed Aug 30, 2017 3:08 pm
dhwalker wrote:Do others see this? Do you have strategies to mitigate it?
There are a few ways to mitigate upload saturation. You can schedule the uploads to occur at a time of night when people are not using the connection, you can limit the upload speed that is being used so it doesn't saturate your connection or you can get a router that provides QOS (quality of service) that can help differentiate and prioritize traffic so that you run into less issues.
dhwalker wrote:Sonic, this feels like a buffer bloat related issue in the 5268AC; is anyone working on a fix?
The Pace 5268AC used for FTTN is an AT&T provided router, so unfortunately firmware updates are out of our hands. That being said, I don't think better bufferbloat will fix the issue. If my understanding of bufferbloat is accurate, it is essentially the router timing out because it's too backed up with requests. If you were to have a device with better bufferbloat - you'd still run into the symptoms of a saturated line like a slow connection, time out requests, etc.
Mike N.
Development Trainer
Sonic
by dhwalker » Wed Aug 30, 2017 4:08 pm
miken wrote:The Pace 5268AC used for FTTN is an AT&T provided router, so unfortunately firmware updates are out of our hands. That being said, I don't think better bufferbloat will fix the issue. If my understanding of bufferbloat is accurate, it is essentially the router timing out because it's too backed up with requests. If you were to have a device with better bufferbloat - you'd still run into the symptoms of a saturated line like a slow connection, time out requests, etc.
I understand that, ultimately, things will slow down. What I'm looking for, though, is a more graceful degradation. Right now, everything stops (other than, I suppose, the upload). What I want is for everything to slow down but not stop.

If I did get a router with better packet queue management, are the recommended alternatives? Would I lose some degree of Sonic support if I replaced the Pace router with something else? Has anyone put a WiFi router behind the Pace router and limit its Ethernet upstream to a little less than what saturates the Pace DSL upstream?
by bmah » Mon Sep 04, 2017 3:26 pm
dhwalker wrote:
miken wrote:The Pace 5268AC used for FTTN is an AT&T provided router, so unfortunately firmware updates are out of our hands. That being said, I don't think better bufferbloat will fix the issue. If my understanding of bufferbloat is accurate, it is essentially the router timing out because it's too backed up with requests. If you were to have a device with better bufferbloat - you'd still run into the symptoms of a saturated line like a slow connection, time out requests, etc.
I understand that, ultimately, things will slow down. What I'm looking for, though, is a more graceful degradation. Right now, everything stops (other than, I suppose, the upload). What I want is for everything to slow down but not stop.

If I did get a router with better packet queue management, are the recommended alternatives? Would I lose some degree of Sonic support if I replaced the Pace router with something else? Has anyone put a WiFi router behind the Pace router and limit its Ethernet upstream to a little less than what saturates the Pace DSL upstream?
I'm in a similar situation (FTTN, Pace 5268AC, CrashPlan backups saturating the uplink, resulting in very poor interactive performance and Nagios timeouts to my VPS). I have a pfSense box behind the Pace, and I ended up using using pfSense's traffic shaping facilities to: 1) set the outgoing link bitrate to something a little less than what the actual uplink bitrate is. 2) put CrashPlan traffic in a low-priority queue. 3) put ICMP and NTP traffic in a high-priority queue. pfSense actually includes a wizard to set up the traffic shaping and queue assignment...it might be doing some other setup under the covers that I'm not aware of. In any case, I've only had this setup in place for a week or so, but it seems to be working pretty well (remote logins work better and I'm no longer getting Nagios complains whenever CrashPlan starts up backups).

I think that in theory it should work to put any other form of bulk upload traffic into the low-priority queue (in particular other online backup services), although you need to be able to write a firewall rule in pfSense to match the traffic.

Let me know if you want more details or have specific questions. I left out a few details such as the fact that almost all of my NAT-ed outbound traffic goes through sonic.net's OpenVPN service and that I've got an IPv6 tunnel running inside the OpenVPN tunnel.

Bruce.
by pockyken007 » Tue Sep 05, 2017 11:17 am
miken wrote:
dhwalker wrote:Do others see this? Do you have strategies to mitigate it?
There are a few ways to mitigate upload saturation. You can schedule the uploads to occur at a time of night when people are not using the connection, you can limit the upload speed that is being used so it doesn't saturate your connection or you can get a router that provides QOS (quality of service) that can help differentiate and prioritize traffic so that you run into less issues.
dhwalker wrote:Sonic, this feels like a buffer bloat related issue in the 5268AC; is anyone working on a fix?
The Pace 5268AC used for FTTN is an AT&T provided router, so unfortunately firmware updates are out of our hands. That being said, I don't think better bufferbloat will fix the issue. If my understanding of bufferbloat is accurate, it is essentially the router timing out because it's too backed up with requests. If you were to have a device with better bufferbloat - you'd still run into the symptoms of a saturated line like a slow connection, time out requests, etc.

Alternatively he could use software that limits the bandwidth available for upload / download leaving him some extra for browsing etc ...
by dhwalker » Sat Sep 23, 2017 9:17 pm
bmah wrote: I'm in a similar situation (FTTN, Pace 5268AC, CrashPlan backups saturating the uplink, resulting in very poor interactive performance and Nagios timeouts to my VPS). I have a pfSense box behind the Pace, and I ended up using using pfSense's traffic shaping facilities to: 1) set the outgoing link bitrate to something a little less than what the actual uplink bitrate is. 2) put CrashPlan traffic in a low-priority queue. 3) put ICMP and NTP traffic in a high-priority queue. pfSense actually includes a wizard to set up the traffic shaping and queue assignment...it might be doing some other setup under the covers that I'm not aware of. In any case, I've only had this setup in place for a week or so, but it seems to be working pretty well (remote logins work better and I'm no longer getting Nagios complains whenever CrashPlan starts up backups).

I think that in theory it should work to put any other form of bulk upload traffic into the low-priority queue (in particular other online backup services), although you need to be able to write a firewall rule in pfSense to match the traffic.

Let me know if you want more details or have specific questions. I left out a few details such as the fact that almost all of my NAT-ed outbound traffic goes through sonic.net's OpenVPN service and that I've got an IPv6 tunnel running inside the OpenVPN tunnel.
Bruce.
Thanks, Bruce. How is pfSense working for you? I decided to get a Linksys WRT3200ACM wireless router that I plan to put DD-WRT router software on. pfSense looked pretty good, but I use WiFi pretty extensively, so I wanted the strong WiFi hardware that tends to be available only for specifically-marketed WiFi devices.

I'm thinking of starting off simple, turning on PIE or some other queue management algorithm that addresses buffer bloat, but not setting any specific priorities. As I said earlier, I don't mind things slowing down; I just don't want them to stop for a few minutes. I think better queue management should do that without me worrying about which protocols are consuming a lot of bandwidth and lowering their priorities. I'll post my experiences after I've gotten some experience.
by dhwalker » Sat Sep 23, 2017 9:25 pm
pockyken007 wrote: Alternatively he could use software that limits the bandwidth available for upload / download leaving him some extra for browsing etc ...
That's what I've done with CrashPlan. The issue is that I have to do that on a per-host basis, so I have to set each host's limit low enough so that more than one host can be running its backup at the same time. That means that a single host will run its backup slowly, even when bandwidth is available. The better solution is for the router to manage the aggregate bandwidth (without "bloating").
by pockyken007 » Mon Sep 25, 2017 12:05 pm
dhwalker wrote:
pockyken007 wrote: Alternatively he could use software that limits the bandwidth available for upload / download leaving him some extra for browsing etc ...
That's what I've done with CrashPlan. The issue is that I have to do that on a per-host basis, so I have to set each host's limit low enough so that more than one host can be running its backup at the same time. That means that a single host will run its backup slowly, even when bandwidth is available. The better solution is for the router to manage the aggregate bandwidth (without "bloating").
hm... fair enough , I didn't think about the fact you have to do it host by host and therefore limit the overall bandwidth .
by bmah » Thu Oct 05, 2017 9:27 am
dhwalker wrote:Thanks, Bruce. How is pfSense working for you? I decided to get a Linksys WRT3200ACM wireless router that I plan to put DD-WRT router software on. pfSense looked pretty good, but I use WiFi pretty extensively, so I wanted the strong WiFi hardware that tends to be available only for specifically-marketed WiFi devices.

I'm thinking of starting off simple, turning on PIE or some other queue management algorithm that addresses buffer bloat, but not setting any specific priorities. As I said earlier, I don't mind things slowing down; I just don't want them to stop for a few minutes. I think better queue management should do that without me worrying about which protocols are consuming a lot of bandwidth and lowering their priorities. I'll post my experiences after I've gotten some experience.
pfSense is working great for me, although admittedly I've used it for many years so I've grown accustomed to any quirks it may have.

Couple points (maybe you're aware of these already): 1) pfSense is software, not hardware...you can run it on many types of x86_64 hardware. At my house, I'm using the Netgate SG-2440 (https://www.netgate.com/products/sg-2440.html), after using a Soekris net5501 for years. There might be less expensive ways to do this, but this fits my needs well and I didn't have to spend a bunch of time doing integration. Also it's a way of financially supporting pfSense.

2) You don't have to have the same piece of hardware doing your routing function and wireless. The above SG-2440 has no wireless interfaces on it...I'm using a couple of Ubiquiti AP-AC-LITE (https://www.ubnt.com/unifi/unifi-ap-ac-lite/) access points, which have excellent radios.

The traffic prioritization worked pretty well for getting CrashPlan to back up over my 1Mbps sonic.net/AT&T FTTN upload, without interfering with other traffic. But right now, my current project is to back up a bunch of stuff to Backblaze B2 (getting ready to migrate away from CrashPlan). Those uploads go out a Comcast connection at 6Mbps (because the 1Mbps upload over sonic.net/AT&T FTTN would take way too long). I didn't set up anything in particular to handle the Backblaze B2 uploads, so I think all I'm using is the default pfSense traffic shaper setup, which allocates some upstream bandwidth for TCP ACKs. In fact I was trying to figure out what ruleset I could use to de-prioritize the Backblaze uploads, and haven't been able to think of a good one yet.

Bruce.

PS. I was hoping to be able to get rid of my Comcast connection, but the FTTN upload speed is too low for me to rely solely on that for network backups, even after I'm done with this CrashPlan -> Backblaze B2 project. To put it mildly, this sucks. :-( I'm probably going to keep the FTTN anyway, since we were paying about the same for sonic.net Fusion (and AT&T POTS before that). Hope these ramblings are useful.
11 posts Page 1 of 2

Who is online

In total there are 29 users online :: 1 registered, 0 hidden and 28 guests (based on users active over the past 5 minutes)
Most users ever online was 999 on Mon May 10, 2021 1:02 am

Users browsing this forum: Bing [Bot] and 28 guests