Bufferbloat over fiber, what to do?

[dtaht]

I *loved* my sonic fiber when I lived in SF - even with 60ms of default jitter and latency when doing a big upload it smoked everything else I've ever worked on. Adding in cake, though: *wow*. no jitter, 3ms latencies cross town even when fully loaded. No matter what we did. I actually achieved a 25 year old dream, of being able to play music live with someone across town,
as the latencies were so low as to be like being in the same bandstand together.

https://www.internetsociety.org/wp-cont ... atency.pdf

I'd so love it if sonic's and/or AT&T's default gear put sqm/fq_codel or cake in front of it for uploads. It's NOT the bandwidth,
but the consistently low RTTs that let you use the bandwidth fully.

Anyway, what I'd done was discard the supplied sonic box and put in an apu2 instead, tell cake it was 106Mbits (I was getting 110mbit raw), and that was the result I got, linked earlier. ~60ms sawtooth originally vs 3ms consistently - a 20x improvement (and about a 5ms sawtooth for tcp also) It also did 6rd pretty good. The apu2 is the lowest end routing platform I know of that can do a gbit down and 100mbit cake up.

Sadly I left SF and sonic for comcast-dominated climes.... I'd even put in for a job at sonic because it was working so good... they never called me back. (their dsl stuff is seriously bufferbloated).

[dtaht]

@danielg4 - do I know you? I know *one* danielg but haven't heard from him in a decade.

The procedures for bypassing and improving on ( https://github.com/aus/pfatt ) at&t's box are intimidating, and useful.

I keep hoping the biggies will just start demanding rfc8290 compliance and a working sqm from their vendors - and autoconfig it for their customers - rather than have to go through all this rigamarole.

[danielg4]

@danielg4 - do I know you? I know *one* danielg but haven't heard from him in a decade.

Not unless you have a perfect memory with regard to everyone you interacted with in the context of CeroWrt and OpenWrt.

The settings you just gave tell us when you were in SF, and are sadly no longer useful, because the 1000/100 service over fiber was very brief, before it was changed to 1000/1000. Is cake still recommended for that? Piece of cake, or layer cake? Egress vs. ingress?

[dtaht]

I have no idea what the gige symmetric service would be like. Except I'm jealous. :)

IF the fiber ONT exerts hw flow control at a reasonable value for internal buffering, then cake running at gigE "line" rate on egress would be a good option. On ingress, well, cake is too cpu intensive to run at 1gige on an apu2, and you'd have to revert to the older sqm system that leverages fq_codel.

You'd have to go measure with a tool like flent as things like dslreports are too unreliable at speeds like this. A potential problem is gpon is a shared network and using a fixed shaped rate in cake may not work at peak hours, in either direction, and if you don't have hw flow control (any sonic folk here?), then you might have issues. Sorry man, you gotta go measure....

Regardless, at these speeds generally the bloat shifts to the wifi, which also had major bufferbloat issues on many chipsets. OSX gets it right now, as do linux based APs using the modernized APIs we put in on the qca ath9k, ath10k, mediatek, and most recently, the newest intel chips. I'm looking forward to benching the intel products soon.