IPv6 prefix changes after every maint!

[graeme_stewart]

I totally get that Sonic has to do maintenance, I just wish the IPv6 prefix delegations wouldn't change after every one. It's frustrating and the spec (RIPE-690 and RFC8978) has lease times a lot longer than any given outage window. https://www.ripe.net/publications/docs/ ... ed-harmful

I also am understanding of the fact that sometimes prefixes need to change, but over the last few months we've had a couple of network maintenances, and each time the prefix has changed. If folks have long lived lease times on the customer hardware, this just makes for a bunch of frustration.

I love you Sonic, but sometimes that love hurts. Hopefully we're done with network maintenance that reallocates prefixes for a while?

[dane]

Both our IPv4 and IPv6 addresses are dynamic, so they’re subject to change from time to time.

[graeme_stewart]

I get that, and Dane, I'm a huge fan of you and the company, but that's a bit of a lazy answer? IPv4 had NAT so whenever you changed the IP for a v4 customer, nobody noticed and everything went along swimmingly, with v6 we're all using the prefixes you allocate for further network segmentation and host IP(v6) allocation, so whenever you change the prefix it creates a lot of churn in the network. Until the client lease expires (weeks/months) we've got all these old v6 address allocations and that's going to cause (eventually) problems. I think RIPE says it best:

If the CPE knows that the delegated prefix has changed, it should send out RA packets with a prefix valid lifetime of 0 to tell all devices that the old addresses are no longer valid. However, the CPE rarely knows that before the reboot there was a different prefix on the network, and the packets to revoke the old IPv6 addresses do not get sent. In this case, multiple IPv6 addresses from completely different assigned prefixes end up on the same network interface, some of which will no longer work and may imply increase the number of claims to the call-centre.

I'm not suggesting that you don't have dynamic allocation, that's totally fine, but maybe at a minimum, as a favor to customers, if you're going to have a maintenance that's anticipated to reallocate v6 prefixes you could let us know?

[matt.kinkele]

Thank you for casting a discerning eye over our v6 behavior. It's an area of development I'm personally very invested in.

Not all maintenance will impact your DHCP leases. Even though the downtime from these recent events in your area have been shorter than our DHCP lease times, these specific tasks were capacity upgrades that involved moving many users to a new gateway. We've wrapped up these upgrades your service area, which added a generous amount of room for growth - we don't expect to repeat this for quite some time.

While we don't intentionally cycle leases, there are roughly three conditions that would result in a change:

1. A subscriber disappears long enough for the lease to lapse.
If a given clients lease expires, its prefix is released back into the pool. This is uncommon, as our lease times are very long, and should significantly exceed the duration of planned maintenance windows.

2. Subscribers are migrated to a new layer 3 aggregation device.
This is typically done as outage mitigation in response to a device failure, but as mentioned previously is also necessary to distribute load as we add additional devices for growth. We plan our capacity improvements generously, so it should be years in between these events for a given service area.

3. The layer 3 aggregation device loses state.
While a reboot can also be necessary for certain failures, this is typically to execute a software upgrade. Since this is service impacting, we try to make these as infrequent as possible, but keeping the software on our equipment up-to-date is important for security and stability.

There are many other network components that may need occasional touching up, like layer 2 equipment or the physical fiber. These events shouldn't impact your leases unless they happen to also involve one of the above conditions.

As for problems due to long-lived leases out of an old PD persisting on client devices in your network, the RIPE BCOP you've cited outlines that, if aware that its assigned PD has changed, the CPE should send an RA invalidating the previous prefix to get client devices to renew (At least for SLAAC, perhaps stateful clients don't care?). I don't know what the state of adoption for this recommendation is, but if your router supports this, that seems like an elegant mitigation.

I am curious however what leases you have locally that don't expire for months, as that seems very long. Are you using SLAAC or stateful DHCPv6? Is this the default behavior of a consumer router?

[graeme_stewart]

Thanks for the background. I believe Comcast reduces the ipv6 lease time prior to maintenance, if this were an option at Sonic, it would be transparent to the majority of customers and everyone else (who cared!) could see the upcoming maint and plan accordingly. Or at a minimum, if the work is happening on systems that could result in a new lease, add a note to the maintenance?

we don't expect to repeat this for quite some time

That's great to hear!

I am curious however what leases you have locally that don't expire for months, as that seems very long.

From: https://www.cisco.com/c/en/us/td/docs/i ... v6-i3.html

All prefixes configured on interfaces that originate IPv6 router advertisements are advertised with a valid lifetime of 2,592,000 seconds (30 days) and a preferred lifetime of 604,800 seconds (7 days).

My bigger point is v6 lease changes are not as innocuous as v4 changes, and anything that can be done to help mitigate / manage on the customer side would be very much appreciated.