Multiple WAN IPv6 RAs on 10g fiber

[jgeboski]

I am trying to (elegantly) enable IPv6 with a 10g fiber connection. I have gotten it working with Prefix Delegation (PD) across a number of my interfaces. I am using a pfSense based firewall+router, which is connected to the AdTran 822v ONT.

I have unfortunately had to hard-code the default IPv6 gateway. My router sends a Router Solicitation (RS), resulting in the firewall's WAN interface receiving 2 Router Advertisements (RAs), see the tcpdumps below. The RAs are appropriately tagged with the preference.

The "low" preference RA is from MAC 00:24:45:fb:53:cf and "medium" preference RA is from MAC 5c:45:27:67:4e:80. The 00:24:45:* OUI is an AdTran MAC and the 5c:45:27:* OUI is a Juniper MAC. I am therefore assuming 00:24:45:fb:53:cf is the ONT and 5c:45:27:67:4e:80 is somewhere upstream on the OLT side.

The RA from 5c:45:27:67:4e:80 is the one that is needed for proper routing, otherwise IPv6 traffic gets dropped. While multiple RAs is valid, pfSense does not seem to handle it well. At least not as a client device on the WAN interface. It always ends up with the gateway from 00:24:45:fb:53:cf, not 5c:45:27:67:4e:80.

I tested with this my Linux+NetworkManager based PC directly connected to the ONT. It also receives the multiple RAs, but it receives the RA from 5c:45:27:67:4e:80 first. pfSense receives the RA from 00:24:45:fb:53:cf first. This behavior for both devices is consistent. I am not yet sure why it differs.

There might be something I can do on the pfSense side to better account for this. I need to explore this more. But I am wondering if I can just avoid the seemingly useless RA from the ONT (00:24:45:fb:53:cf) altogether?

Router Solicitation (from tcpdump):

Code: Select all

03:01:12.833032 xx:xx:xx:xx:xx:xx > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 70: (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::3eec:efff:fe42:760 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
	  source link-address option (1), length 8 (1): xx:xx:xx:xx:xx:xx
	    0x0000:  xxxx xxxx xxxx

Router Advertisements (from tcpdump):

Code: Select all

03:01:12.834313 00:24:45:fb:53:cf > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 78: (hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::224:45ff:fefb:53cf > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 24
	hop limit 64, Flags [other stateful], pref low, router lifetime 0s, reachable time 0ms, retrans timer 0ms
	  source link-address option (1), length 8 (1): 00:24:45:fb:53:cf
	    0x0000:  0024 45fb 53cf

03:01:12.848009 5c:45:27:67:4e:80 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 78: (class 0xc0, hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::5e45:27ff:fe67:4e80 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 24
	hop limit 64, Flags [managed], pref medium, router lifetime 1800s, reachable time 0ms, retrans timer 0ms
	  source link-address option (1), length 8 (1): 5c:45:27:67:4e:80
	    0x0000:  5c45 2767 4e80

[jgeboski]

The issue seems to be related to not using Rapid Commit (RC) with the DHCPv6 solicitation. Enabling the RC option prevents the RA from 00:24:45:fb:53:cf from arriving before the RA 5c:45:27:67:4e:80. This allows the WAN interface managed by pfSense to get the proper gateway.

My Linux+NetworkManager based PC enables RC by default. I suspect a lot of devices do this. It would explain why IPv6 works for many use cases out of the box.

pfSense does not have an elegant way of enabling the dhcp6c "rapid-commit" option when doing interface tracking, but this is another issue all together. I am exploring a few options here, but will probably just patch pfSense to add the option.

My questions at this point are:

1. Is Rapid Commit effectively required for DHCPv6 clients?
2. Should the ONT itself still be sending its own RA?

[tarzxf]

I'm using pfSense in downtown San Jose and able to get a DHCPv6 address assigned with the attached settings. I don't recall changing anything besides MAYBE the prefix size from /64 to /56

https://imgur.com/avRXIge

[js9erfan]

I checked a remote pfSense box that's on Sonic 10g. Seeing 1 ra every 10 min from Sonic (b4:8a:5f:97:5f:94).

As a reference my pfSense ipv6 related wan settings are here: viewtopic.php?t=16915&start=113

Code: Select all

No.     Time               Source                Destination           Protocol Length Info
      2 09:57:27.626902    fe80::b68a:5fff:fe97:5f94 ff02::1               ICMPv6   78     Router Advertisement from b4:8a:5f:97:5f:94

Frame 2: 78 bytes on wire (624 bits), 78 bytes captured (624 bits)
Ethernet II, Src: JuniperNetwo_97:5f:94 (b4:8a:5f:97:5f:94), Dst: IPv6mcast_01 (33:33:00:00:00:01)
Internet Protocol Version 6, Src: fe80::b68a:5fff:fe97:5f94, Dst: ff02::1
Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    Code: 0
    [Checksum Status: Good]
    Cur hop limit: 64
    Flags: 0x80, Managed address configuration, Prf (Default Router Preference): Medium
    Router lifetime (s): 1800
    Reachable time (ms): 0
    Retrans timer (ms): 0
    ICMPv6 Option (Source link-layer address : b4:8a:5f:97:5f:94)

No.     Time               Source                Destination           Protocol Length Info
     46 10:07:29.606544    fe80::b68a:5fff:fe97:5f94 ff02::1               ICMPv6   78     Router Advertisement from b4:8a:5f:97:5f:94

Frame 46: 78 bytes on wire (624 bits), 78 bytes captured (624 bits)
Ethernet II, Src: JuniperNetwo_97:5f:94 (b4:8a:5f:97:5f:94), Dst: IPv6mcast_01 (33:33:00:00:00:01)
Internet Protocol Version 6, Src: fe80::b68a:5fff:fe97:5f94, Dst: ff02::1
Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    Code: 0
    [Checksum Status: Good]
    Cur hop limit: 64
    Flags: 0x80, Managed address configuration, Prf (Default Router Preference): Medium
    Router lifetime (s): 1800
    Reachable time (ms): 0
    Retrans timer (ms): 0
    ICMPv6 Option (Source link-layer address : b4:8a:5f:97:5f:94)

No.     Time               Source                Destination           Protocol Length Info
     92 10:17:31.681913    fe80::b68a:5fff:fe97:5f94 ff02::1               ICMPv6   78     Router Advertisement from b4:8a:5f:97:5f:94

Frame 92: 78 bytes on wire (624 bits), 78 bytes captured (624 bits)
Ethernet II, Src: JuniperNetwo_97:5f:94 (b4:8a:5f:97:5f:94), Dst: IPv6mcast_01 (33:33:00:00:00:01)
Internet Protocol Version 6, Src: fe80::b68a:5fff:fe97:5f94, Dst: ff02::1
Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    Code: 0
    [Checksum Status: Good]
    Cur hop limit: 64
    Flags: 0x80, Managed address configuration, Prf (Default Router Preference): Medium
    Router lifetime (s): 1800
    Reachable time (ms): 0
    Retrans timer (ms): 0
    ICMPv6 Option (Source link-layer address : b4:8a:5f:97:5f:94)

[jgeboski]

I'm using pfSense in downtown San Jose and able to get a DHCPv6 address assigned with the attached settings. I don't recall changing anything besides MAYBE the prefix size from /64 to /56

https://imgur.com/avRXIge

I tried this (requesting over IPv4) and still got the two RAs.

I checked a remote pfSense box that's on Sonic 10g. Seeing 1 ra every 10 min from Sonic (b4:8a:5f:97:5f:94).

As a reference my pfSense ipv6 related wan settings are here: viewtopic.php?t=16915&start=113

These are the setting I effectively started with, I had all of those options selected. This still yields the double RAs for me.

I still get the double RAs using rapid commit. But the non-ONT RA comes first. I am using this patch with pfSense to toggle rapid commit, which has been working nicely so far.

[jgeboski]

Even with Rapid Commit being enabled and/or requesting the prefix over IPv4, I still had flakiness. I guess Rapid Commit or requesting the prefix over IPv4 just causes the non-ONT RA to come first more often. It seems this is just a race condition that I was mostly, but not completely winning.

pfSense has hidden rules for accepting RAs. It is not possible to block RAs in the Firewall Rules UI. Blocking the RA requires a rule be added very early before the rule to accept all RAs. I ended up creating a one-liner patch to block the RAs from AdTran ONTs and used the System Patches package to apply it:

Code: Select all

diff --git a/src/etc/inc/filter.inc b/src/etc/inc/filter.inc
index dec6f9d109..001ee2fb71 100644
--- a/src/etc/inc/filter.inc
+++ b/src/etc/inc/filter.inc
@@ -3802,6 +3802,7 @@ EOD;
 pass {$log['pass']} quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} ridentifier {$increment_tracker()} keep state

 # Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)
+block in {$log['block']} quick inet6 proto ipv6-icmp from fe80::224:4500:0:0/88 to ff02::/16 icmp6-type 134 ridentifier 99999 label "Block Adtran ONT RAs"
 pass out {$log['pass']} quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} ridentifier {$increment_tracker()} keep state
 pass out {$log['pass']} quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} ridentifier {$increment_tracker()} keep state
 pass in {$log['pass']} quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} ridentifier {$increment_tracker()} keep state

This has been working pretty reliably so far, I have yet to experience any flakiness. The source address of the RA is just an EUI64 local-link address using the ONT NIC MAC (an assumption based on the AdTran OUI). The patch will ignore RAs from local-link addresses with the AdTran OUI prefix. This should require very little maintenance as long as the patch continues to apply cleanly.


For anyone else that runs across this issue, it should be as simple as:

1. Installing the System_Patches package (if not already installed).
2. Navigate to: System -> Patches -> Add New Patch
3. Copy the patch contents above to the Patch Contents.
4. Check the Auto Apply box.
5. Save then apply the patch.
6. Navigate to: Status -> Filter Reload -> Reload Filter

You can sanity check the rule got added by running the following under Diagnostics -> Command Prompt:

Code: Select all

$ pfctl -s rules | grep "Block Adtran ONT RAs"

[mgoldburg]

@jgeboski nice detective work.

Worth posting in the Netgate forums if you haven't already done so, as pfSense seems to be ignoring the low/medium/high preference.

[jgeboski]

On the topic of pfSense supporting router preferences:

I spent sometime looking into writing a patch for this. But it is far more effort than I am looking to contribute, especially given the solution in my previous post is working solidly. pfSense makes the assumption that there will only ever be a single RA received over an interface. pfSense and I suppose even OPNsense are not likely to support router preferences anytime soon. Let me explain why...

pfSense disables stateless address auto-configuration (for good reason). pfSense instead relies on rtsold to send the RS and then listen for the RA. When rtsold gets an RA, it will call a script (rtsold argument -A) and pass it the sending router's address. The script (/var/etc/rtsold_<if>_script.sh) generated by pfSense then writes to a temporary file with the router's address. This temporary file is then referenced by pfSense.

There are two issues on the pfSense side:

The first issue: pfSense currently invokes rtsold with the -1 argument. This causes rtsold to exit as soon as it receives a single RA. This is why when the proper RA comes first, things work as intended. If the incorrect RA comes first, pfSense will only ever work with that RA, ignoring the proper RA.

The second issue: rtsold has no notion of router preference. However, the kernel does. But auto-configuration is disabled, so the kernel itself is just ignoring the RAs. The router address or priority are not stored anywhere. This means pfSense cannot even query the system for the list of routers attached to an interface.

The first issue on its own would be fairly straightforward to workaround. Either just keep rtsold alive indefinitely or adopt some magic with timeout and pkill -F. The second issue, however, will require rtsold to support router preferences. There are a few approaches that could be taken here, some putting more logic in pfSense, some putting less.

For now, the simplest fix is what I mentioned in my previous reply, using the one-liner patch for the early block rule.

[mgoldburg]

On the topic of pfSense supporting router preferences:

I spent sometime looking into writing a patch for this. But it is far more effort than I am looking to contribute, especially given the solution in my previous post is working solidly. pfSense makes the assumption that there will only ever be a single RA received over an interface. pfSense and I suppose even OPNsense are not likely to support router preferences anytime soon. Let me explain why...

Appreciate the detailed response and research that went into it! Will keep this post in mind for the time that Sonic infrastructure reaches my house (still running on their resold ATT fiber service).

I'd still encourage you to file this as a feature request, maybe on redmine, in the hope that Netgate will implement a general fix.

[jgeboski]

Created pfSense feature request for supporting multiple RAs and router preferences.