pfSense + Dual (Multi) WAN ONT | WAN2 Issues

[mathewrice]

Hi everyone,
Any help or feedback is appreciated.
Here's a high level overview of my setup and the issue I am experiencing:

• Fiber service from Sonic supporting ONT with two WAN ports/connections
• pfSense configured for two WAN connections (WAN & WAN2)
• each ONT WAN port is connected to an independent Ethernet port on the pfSense router (igb0 & igb2)
• igb0 & igb2 are configured in pfSense as independent WAN connections
• until recently, there have been no issues
• currently, pfSense, at the dashboard, is indicating that igb2 is Pending for RTT, RTTsd and Loss, and Unknown for Status
• due to these statuses, anything on my network utilizing igb2 (WAN2) cannot connect to the internet
• Restarting/Power Cycling pfSense or the ONT is not resolving the issue
• in pfSense for Status > Interfaces, both igb0 & igb2 are receiving their own IP Addresses from Sonic (which is expected)
• called Sonic and had them clear my ONT MAC Tables but that did not fix the issue
• in pfSense, both WAN connections are receiving the same Gateway IPv4 from Sonic
• FWIW, I have IPv6 Configuration Type for igb0 & igb2 in the Interfaces settings set to None
• I've also tried swapping the connections between the ONT and the pfSense router
• when swapping the ethernet connections, I get the same IPv4 Addresses assigned to igb0 regardless of which ONT port I am connected to
• running pfSense v2.8.0-RELEASE

[mathewrice]

Seems the issue is stemming from the Gateway IPv4 Address that is being monitored in pfSense.
pfSense does not like that both WAN connections are using the same Gateway Address.
If I disable Gateway Monitoring for WAN2 (igb2), pfSense indicates that the internet connection is working for igb2 (assuming this is because it is being assigned a valid/unique IPv4 address and that it is no longer attempting to monitor the connection from the same Gateway IP address as igb0).

It would appear that the issue could be that Sonic has changed how it assigns IPv4 Addresses via DHCP for each of the ONTs WAN ports.

Is anyone familiar with Sonic's Multi WAN ONTs?

[js9erfan]

Try assigning a unique monitoring ip for each gateway interface (e.g. 8.8.8.8 and 8.8.4.4). Are you using these wan interfaces in a gateway group for failover, etc.? If so, you don't want to disable gateway monitoring. You should also set a default ipv4 gateway if you haven't done so already.

[mathewrice]

Try assigning a unique monitoring ip for each gateway interface (e.g. 8.8.8.8 and 8.8.4.4). Are you using these wan interfaces in a gateway group for failover, etc.? If so, you don't want to disable gateway monitoring. You should also set a default ipv4 gateway if you haven't done so already.

Thanks for the advice! If I am interpreting your response correctly, you're indicating that it doesn't really matter what the monitoring IP is so long as it is reachable by the monitoring service?
I.e., the monitoring service will check to see if the WAN interface can reach the monitoring IP address and that is how it validates that the internet connection is valid, correct?

Regarding my WAN interfaces and how they're configured for gateways, they're not setup in a gateway group or failover. I just have them segregated so I can specify which LAN interfaces use which WAN interface (e.g., LAN1 : WAN1 and LAN2 : WAN2).
I have specified one of them as the default IPv4 Gateway as well.

[js9erfan]

Correct, the monitor ip you set needs to respond to icmp or pfsense will show the gateway as down.

However, after reading your post again the issue is more likely due to Sonic using the same gateway ip for your multi-wan service. See here: https://docs.netgate.com/pfsense/en/lat ... gateway-ip

I have used pfsense for multi-wan setups but never with the same gateway ip. There are probably hacks you can do but the cleanest way to fix it would be for Sonic to provision different subnets/gateways for your multi-wan service as mentioned in the netgate doc.

[mathewrice]

Correct, the monitor ip you set needs to respond to icmp or pfsense will show the gateway as down.

However, after reading your post again the issue is more likely due to Sonic using the same gateway ip for your multi-wan service. See here: https://docs.netgate.com/pfsense/en/lat ... gateway-ip

I have used pfsense for multi-wan setups but never with the same gateway ip. There are probably hacks you can do but the cleanest way to fix it would be for Sonic to provision different subnets/gateways for your multi-wan service as mentioned in the netgate doc.

Thanks for confirming my interpretation. Setting a unique monitoring IP for both gateways resolved the surface level issue for the Gateways.
I agree that I need to contact Sonic to address the duplicate subnet/gateway addresses it is assigning to my ONTs WAN ports. I can confirm this wasn't how it was working before I encountered the issue. So something definitely changed on Sonic's end.

Thanks again for the perspective, tips, and all around help!