[Solved] Fiber ONT + OPNSense appliance = WAN DHCP issues?

Internet access discussion, including Fusion, IP Broadband, and Gigabit Fiber!
4 posts Page 1 of 1
by marcmeszaros » Tue Jan 18, 2022 10:44 pm
Context
I recently purchased an OPNsense appliance (https://shop.opnsense.com/dec700-series ... appliance/). The appliance is basically a dedicated computer/device running OPNsense (fork of pFsense) and is HardenedBSD (a fork of FreeBSD) based.

I previously had a Netgear R7000 with DD-WRT as my main gateway from the ONT to the rest of my local network. After getting the OPNsense appliance, I swapped it in as the primary device connected to the ONT and the Netgear (DD-WRT) an unmanaged switch+access point after the OPNsense appliance.

Before the getting the new 10GB service I was using the Netgear router running DD-WRT with the 1GB service via AT&T until Sonic recently hooked up their lines to our neighborhood. Got the 10GB service and was running that without issue on the Netgear for 2 weeks before getting the appliance and using that as primary device attached to the ONT.

Issue
At first, everything was fine for a few days. Then after playing around with network topology and settings I seemed to have lost my public IP. Factory resetting both the OPNsense and Netgear router yielded no results in getting a public IPv4.

While on the phone with tech support, I found out the IP table was full (ONT holds 8 most recent IPs). Remote clear of the table instantly allowed the Netgear to get a public IPv4 via the ONT and showed up in the DD-WRT details (I had swapped back the Netgear to be directly connected to the ONT to simplify troubleshooting). After ending the call, I decided to try the new setup again with the OPNSense appliance connected to the ONT.

After about 6 hours, everything just dropped again. I just assumed it was the same thing as the first time and the IP table was full again and called support and called them back again. A few mins later after giving some context, and restarting the appliance everything was up again.

On the call I did mention that the OPNsense has default firewall rules of ignoring incoming ICMP requests because when the Sonic rep tried to ping the currently connected device there was no response.

It was also mentioned that IPv6 addresses take up more space in the IP table. I opted to disable IPv6 DHCP requests on the OPNsense appliance just in case it exhausted the 8 IP limit in memory (everything I read online says Sonic doesn't currently support native IPv6 on fiber... so not sure if IPv6's are actually being requested by the ONT or not... :shrug:).

I added a rule to the firewall to allow incoming ICMP requests and allow responses. Ironically enough, it cut out while I was writing this post, but an appliance reboot seemed to fix it. I'm starting to wonder if there's just something weird about the defaults in the OPNsense appliance causing issues with Sonic DHCP services...
Screenshot_20220118_224257.png
WAN firewall ICMP allow
Screenshot_20220118_224257.png (81.96 KiB) Viewed 17810 times
NOTE: No issues with LAN network at any point. Just WAN side public IP.

Questions
- Does anyone run an OPNsense instance between the ONT and the rest of their network?
- If so, has anyone specifically used the appliances you can get from opnsense.org partners?
- Would an ICMP deny firewall rule cause weird DHCP/ONT issues (in general and for Sonic specifically)?
- Any other hypothesis?
by js9erfan » Wed Jan 19, 2022 6:14 am
I don't use OPNsense but I have used pfSense quite a bit and have seen the same behavior over the years... The fix for me at 2 different locations facing this issue was to manually set the speed/duplex on the WAN interface. Its also said placing a dumb switch between the ONT and pfSense/OPNsense box can fix the WAN IP renewal issue (not that anyone would want to sacrifice a 10G switch for this purpose :().

Have you checked your DHCP logs or done a packet capture on the WAN interface when it fails to renew? I would look into that and contact the OPNsense devs for assistance on this as well. You might also try disabling gateway monitoring as a test or change the gateway monitoring IP to something else if monitor pings are getting blocked/ignored upstream. This would trigger a downed/offline gateway (assuming OPNsense operates similar to pfSense in this way).
by dane » Wed Jan 19, 2022 11:18 am
If you are causing your own connection to fail by filling up the MAC table on the Sonic equipment, if you could at least avoid calling support by phone and instead just text them, that would reduce the impact on other customers and be appreciated. Thank you!
Dane Jasper
Sonic
by marcmeszaros » Thu Jan 20, 2022 1:12 pm
dane wrote:If you are causing your own connection to fail by filling up the MAC table on the Sonic equipment, if you could at least avoid calling support by phone and instead just text them, that would reduce the impact on other customers and be appreciated. Thank you!
Didn't realize you can text support. Never noticed that as an option on the support page. Doesn't really matter anyway, only ever ended up calling twice. Once for initial MAC table clear and second time to make sure it wasn't full again a bit later.

After some digging around on the web I factory reset again, removed all plugins from OPNsense and started from scratch again. I setup gateway monitoring by pinging Google DNS 8.8.8.8. Checking out the client lease in "/var/db/dhclient.leases.igb1" everything seemed ok.

Code: Select all

lease {
  interface "igb1";
  fixed-address 192.184.x.x;
  option subnet-mask 255.255.252.0;
  option routers 192.184.160.1;
  option domain-name-servers 50.0.1.1,50.0.2.2;
  option dhcp-lease-time 21600;
  option dhcp-message-type 5;
  option dhcp-server-identifier 157.131.132.118;
  renew 4 2022/1/20 10:28:07;
  rebind 4 2022/1/20 12:43:07;
  expire 4 2022/1/20 13:28:07;
}
Deleted the file and renewed with the "dhclient" command in freebsd. Manually renewing in the web UI always worked without having to restart the device so that's a clue. I also found the logs after figuring out how to SSH into the device.

For good measure just did a packet capture and dumped into wireshark to make sure a renew without config does the right thing. Everything looked fine as far as ACK is concerned.
Screenshot_20220120_000235.png
Wireshark
Screenshot_20220120_000235.png (139.07 KiB) Viewed 17712 times
When I originally setup the new device I was unplugging and replugging a bunch of things in the ONT. I also read somewhere that other users in past versions of pfSense/OPNsense had WAN issues with other ISP's because of the fireawall booting before the ISP modem. Probably all of those things and the full MAC table just put everything in a weird state.

It's been almost 12hours now and no issues. So hopefully it keeps working. I have a hunch part of it may have been one of the plugins (maybe the Dynamic DNS plugin that was installed but not configured). At least now I have a working configuration I can make a backup of and do diffs if I ever manage to get back into the weird DHCP renew failure.
4 posts Page 1 of 1

Who is online

In total there are 31 users online :: 2 registered, 0 hidden and 29 guests (based on users active over the past 5 minutes)
Most users ever online was 999 on Mon May 10, 2021 1:02 am

Users browsing this forum: Bing [Bot], Google [Bot] and 29 guests