Sonic.net

Hello,
I am trying to troubleshoot a problem with my connectivity to sonic fiber service.

My setup includes 2 services into my home. The primary is using the sonic fiber service and I have secondary comcast as backup. I am using pfsense as my firewall.

I have had configured, and has been working for a while, a ddns service that updates the interface gateways. I am using Digital Ocean as the DNS service to redirect domain/sub-domain calls to the active IP in pfsense.

I also have HAProxy running on pfsense which redirects calls to a given web site running on my internal LAN.

All this has been working beautifully for at least 6 months. The problem is that I cannot get access to my web site through the sonic path.

I have gone through numerous tests and ruled out a failed web site, letsencrypt certs, access from haproxy backend. It's been frustrating. Last night I pulled the sonic fiber ethernet connection and pfsense switched gracefully to the cable connection (via gateway groups) and voila... its works.

We've had a series of power outages here recently and I even got a notification that there was a separate service outage/maintenance. I have re-powered the OTN a few times but it has stayed steady with the same IP address.

To summarize, my laptops and TVs can access the internet but I cannot get access into my network from outside. I get an immediate ERR_CONNECTION_CLOSED response from my sonic service but it works as expected when failed over to my cable connection.

Sorry to be so long in explanation, but any tips appreciated.

Thank you, Chris.

I am on 1 Gbps fiber fusion service and a pfsense router.
I have some services running from my house (blue iris on a server, openvpn on the pfsense device) and use dynamic dns to resolve my name into my sonic assigned ip and it works fine for me.

let's see i understand your setup.

You have pfsense with two wan ports with sonic as primary and the other as fail-over.
You have a ddns service and you advertise to them your sonic-assigned ip address so traffic to your domain name is routed to your sonic connection.

Locally you have a web server and you can reach it locally.

You have some routing in PF sense to send incoming traffic to the webserver. (by port or subdomain?)

It was working

It is not working anymore in that http queries to your domain are not served by your web server.

Have you tried to put some recording of the traffic?

You could configure one port of your pfsense device as a span port (mirror), connect a computer to it and record the traffic with wireshark for example.

Have you tried rebooting pfSense or restarting the HAproxy service? I’ve experienced issues like this with failed connections to WG/OpenVPN servers on pfSense after unexpected power loss and it’s occasionally been that simple.

Have you flushed DNS on your client device and confirmed that ddns is resolving to the current Sonic IP? A quick dig lookup should confirm.

Port check from the WAN side to ensure port 443 (or whatever you’ve configured HAproxy to use) is open and accepting connections on the Sonic WAN interface?

Are you running pfBlocker w/ inbound filtering? Any other packages that could be filtering on the Sonic WAN interface? I would confirm the rules on the Sonic WAN interface are correct.

Might double/triple check that the HAproxy frontend is correctly bound to the Sonic WAN interface and check the ACLs to ensure names match (case sensitive) between frontend/backend, etc.

Have you recently upgraded pfSense or the HAproxy package? Any other changes? I’ve found in the past small changes I’ve forgotten about have come back to bite – especially down the road for services I rarely use. Never hurts to check your config history just in case

Lastly, you might try bypassing dns and HAproxy as a test. Create a temp rule on the Sonic WAN interface to allow inbound traffic to that local host using the same port HAproxy is using. This would help narrow down where the issue lies (upstream, dns, HAproxy, SSL, etc.). Also check your state table to confirm whether packets are hitting the correct destination.

Update - I asked James in Support if he could reset the ONT and he graciously did so. It did not fix my problem but at least I could eliminate the Sonic fiber service as the problem.

I dug in deeper and found a reference to the concept of resetting the ARP table and/or the State table in pfsense. In a security cleanup in pfsense, I had cleared some unwise NATs and Rules and so it seems some of these are persistent in these tables.
Resetting the state table seems to have corrected my problem.

@js9erfan - Thank you for the thorough troubleshooting list.

Which IP address is the DDNS pointing to? Does it reflect which ISP is "active" at any time?

virtualmike wrote:
Which IP address is the DDNS pointing to? Does it reflect which ISP is "active" at any time?

Yes. In the Routing definitions of pfsense, there's a place to set up a fail-over. If the primary/tier1 (fiber) drops (Member Down) it automatically fails to the secondary/tier2 (cable). When primary comes back, it switches back to the primary. It's not immediate and I think it might take up to 5 minutes for the switchover. I sometimes notice it when I'm streaming a movie but it is very rare.

When it switches over, the DDNS service for each frontend I've configured switches in haproxy. I use Digital Ocean as my DDNS and I have an A record for each of my subdomains. When I test by pulling the fiber connection, I can watch the DNS A records in Digital Ocean switch to the cable IP address. It's fairly tight.

I really like pfsense.

ngufra wrote:
let's see i understand your setup...

You understand almost perfectly.

ngufra wrote:
You have some routing in PF sense to send incoming traffic to the webserver. (by port or subdomain?)

It is actually using haproxy service (detecting the FQDN including the subdomain part) with ssl terminated there. Between haproxy and the web servers behind it, it just uses http. I have set up letsencrypt/acme with a wildcard cert so it manages all frontends quite well and I avoid having to deploy the letsencrypt update config on each server. Internally, I access and test via http + hostname and with https + FQDN from outside my network.

ngufra wrote:
It was working - It is not working anymore in that http queries to your domain are not served by your web server.

Well, it is working now. I had executed an update of pfsense to the latest and made changes to NAT rules and bits and pieces. I found a reference that sometimes the states get messed up and that sometimes a reset can help. It did.

Apologies that I did not see your note earlier. Thank you.

chrisjx wrote:
I dug in deeper and found a reference to the concept of resetting the ARP table and/or the State table in pfsense. In a security cleanup in pfsense, I had cleared some unwise NATs and Rules and so it seems some of these are persistent in these tables.
Resetting the state table seems to have corrected my problem.

Yea, Netgate recommends resetting states after making NAT & rule changes. Good to hear you got it sorted.