Sonic.net

I was having some trouble using a few of my static addresses this morning, and I've tracked it down to some very strange behavior, apparently on Sonic's end.

I have addresses x.x.x.208-x.x.x.215. From an outside box, if I ping .208, all is well. If I pinged .209-.215 I saw no traffic whatsoever on my DSL link, *including ARP*. This happens regardless of whether I've claimed .209 on my router or not. So only one of my addresses was working this morning.

But here's the really weird part. I can do this from my router:

root@Aether:~# arping -I eth1.1 -s x.x.x.208 x.x.x.1
ARPING to x.x.x.1 from x.x.x.208 via eth1.1
Unicast reply from x.x.x.1 [ec:44:76:63:5f:43] 23.393ms
Unicast reply from x.x.x.1 [ec:44:76:63:5f:43] 18.618ms
^CSent 2 probe(s) (1 broadcast(s))
Received 2 reply (0 request(s), 0 broadcast(s))

That's expected -- that's sonic.net answering with proxy ARP to maintain the illusion that I have eight addresses on a /24 (when, of course, I actually have eight addresses on a /29).

But this is wrong:

# arping -I eth1.1 -s x.x.x.208 x.x.x.215
ARPING to x.x.x.215 from x.x.x.208 via eth1.1
Unicast reply from x.x.x.215 [ec:44:76:63:5f:43] 23.413ms
^CSent 1 probe(s) (1 broadcast(s))
Received 1 replies (0 request(s), 0 broadcast(s))

Hi, Sonic's router. You're answering proxy ARP on *my* address. If I send an unsolicited ARP to forcibly update your cache on that address, then you stop answering and that address starts working. So I can bring my IPs to life one at a time by sending unsolicited ARPs.

I think you have a bug. I have the ZTE single-port device in bridge mode. I doubt that it's the problem.

What's the router type on your side?

OpenWRT, currently somewhat out of date, running on a Ubiquiti RouterStation Pro. It has arp_ignore=1, which should not cause a problem -- it will just prevent my router from answering ARP for addresses that are internal. I'll flash it tonight and see if anything changes. I can also try to reproduce the problem from a directly-connected laptop.

I would be very surprised if it has a bug that causes it to show completely made-up output from arping, though.

I'm at work right now and I can't ping or ssh in. I could earlier today, so I suspect that the ARP cache has forgotten about me again.

I did some more digging. I can trigger this problem at will by hardcoding the mac address of my gateway, using arptables to suppress all outgoing arp requests, and changing my mac address. The gateway will never issue a new arp query to determine my new mac.

Once I do this, outgoing traffic from any of my IP addresses works (checked with tcpdump on a remote machine I control), but I get no incoming packets. As soon as I send an unsolicited ARP packet (e.g. arping -A or arping -U) from my router on any of my IPs, that IP starts to work. If I don't send an unsolicited ARP packet, though, the gateway will never send me a request.

This is consistent with a failure of my modem to forward incoming broadcast packets to me (although outgoing would still have to work) or a failure of the sonic.net gateway to send me arp requests at all.

Given the gateway's other strange behavior (sending proxy arp replies when queried for some of my addresses), I suspect the latter.

Sounds like there's a known issue with multiple static addresses on the same physical interface. Here's my guess as to why:

If I have a single address configured, I have a normally functioning computer or router, and I send IP packets on occasion, then I'll send an ARP request along with the packet, and the router's cache gets refreshed. All is well (until I have just a server that doesn't send anything for long enough for the router's cache to expire my entry).

If I have multiple addresses on the same interface, though, my system will choose one of them from which to send its ARP request. That will refresh the router's record of the mac address for *only that IP*. The secondary addresses will never send ARP requests, so they won't work.

FWIW, there's another bug, I think. If I set the high bit of my MAC address, even gratuitous ARP requests don't seem to reliably do the job.