(in response to Joss)
1) Was I correct in my assumption that once the VPN was turned on, the router would not allow any traffic that did not use the VPN tunnels? My original thought was that the router would take that unencrypted, incoming traffic and pass it on as is. Apparently not.
I have not seen your configuration, so I can't be sure, but my guess is that once you connected to the Sonic VPN, you accepted their VPN as a default route. The fact that you were able to keep the dyndns pointer accurate even after routing had (probably) gotten jacked up is interesting, though. Usually, if dyndns is able to work from a host, both the inbound and outbound routes are working correctly.
That's going to break a lot of things if you need to access a server on that network.
I don't know your router, but is there an option to write a rule that says
"if a packet comes from 192.168.100.4, assign it ATT as its default gateway" and another that says "if a packet reaches ATT headed for my.dyndns.address, always pass it on to 192.168.100.4"
If so, you should not need a second router.
You can confirm by, wth the Sonic VPN connected, firing off the dyndns client from your server and seeing if it picks up your ATT address or the Sonic vpn address.
2) Why could I access the server through its external address, even though I had the VPN turned on? One of my friends posited that my router saw the IP address I was requesting, recognized the address as its own address and just kept the whole transaction internal.
A lot of things have various ways of doing just that automatically. A default route will almost always lose to a connected route, so the traffic is unlikely to be passed to the VPN tunnel if your setup knows everything is on the LAN. I'm not sure how much delay is built into the dyndns update system. I know that I needed to go find all the hosts dyndns uses and write rules to send their traffic out over ATT and not the VPN in order to have my lookup for local machines get sent in over ATT.
3) Did I handle this the best way?
You used a beta service on a production network that includes an element of income security for you on it.
Also, you tested firewall functionality from inside a firewall and trusted the results.
I am willing to bet that you learned a lot from this, though, and are unlikely to make either mistake again without being keenly aware that you are doing so, and probably enlisting someone to test for you, or going to another site to test for yourself.
Were it me, I'd ditch the second router and try seeing if it is possible to do all the routing all on your main gateway. It's much simpler not to have to chase routing problems across multiple routers.
I don't know if either of these will work with your gear, but they were helpful to me to figure out:
The default gateway does not have to have a default route tied to it.
The ATT-facing interface does not have to be the default gateway. On my network, the "WAN" interface has nothing at all crossconnected to it. This gives me much better control over my routes.
My default gateway is the VPN handoff - when it is up.
When it goes down, I have static routes to a work network and to Google's DNS that always run over ATT. The internet breaks as far as web browsing goes, though, and I'm OK with that.