As I'm writing this post now, the latency has started creeping up again this evening.
The latency as you can see is way higher than usual (when there isn't this issue, everything is around 3-5 ms). Sometimes it takes longer than 5 seconds for a ping response, is why it logs all the failures in the 2nd screenshot. This doesn't disrupt my service/disconnect me, but I do notice lag and my speedtest results slow down drastically. I usually get about 930 down and 930 up, but at it's worst I've seen it result in 6 down 20 up.
This doesn't interrupt my service, but I would like to know what's happening.