WiFi connection issue

I have been struggling with my WiFi connection for months now. I was on a paranoid android build for a couple months when I think it first started. Then I tried a version of CM12.1 and now I am on stock YOG4PASTN0. Each time, when I go with a new ROM (after wiping data and system partitions), the WiFi works fine. However, after about a week or 2, it goes sour. Using Speedtest.net, I get about 15Mbps up and 2Mbps down with a ping of about 22ms. When it is haywire, I get pings of 140ms with less than 1Mbps down but still 2Mbps up. So the up still works but the down stinks. Sometimes, it can't even complete the speedtest without "network connectivity issue".
I have tried messing with my ATT router, but nothing seems to help other than wiping data which is a pain. Any thoughts?