I have a small homelab that’s not nice enough for /r/homelab but is a bit more than just self hosting. Since I’m a decently knowledgeable sysadmin and network engineer, my goal is to build an enterprise-ish environment for myself to tinker around and play inside. This means a lot of my setup is more complicated than it needs to be and I spend a lot of time troubleshooting and debugging my overengineering, so when something breaks my first assumption is that it was something I did. I usually build my stuff to be relatively aelf sufficient when I leave it alone.

But this weekend and today I simply couldn’t find what I broke. I was attempting to move a clunky lets encrypt cert renewal job off of my DNS server to somewhere I could better manage it. Why was it on my DNS server? Because for a while now, dynamic updates only half worked for me. My bind9 server was fully capable and I have a custom nsupdate cronjob to update my DDNS records that I installed on my UDM-Pro. But for whatever reason, as soon as I entered my home network1 it wouldn’t work. Since I thought it better to manage my certs from Proxmox or another internal service, I needed to figure out why this was. I looked high, I looked low, I looked in /etc but there was no configuration error that I could find. I tested the same TSIG key on another machine in my VPC and on my UDM-Pro but there it went without a hitch. The error was weird — NOTIMP — and I couldn’t find anything relevant online. As a last resort I turned to ChatGPT2, but all this confirmed was that there should be no errors with my configuration. It’s conclusion was that it had to be networking.

So i scoured the configuration of my UDM looking for any filtering or traffic rules I had, but nothing was clicking. This wasn’t a connection issue, this is the server telling me that updates were not allowed for this zone. I was clearly hitting the DNS server, right? Well there was nothing in the update logs on the server, so I suspected that for some reason the requests weren’t making it through. So I spun up wireshark on my UDM and on my DNS server, and saw for myself that the dynamic update requests weren’t even reaching the bind server. I would see the update come into the router, and a response from the bind server, so what was responding? This was either some crazy filtering from my ISP — which i knew to be false because updates from the router worked — or my UDM doing something. Finally after some sleep I came back and looked at the UDM cobsole again and it hit me.

Ad block.

I quickly paused it and lo and behold it was blocking my dynamic updates. There was no record of this in the Insights tab; it was just silently absorbing my dynamic updates and masquerading as my name server. I can understand masquerading as name servers due to what its supposed to do, but I have no idea why it would steal my dynamic updates. I wouldn’t think what DNS filtering that enables is fail closed. For being a prosumer company, Ubiquiti’s features always feel halfway implemented to work in most scenarios but never actually developing full support for things. Yes, I brought this onto myself for enabling ad-blocking (it was good while it lasted, I’ll have to reimplement it in a non stupis way) but the fact that it does zero inspection of the DNS opcode before forwarding requests feels dumb.

1I have two “sites”, my homelab and a cloud VPC; critical infra like DNS and mail is hosted in the VPC.

2I minimally use AI for troubleshooting as a last resort to either turn me on a new path to the solution or as a sanity check before I blame a different component.