Effectively the Synology Surveillance Station retried too many times to connect to an offline camera resulting in a overloaded Pi-hole. Removing the overloaded files and restarting the DNS daemon fixes it again. Here is my investigation and fix to the problem.
Suddenly my internet browsing became sluggish. Luckily i have a redundant Pi-hole infrastructure setup for resolving internal and external DNS records. The reason for thisredundancy is that the primary Pi-hole DNS server is residing on my HCI-server (Hyper-converged infrastructure). This means that rebooting the HCI results in a offline DNS server at the time that the hypervisor is booting. Effectively not being able to resolve the data stores that also reside on my HCI. To overcome this a Raspberry Pi Zero is acting as the secondary DNS server, sporting its own Pi-hole instance. The internal DNS entries in the custom.list are synced by using a SCP script.
In my case, when the primary DNS server fails, the resolving becomes a little sluggish. When you have only a single Pi-hole instance you will run into a non-resolving situation which restricts network access based on host names.
While examining the issue I noticed that the
pihole-FTL.db has grown substantially to a 1.3GB size;
berry@pihole01:~$ ls -h -l /etc/pihole/pihole-FTL.db -rw-r--r-- 1 pihole pihole 1.3G May 26 21:18 /etc/pihole/pihole-FTL.db
This shouldn’t be a problem but is seems kinda odd to me. I wonder what the log files can tell me. Whoah! The Pi-hole log file is also a whopping 1.3GB in size;
berry@pihole01:~$ ls -l -h /var/log/pihole.log -rw-r--r-- 1 pihole pihole 1.3G May 26 23:15 /var/log/pihole.log
Something fishy is going on for sure! Lets dive into the log file;
berry@pihole01:~$ tail -n 10000 /var/log/pihole.log
Thousands of similar entries showed up in the log file;
May 26 21:12:02 dnsmasq: query[A] cam01.testlab.home from 10.00.00.25 May 26 21:12:02 dnsmasq: /etc/pihole/custom.list cam01.testlab.home is 10.0.0.86
The output of this command showed me that my surveillance camera, which i disconnected temporarily from the network, is being polled every second for multiple times. This is obviously is a bug/undocumented feature of the Synology Surveillance Station software.
Looking closer at the
pihole-FTL.log I noticed that it was resizing the
/FTL-queries multiple times per second. This process was eating up the CPU and RAM resources of the VM.
So I need to make sure that the offline devices are not being polled anymore. This means setting the individual cameras to disabled or enabling ‘home mode’ in the Synology Surveillance Station software. Next thing up was to remove the overloaded database and the massive log file from clogging the Pi-hole system.
Removing the overweight database. This will be automagically recreated at restart of the DNS daemon.
berry@pihole01:~$ sudo rm /etc/pihole/pihole-FTL.db
Removal of the obese log file;
berry@pihole01:~$ sudo rm /var/log/pihole.log
Restarting the DNS server on the Pi-hole closed the deal and made the Pi-hole a happy dandy again.
berry@pihole01:~$ pihole restartdns [✓] Restarting DNS server