Ad block users are already the minority.
Adblockers are a must have for protecting against mailware and phishing attacks. Google and other ad services have proven to be incompetent in protecting their service against those kind of attacks targeting users.
Those attacks you see are mostly (close to 100%) harmless bots, scripts. Yes they are trying default passwords and exploits that got patched years ago.
If you do not use default credentials and do run up to date software there is nothing to worry about.
Even brute force attacks are rare.
This is just “noise” so to speak.
If you are scared by this, you should reconsider hosting something on the internet. Yes things like fail2ban can help but only if they knock on your server multiple times and mostly only to keep your logs clean.