/** * "System Health and Security Probe" is a program that runs periodically to * investigate general system health and security problems and to send report * emails to admins. It's designed to be used as part of RED SCARF Suite. * It requires at least an abridged version of RED SCARF Suite and Postfix, * installed and configured on a Debian server as described in the "Complete * Guide to a Complete Linux Server". For more details, please see the * "README.txt" file. * * Copyright (C) 2024 Double Bastion LLC <www.doublebastion.com> * * This file is part of "System Health and Security Probe". * * "System Health and Security Probe" is free software: you can redistribute * it and/or modify it under the terms of the GNU General Public License as * published by the Free Software Foundation, either version 3 of the License, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program. If not, see <http://www.gnu.org/licenses/>. */ DESCRIPTION: This program runs periodically and sends a report email to the administrator(s) if it detects at least one of the following problems: - One or more services running on the host server is in failed state. - The periodic ClamAV scan has detected any viruses in the mail directories or in the Nextcloud upload directories or the logs show recent virus detections in the incoming emails or in the files uploaded to Nextcloud. - Any new IP address has been banned during the last run interval due to repeated failed log in attempts against one of the applications monitored by Fail2ban. - The free disk space on all partitions is less than a threashold established by the admin. - The average CPU load in the last 15 minutes exceeded 100% utilization of all the CPU cores. It stores all the IP addresses banned by Fail2ban in the database and if one IP has been banned more than once, it includes the 'whois' data for that IP in the email report. It searches for the 'abuse' email address in the 'whois' data and writes a draft email that can be manually sent by the admin to the entity that owns the offending IP, to report the repeated attacks. The draft email includes the log lines containing the failed log in attepts of that IP. The text of the draft email is included in the periodic email report sent to the admin, below the 'whois' data of each IP that was banned more than once. If the 'automatic_emails_to_isp' parameter in the 'config' file is set to 'yes', the program will automatically send the abuse report email to the entity that owns the offending IP, at the moment its number of bans increases by 1, if it has been banned at least once in the past. Before sending the email report to the admin, the program compares it with the last sent report, and if there is nothing important or new to report, the new email will not be sent, since it is assumed that the admin has already been informed about the current problems in the previous email. This is to avoid flooding the admin with too many emails. For example, if at a particular moment it detects that the only problem of the server is that there are 75 banned IPs, and at the moment of the next run it detects that 4 of those IPs have been debaned due to their bantime being over, it won't send the email report, because the debanning of the 4 IPs is not important enough to deserve a new email. However, if it detects that since the last run a new IP has been banned, the email report will be sent. REQUIREMENTS: This program is designed to be used as part of 'RED SCARF Suite'. It may work in other contexts but only if heavily modified. It requires at least an abridged version of RED SCARF Suite and Postfix, installed and configured on a Debian server as described in the "Complete Guide to a Complete Linux Server". At least the following jails need to be set in '/etc/fail2ban/jail.local': sshd, postfix, postfix-sasl In general, the log files written by the programs installed on the server should be rotated only when they reach about 2M in size (with the exception of WordPress websites, whose logs need to be rotated once every day because of Matomo), which can be configured in their respective logrotate files located in the '/etc/logrotate.d' directory. This is because 'System Health and Security Probe' needs to read a large number of log lines in order to find enough failed log in attempts that can be included in the draft abuse report emails. Important !!! The ban time set for all the jails in '/etc/fail2ban/jail.local' (in seconds) must be greater than the interval between two consecutive runs of the program, that you set in the 'shsp-config.php' file in the '$time_in_hours' parameter. If the run interval is set to 8 hours, which is the default, all the jails in Fail2ban must have a ban time greater than 8 hours (which means greater than 28800 seconds). Also, in '/etc/fail2ban/jail.local' it's necessary to specify a ban time (in seconds) for each and every jail, even for the jails who use the default ban time, for which it's not required. So, in every jail block, a 'bantime' parameter should be included. Important !!! If you change the run interval in the 'shsp-config.php' file (in the '$time_in_hours' parameter), you should change the cron job accordingly. To run the script every 8 hours, at 5 minutes past, the cron job should be: 5 */8 * * * php /srv/scripts/shas-probe/system-health-and-security-probe.php > /dev/null 2>&1 INSTALLATION: Once the applications that make up RED SCARF Suite are installed (at least the 10 kernel components of the suite and Postfix), to start using this program just copy all its files and directories to '/srv/scripts/shas-probe' (or in another suitable location of your choice), then create a MariaDB database, user and password, give the user all the priviledges over that database, except GRANT, then copy the database name, user and password in the corresponding parameters of the 'shsp-config.php' file. Also, set the value of all the other parameters in the 'shsp-config.php' file. Add a cronjob to crontab (use the 'crontab -e' command) like the following: # Run System Health and Security Probe every 8 hours 5 */8 * * * php /srv/scripts/shas-probe/system-health-and-security-probe.php > /dev/null 2>&1 Next, create the directory to store the periodic ClamAV scan reports. The default name and location for this directory is '/srv/scripts/detections' but you can change it and mention it as such in the 'shsp-config.php' file. Then set up two cronjobs to get ClamAV to scan two sensitive directories. Keep the file names 'clamav_nextcloud_report' and 'clamav_mail_report' unchanged, as shown below. You can change the directory for these files ('/srv/scripts/detections') and the time and frequency of scanning, according to your needs. Replace 'example.com' with your domain: # Scan the '/var/www/cloud.example.com/data' directory and the '/var/vmail' directory with ClamAV every three days 5 3 */3 * * cat /dev/null > /srv/scripts/detections/clamav_nextcloud_report && clamdscan --fdpass --quiet /var/www/cloud.example.com/data -l /srv/scripts/detections/clamav_nextcloud_report 40 3 */3 * * cat /dev/null > /srv/scripts/detections/clamav_mail_report && clamdscan --fdpass --quiet /var/vmail -l /srv/scripts/detections/clamav_mail_report That's it, the program is ready to fulfill its mission. To be able to see the email reports properly, your email client should accept html emails. LICENCE: GNU GENERAL PUBLIC LICENSE v3+