/**
 *  "System Health and Security Probe" is a program that runs periodically to 
 *  investigate general system health and security problems and to send report
 *  emails to admins. It's designed to be used as part of RED SCARF Suite.
 *  It requires at least an abridged version of RED SCARF Suite and Postfix, 
 *  installed and configured on a Debian server as described in the "Complete
 *  Guide to a Complete Linux Server". For more details, please see the
 *  "README.txt" file.         
 *                                                                                         
 *  Copyright (C) 2024  Double Bastion LLC <www.doublebastion.com>           
 *                                                                                         
 *  This file is part of "System Health and Security Probe".                               
 *                                                                                         
 *  "System Health and Security Probe" is free software: you can redistribute
 *  it and/or modify it under the terms of the GNU General Public License as                  
 *  published by the Free Software Foundation, either version 3 of the License,            
 *  or (at your option) any later version.                                                 
 *                                                                                         
 *  This program is distributed in the hope that it will be useful,                        
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of                         
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the                          
 *  GNU General Public License for more details.                                           
 *                                                                                         
 *  You should have received a copy of the GNU General Public License                      
 *  along with this program.  If not, see <http://www.gnu.org/licenses/>.                 
 */



DESCRIPTION:

 This program runs periodically and sends a report email to the administrator(s) if it detects at least
 one of the following problems:

 - One or more services running on the host server is in failed state.

 - The periodic ClamAV scan has detected any viruses in the mail directories or in the Nextcloud upload
   directories or the logs show recent virus detections in the incoming emails or in the files uploaded
   to Nextcloud.

 - Any new IP address has been banned during the last run interval due to repeated failed log in
   attempts against one of the applications monitored by Fail2ban.

 - The free disk space on all partitions is less than a threashold established by the admin.

 - The average CPU load in the last 15 minutes exceeded 100% utilization of all the CPU cores.

 It stores all the IP addresses banned by Fail2ban in the database and if one IP has been
 banned more than once, it includes the 'whois' data for that IP in the email report. It searches
 for the 'abuse' email address in the 'whois' data and writes a draft email that can be manually sent
 by the admin to the entity that owns the offending IP, to report the repeated attacks. The draft email
 includes the log lines containing the failed log in attepts of that IP. The text of the draft email is
 included in the periodic email report sent to the admin, below the 'whois' data of each IP that was
 banned more than once. If the 'automatic_emails_to_isp' parameter in the 'config' file is set to 'yes',
 the program will automatically send the abuse report email to the entity that owns the offending IP,
 at the moment its number of bans increases by 1, if it has been banned at least once in the past.

 Before sending the email report to the admin, the program compares it with the last sent report, and if 
 there is nothing important or new to report, the new email will not be sent, since it is assumed that
 the admin has already been informed about the current problems in the previous email. This is to avoid
 flooding the admin with too many emails. For example, if at a particular moment it detects that the only
 problem of the server is that there are 75 banned IPs, and at the moment of the next run it detects that
 4 of those IPs have been debaned due to their bantime being over, it won't send the email report, because
 the debanning of the 4 IPs is not important enough to deserve a new email. However, if it detects that
 since the last run a new IP has been banned, the email report will be sent.


REQUIREMENTS:

 This program is designed to be used as part of 'RED SCARF Suite'. It may work in other contexts but
 only if heavily modified. It requires at least an abridged version of RED SCARF Suite and Postfix, 
 installed and configured on a Debian server as described in the "Complete Guide to a Complete Linux 
 Server". At least the following jails need to be set in '/etc/fail2ban/jail.local':

    sshd, postfix, postfix-sasl

 In general, the log files written by the programs installed on the server should be rotated only when 
 they reach about 2M in size (with the exception of WordPress websites, whose logs need to be rotated 
 once every day because of Matomo), which can be configured in their respective logrotate files located 
 in the '/etc/logrotate.d' directory. This is because 'System Health and Security Probe' needs to read 
 a large number of log lines in order to find enough failed log in attempts that can be included in the 
 draft abuse report emails.

 Important !!! The ban time set for all the jails in '/etc/fail2ban/jail.local' (in seconds) must be 
    greater than the interval between two consecutive runs of the program, that you set in the 
    'shsp-config.php' file in the '$time_in_hours' parameter. If the run interval is set to 8 hours,
    which is the default, all the jails in Fail2ban must have a ban time greater than 8 hours (which
    means greater than 28800 seconds). Also, in '/etc/fail2ban/jail.local' it's necessary to specify a 
    ban time (in seconds) for each and every jail, even for the jails who use the default ban time, for 
    which it's not required. So, in every jail block, a 'bantime' parameter should be included.

 Important !!! If you change the run interval in the 'shsp-config.php' file (in the '$time_in_hours' 
    parameter), you should change the cron job accordingly. To run the script every 8 hours, at 5 minutes 
    past, the cron job should be:

    5 */8 * * * php /srv/scripts/shas-probe/system-health-and-security-probe.php > /dev/null 2>&1


INSTALLATION:

 Once the applications that make up RED SCARF Suite are installed (at least the 10 kernel components of 
 the suite and Postfix), to start using this program just copy all its files and directories to 
 '/srv/scripts/shas-probe' (or in another suitable location of your choice), then create a MariaDB database, 
 user and password, give the user all the priviledges over that database, except GRANT, then copy the 
 database name, user and password in the corresponding parameters of the 'shsp-config.php' file. Also, set 
 the value of all the other parameters in the 'shsp-config.php' file.

 Add a cronjob to crontab (use the 'crontab -e' command) like the following:

    # Run System Health and Security Probe every 8 hours
    5 */8 * * * php /srv/scripts/shas-probe/system-health-and-security-probe.php > /dev/null 2>&1

 Next, create the directory to store the periodic ClamAV scan reports. The default name and location for this
 directory is '/srv/scripts/detections' but you can change it and mention it as such in the 'shsp-config.php' 
 file. Then set up two cronjobs to get ClamAV to scan two sensitive directories. Keep the file names 
 'clamav_nextcloud_report' and 'clamav_mail_report' unchanged, as shown below. You can change the directory 
 for these files ('/srv/scripts/detections') and the time and frequency of scanning, according to your needs.
 Replace 'example.com' with your domain:

    # Scan the '/var/www/cloud.example.com/data' directory and the '/var/vmail' directory with ClamAV every three days
    5 3 */3 * * cat /dev/null > /srv/scripts/detections/clamav_nextcloud_report && clamdscan --fdpass --quiet /var/www/cloud.example.com/data -l /srv/scripts/detections/clamav_nextcloud_report
    40 3 */3 * * cat /dev/null > /srv/scripts/detections/clamav_mail_report && clamdscan --fdpass --quiet /var/vmail -l /srv/scripts/detections/clamav_mail_report
 
 That's it, the program is ready to fulfill its mission. To be able to see the email reports properly, your email client should accept html emails.


LICENCE:
 GNU GENERAL PUBLIC LICENSE v3+

