This is simply a record of what work was performed on ALBERT, in case similar work will be applied to other VMs.

1. Updates and install:yum update; yum install httpd php
2. Configure firewall to permit connections in:

firewall-cmd --add-service=http --permanent
firewall-cmd --add-service=https --permanent
firewall-cmd --reload firewall-cmd --list-services

systemctl start httpd.service

Test with: http://albert.tombstones.org.uk/

<blockquote>

This should be done to accept any requests resolving to albert that don't match a vhost header name, e.g.: an incorrect DNS entry or someone sniffing port 80 with no "host-header" attribute. Generally it keeps the logfiles of real websites free from other activity not intended for it.

</blockquote>

mkdir /etc/httpd/vhosts.d

echo "ServerName albert"> /etc/httpd/conf.d/servername.conf
echo "Include vhosts.d/*.conf"> /etc/httpd/conf.d/vhosts.conf

However, vhost.conf contains:

## ================ new stuff added:
include vhosts.d/.defaults.d/*.conf
include vhosts.d/dave.d/control.conf
#include vhosts.d/sairuk.d/all-sites.conf
#include vhosts.d/trac.d/all-sites.conf
#Include vhosts.d/utadmin.d/all-sites.conf

.. the idea is that subdirectories in each vhosts.d/ area can be controlled with control.conf that determines which to be added/removed.

Add a default site to /etc/httpd/vhosts.d/a-default.conf:

##
## Default blackhole/null vhost for IP-address sniffers
## - also traps those vhosts not properly matching a SERVERNAME/SERVERALIAS
##
<VirtualHost *:80>

  ServerName 78.129.208.174
    ServerAdmin webmaster@localhost.localdomain

    DocumentRoot /home/dave/websites/default/htdocs
    ErrorLog /home/dave/websites/default/logs/error.log
    CustomLog /home/dave/websites/default/logs/access.log combined
    LogLevel info

    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>

    <Directory /home/dave/websites/default/htdocs/>
        AllowOverride FileInfo Limit
        Include include/directoryindex.inc
        Include include/allow-public.inc
    </Directory>

</VirtualHost>

Configure SELinux to permit those directories the right context:

semanage fcontext -a -t httpd_sys_content_t  "/home/dave/websites/albert(/.*)?"
semanage fcontext -a -t httpd_log_t            "/home/dave/websites/albert/logs(/.*)?"

chgrp -R apache /home/dave/websites/albert/logs
chmod -R g+w  /home/dave/websites/albert/logs

restorecon -Rv /home/dave/websites

ls -lZ /home/dave/websites/default/

Note: this is to provide a default blackhole for IP sniffers and scanners so that (a) F2B can monitor those logfiles, and (b) website logfiles are free of casual intrusion attempts aimed at the server and not directed

yum install fail2ban

<blockquote>

Note: F2B has become "refactored hell", meaning that most settings have been moved into " paths-common.conf " containing global (distro-agnostic) default settings, overridden by " paths-fedora.conf " (CentOS-specific differences) but customisations should be added to " paths-overrides.local " rather than amending these files. This introduces a number of potential issues:

</blockquote>

1. knowing where to place customisations: the philosophy is that these should be added to new files (rather than existing) so that they are outside the scope of change impact during distro updates
2. knowing what the final configuration actually is: several global configurations are overridden by the .d/ files which themselves have *common and *.local files influencing behaviour. To ascetain exactly what Fail2Ban believes is the current configuration, run fail2ban-server -d or fail2ban-server –dp.
3. action at a distance: the strong dependency chain means that a single change is potentially inherited across multiple configurations, increasing risk of regression during amendments.

Add the following defaults to a [DEFAULT]section in /etc/fail2ban/jail.d/ (note: this directory contains default jail configurations as well as jails for services, so is a mixture of generic and specific settings)

• bantime = 60m to 10-bantime.conf
• findtime = 1h to 10-findtime.conf
• ignoreip = 127.0.0.1/8 ::1 81.187.254.92 142.4.214.142 78.129.208.24 added to 10-ignoreip.conf
• maxretry = 3 to 10-maxretry.conf

<blockquote>

Note: it's possible to add all of the above into one config file (or even into a .local file) but the over-segregation was intended so that each filename was self-describing and could easily be disabled by renaming.

</blockquote>

• added 20-email.conf containing email notification directives and default action for jails:

## -- this will be changed by canonical mapping in postfix
sender = root@albert.server
## -- destination - will be picked up by /etc/aliases
destemail = root

# -- changed MTA from "sendmail" to "mail".
# -- means it uses mail-whois-lines not sendmail-whois-lines
mta = mail

## -- the jail name is set to the name by default
## use "jailname = Annoying Portscan Sniffer" in a jail to change the name
## .. just for emails, not for chains, etc.
jailname = %(__name__)s

# ban & send an e-mail with whois report and relevant log lines
# to the destemail, but with a Jail name for forwarding
action_mwlj = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
  %(mta)s-whois-lines[name=%(jailname)s, sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]

• added 50-ssh-jail.conf to this directory (first jail) containing:

## ------------- ssh sniffers ---------------
[sshd]

# To use more aggressive sshd modes set filter parameter "mode" in jail.local:
# normal (default), ddos, extra or aggressive (combines all).
# See "tests/files/logs/sshd" or "filter.d/sshd.conf" for usage example and details.
#mode  = normal

enabled = true
## -- this is a different port now
port    = 22022
logpath = %(sshd_log)s
backend = %(sshd_backend)s

## -- just for testing
#bantime  = 1m

## -- 24-hour bantime... see if reports work
bantime = 24h

## -- this is used for emails instead of <name>
## -- if "jailname" is missing, defaults to <name>
jailname = SSH Sniffer

## -- email, whois, logs... and a custom subject line.
action =%(action_mwlj)s

[selinux-ssh]

enabled = false
port     = ssh
logpath  = %(auditd_log)s


</file>
  * added **60-apache-jail.conf** for the apache sniffers: (note: this needs updating...)

<code>#
# HTTP servers
#

[DEFAULT]
## -- these are different paths here
## -- sadly, even though it takes these different locations...
## F2B and SEinux don't play well together.
bitbucket_error_log = /home/dave/websites/default/logs/error.log
bitbucket_access_log = /home/dave/websites/default/logs/access.log

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-botsearch]
## -- this is used for phpmyadmin sniffers and the like
##
port     = http,https

## -- works: it's whitelisted
#logpath  = %(apache_error_log)s

## -- let's see if a different location works
#logpath  = %(bitbucket_error_log)s
logpath  = /home/dave/websites/default/logs/error.log
## -- issues with sodding python2 and SELinux
## *** seems you can't have a different path to the error log.
## .. .now you can, thanks to Maarten's SELinux policy fix

bantime = 48h

## -- to enable and send notifications
jailname = Apache Sniffer
action =%(action_mwlj)s

enabled = true

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-badbots]
# Ban hosts which agent identifies spammer robots crawling the web
# for email addresses. The mail outputs are buffered.
port     = http,https
logpath  = %(apache_access_log)s
bantime  = 48h
maxretry = 1

jailname = Apache Bot Checker
action =%(action_mwlj)s
enabled = true

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-shellshock]

port    = http,https
logpath = %(apache_error_log)s
maxretry = 1

jailname = Apache Shellshock scanner
action =%(action_mwlj)s
#enabled = true

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-auth]

port     = http,https
logpath  = %(apache_error_log)s

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-noscript]

port     = http,https
logpath  = %(apache_error_log)s

[apache-overflows]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-nohome]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-fakegooglebot]

port     = http,https
logpath  = %(apache_access_log)s
maxretry = 1
ignorecommand = %(ignorecommands_dir)s/apache-fakegooglebot <ip>

[apache-modsecurity]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

Add blocktype = DROP in firewallcmd-common.conf, meaning that the default action will to be adding a DROP rule.

Test with SSH connections to an unnamed account (ssh nothing@albert) and watch logfiles: /var/log/secure and /var/log/fail2ban.log

Some other work done:

Add the following to /usr/local/bin/server_timezone - this is simply to help with emailed reports

#!/bin/sh
##
## -- this is used simply because Fail2Ban has issues with any command containing a percent
echo UTC$(/bin/date +%z)

Add the following to /etc/fail2ban/action.d/mail-whois-common.local:

[DEFAULT]
## change this to be the more verbose version
## however, not using it at the moment
#_whois = whois.arin <ip> || echo "missing whois program"

## -- seems F2B can't cope with an embedded % in the command
#_test_var = `whoami`
#_test_var_new = `(date +%z)`
#_server_timezone = `date +%z`
##
## -- none of those worked...
##

## references /usr/local/bin/server_timezone
_server_timezone = `server_timezone`

The /etc/fail2ban/action.d/mail-whois-lines.conf action differs to include a more information, including details of IP ownership for reporting:

# Fail2Ban configuration file
#
# Author: Cyril Jaquier
# Modified-By: Dave - altered subject and included whois details.
#
[INCLUDES]

before = mail-whois-common.conf
   helpers-common.conf

[Definition]

# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
#
actionstart = printf %%b "Jail \"<name>\" started.\n
               Using logfile: <logpath>\n
               "|mail -s "[Fail2Ban] <name>: started on <fq-hostname>" <dest>

# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
#
actionstop = printf %%b "Jail \"<name>\" has stopped.
        "|mail -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest>

# bypass ban/unban for restored tickets
norestored = 1

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionban = printf %%b "============= Fail2Ban report ====================\n
            Server Timezone: %(_server_timezone)s
            IP Address blocked: <ip>
            Number of attempts: <failures>
            Jail name: '<name>'
            \n
            +++++++++++++ logfile analysis ++++++++++++++++++++\n
            `%(_grep_logs)s`\n\n
            ------------- WHOIS information -------------------\n
            WHOIS information about <ip> is:\n
            `%(_whois_command)s`\n\n
            "|mail -s "[Fail2Ban] detected <ip> running <name>" <dest>

# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionunban =

[Init]

# Default name of the chain
#
name = default

# Path to the log files which contain relevant lines for the abuser IP
#
logpath = /dev/null

# Number of log lines to include in the email
#
grepmax = 1000
greplimit = tail -n 20
#grepopts = -m 1000 -a
grepopts = -m <grepmax>


</file>

For this action to be used, we define a new action in **/etc/fail2ban/jail.d/20-email.conf**
<code>


[DEFAULT]

# -- sorts out sender and destination email addresses

## -- this will be changed by canonical mapping in postfix
sender = root@albert.server

## -- destination - will be picked up by /etc/aliases
destemail = root

# -- changed it from "sendmail" to "mail".
# -- means it uses mail-whois-lines not sendmail-whois-lines

mta = mail

## -- the jail name is set to the name by default
jailname = %(__name__)s

## use "jailname = Annoying Portscan Sniffer" in a jail to change the name
## .. just for emails, not for chains, etc.

# ban & send an e-mail with whois report and relevant log lines
# to the destemail, but with a Jail name for forwarding

action_mwlj = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
   %(mta)s-whois-lines[name=%(jailname)s, sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]

This now means that specific jails can make use of a different action (action_mwlj = Action, mail, whois, log, jailname) and also specify a custom name in the mail:

## ------------- ssh sniffers ---------------

[sshd]

# To use more aggressive sshd modes set filter parameter "mode" in jail.local:
# normal (default), ddos, extra or aggressive (combines all).
# See "tests/files/logs/sshd" or "filter.d/sshd.conf" for usage example and details.
#mode   = normal

enabled = true
## -- this is a different port now
port    = 22022
logpath = %(sshd_log)s
backend = %(sshd_backend)s

## -- just for testing
#bantime  = 1m

## -- 24-hour bantime... see if reports work
bantime = 24h

## -- this is used for emails instead of <name>
## -- if "jailname" is missing, defaults to <name>
jailname = SSH Sniffer

## -- email, whois, logs... and a custom subject line.
action = %(action_mwlj)s

Note: when testing, to manually remove an IP that's been banned use: fail2ban-client set sshd unbanip IP.ADD.RS.HERE

Also added some work to blocking apache sniffers on ALBERT (probably not needed on HAL):

[Definition]

# Webroot represents the webroot on which all other files are based
webroot = /var/www/html/

Note: SELinux prevented F2B reading logfiles stored in non-default locations (specifically webserver logs stored outside of /var/log/httpd) thanks to SELinux's policy on Python. Maarten created a new SELinux policy as a workaround:

module custom_fail2ban 1.1.0;

require {
  type fail2ban_t;
        type user_home_t;
        type user_home_dir_t;
        class dir { search };
}

#============= fail2ban_t ==============
allow fail2ban_t user_home_dir_t:dir { search };
allow fail2ban_t user_home_t:dir { search }; <font inherit/inherit;;inherit;;inherit></font>

Compile this Type Enforcing file into a Policy module: checkmodule -M -m -o custom.mod custom.te

Then build a Policy Package from this module: semodule_package -o custom.pp -m custom.mod

Then install this package: semodule -i custom.pp

#
# HTTP servers
#

[DEFAULT]
## -- these are different paths here
## -- sadly, even though it takes these different locations...
## F2B and SEinux don't play well together.
bitbucket_error_log = /home/dave/websites/default/logs/error.log
bitbucket_access_log = /home/dave/websites/default/logs/access.log

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-botsearch]
## -- this is used for phpmyadmin sniffers and the like
##
port     = http,https

## -- works: it's whitelisted
logpath  = %(apache_error_log)s

## -- let's see if a different location works
#logpath  = %(bitbucket_error_log)s
#logpath  = /home/dave/websites/default/logs/error.log
## -- issues with sodding python2 and SELinux
## *** seems you can't have a different path to the error log.

bantime = 48h

## -- to enable and send notifications
jailname = Apache Sniffer
action =%(action_mwlj)s

enabled = true

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-badbots]
# Ban hosts which agent identifies spammer robots crawling the web
# for email addresses. The mail outputs are buffered.
port     = http,https
logpath  = %(apache_access_log)s
bantime  = 48h
maxretry = 1

jailname = Apache Bot Checker
action =%(action_mwlj)s
enabled = true

yum install mysql mariadb-server php-mysql

Start the service then set the root password:

SET PASSWORD FOR 'root'@'localhost' = PASSWORD('SomePassHere');

Create another account with DBA privileges:

CREATE USER dave_dba'@'localhost' IDENTIFIED BY '!NewPass!';
GRANT ALL PRIVILEGES ON *.* TO 'dave_dba'@'localhost' WITH GRANT OPTION;

Add this info to a .my.cnf file in the home directory if needed:

[client]
host=localhost
user=dave_dba
password=!NewPass!

The challenge here is running a MTA that only listens on localhost to forward mail off - not to accept mail. The following rules should apply:

• mails sent FROM an account on albert (e.g.: fred) should originated from a real FQDN (fred.on.albert@gmail.com) - use sender_canonical_maps
• mails sent TO an account on albert (e.g.: fred) should be forwarded to an off-server address (fred.albert@tombstones.org.uk) - use /etc/aliases

This ensures that:

• albert accepts no mails from outside
• albert can send mails off the server
• mails sent to local accounts are forward off.

Solution: in /etc/postfix/main.cf, set the following:

1. configure a myhostname setting of a non-deliverable domain, e.g.: myhostname = albert.server - this ensures mails sent from a local login of fred appear to originate from fred@albert.server
2. set mydomain = $myhostname and myorigin = $myhostname (else HELO will use albert.albert.server or albert.locahost)
3. add albert.server to mydestination so that locally-sent mails (from fred@albert.server) are "accepted" by postfix
4. add sender_canonical_maps = hash:/etc/postfix/outbound.sender_canonical
5. configure outbound.sender_canonical to map fred@albert.server to fred.on.albert@gmail.com

- this ensures that the mail is sent from a "real" domain

1. any addresses not added to outbound.sender_canonical will not be rewritten (and thus identify them as fake)
2. run "postmap outbound.sender_canonical" to rebuild the DBM hash when changes are made

Lastly, to ensure any mails sent TO the local account are forwarded off, add entries to /etc/aliases in the form:

root:  security@tombstones.org.uk

(don't forget to run "newaliases" to rebuild aliases.db when changing)

To test:

• mail -s"root sending to fred locally" fred < /etc/hosts
• mail -s"root sending off the server" fred@gmail.net < /etc/hosts

Note: the sender alias appearing in the "From" mail header is the comment field in /etc/passwd: usermod -c "Fred Blogs on Albert" fred.

Install - no service to configure. Add new settings to /etc/logwatch/conf/logwatch.conf:

## -- this is transformed into logwatch@albert.server -> albert-logwatch@tombstones
MailFrom = Logwatch
Detail = 3

Note: minor change to the perl script just to show "[Logwatch]" in the subject line.

Took quite a bit of effort:

1. apache + php + mod_ssl + php-process

2. create 2 vhosts (oe being default), don't forget SELinux

3. Fix permissions on /var/lib/letsencrypt/, install certbot (install snapd first)

4. run certbot to build initial cert

## renew the (already-installed) certificate for albert
certbot renew --cert-name albert.tombstones.org.uk

## force the renewal if it's not yet expired
certbot renew --cert-name albert.tombstones.org.uk --force-renewal

To perform post-processing, add a script to /etc/letsencrypt/renewal-hooks/ (then check it works!) and pass the path to the "–manual-cleanup-hook" option, e.g.:

certbot renew --cert-name albert.tombstones.org.uk --manual-cleanup-hook post/example-script.sh

(nb: path is relative to /etc/letsencrypt/renewal-hooks/)

See: https://certbot.eff.org/docs/using.html

To restore the puppet configs, the following steps were taken:

1. rebuild configs.tombstones Gitea
2. add a "deploy" account to this site
3. create a API token to the "deploy" account
4. (re)create repos on Gitea
5. push existing puppet manifests back to the repo to keep it in synch

For the last part, kickstarts / common / post.cfg performs the following…

1. create a separate set of root keys, using:

 ssh-keygen -q -b 4096 -t rsa -f /root/.ssh/id_rsa_deploy -N "" -C"deploy@$(hostname -s)"

2. Add an entry to /root/.ssh/config:

Host deploy
  Hostname config.tombstones.org.uk
  Port 22022
  StrictHostKeyChecking no
  IdentityFile /root/.ssh/id_rsa_deploy

3. Register these keys with the "deploy" account:

curl -X POST "https://config.tombstones.org.uk:23000/api/v1/user/keys" \
-k \
-H "accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: token 2b2182bbbb7e52b3193c4c9718c6e96c372f8156" \
-d "{  \"key\": \"$(cat /root/.ssh/id_rsa_deploy.pub)\",  \"read_only\": true,  \"title\": \"$(hostname -s)-deploy-$(date +'%s')\"}"

4. This means that puppet can check out a repo with:

git clone git@deploy:tombstones/puppet-common.git

From there, git on each VM needs to be instructed to switch to this repo instead. To examine the URLs, use:

tupper:/var/lib/puppet/manifests/puppet-content ## git remote -v
origin    git@deploy:tombstones/puppet-content.git (fetch)
origin    git@deploy:tombstones/puppet-content.git (push)

To switch a URL over, use git remote set-url:

tupper:/root/puppet-manifests/puppet-common ## git remote -v
origin  ssh://git@config.tombstones.org.uk:22022/tombstones/puppet-common.git (fetch)
origin  ssh://git@config.tombstones.org.uk:22022/tombstones/puppet-common.git (push)

tupper:/root/puppet-manifests/puppet-common ## git remote set-url origin ssh://git@deploy:tombstones/puppet-common.git
tupper:/root/puppet-manifests/puppet-common ## git remote -v
origin    ssh://git@deploy:tombstones/puppet-common.git (fetch)
origin    ssh://git@deploy:tombstones/puppet-common.git (push)

Using the .ssh/config aliases, the URL is much shorter (but will make use of the "config" aliases, which may need updating). However, this also means that the keys have been registered for FETCH (read-only) operations; it cannot be used to PUSH updates back to gitea.