====== Work Done on Albert ======

===== Precis =====

This is simply a record of what work was performed on ALBERT, in case similar work will be applied to other VMs.

===== A. Adding Apache =====

  - Updates and install:''yum update; yum install httpd php''
  - Configure firewall to permit connections in:

<code>
firewall-cmd --add-service=http --permanent
firewall-cmd --add-service=https --permanent
firewall-cmd --reload firewall-cmd --list-services

systemctl start httpd.service

</code>

Test with: [[http://albert.tombstones.org.uk/|http://albert.tombstones.org.uk/]]

===== B. Configuring first vhost =====

<blockquote>

This should be done to accept any requests resolving to **albert **that don't match a vhost header name, e.g.: an incorrect DNS entry or someone sniffing port 80 with no "host-header" attribute. Generally it keeps the logfiles of real websites free from other activity not intended for it.

</blockquote>
<code>

mkdir /etc/httpd/vhosts.d

echo "ServerName albert"> /etc/httpd/conf.d/servername.conf
echo "Include vhosts.d/*.conf"> /etc/httpd/conf.d/vhosts.conf

</code>

However, vhost.conf contains:

<file>
## ================ new stuff added:
include vhosts.d/.defaults.d/*.conf
include vhosts.d/dave.d/control.conf
#include vhosts.d/sairuk.d/all-sites.conf
#include vhosts.d/trac.d/all-sites.conf
#Include vhosts.d/utadmin.d/all-sites.conf

</file>

.. the idea is that subdirectories in each vhosts.d/ area can be controlled with control.conf that determines which to be added/removed.

Add a default site to ''/etc/httpd/vhosts.d/a-default.conf'':

<code>
##
## Default blackhole/null vhost for IP-address sniffers
## - also traps those vhosts not properly matching a SERVERNAME/SERVERALIAS
##
<VirtualHost *:80>

  ServerName 78.129.208.174
    ServerAdmin webmaster@localhost.localdomain

    DocumentRoot /home/dave/websites/default/htdocs
    ErrorLog /home/dave/websites/default/logs/error.log
    CustomLog /home/dave/websites/default/logs/access.log combined
    LogLevel info

    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>

    <Directory /home/dave/websites/default/htdocs/>
        AllowOverride FileInfo Limit
        Include include/directoryindex.inc
        Include include/allow-public.inc
    </Directory>

</VirtualHost>

</code>

Configure SELinux to permit those directories the right context:

<code>
semanage fcontext -a -t httpd_sys_content_t  "/home/dave/websites/albert(/.*)?"
semanage fcontext -a -t httpd_log_t            "/home/dave/websites/albert/logs(/.*)?"

chgrp -R apache /home/dave/websites/albert/logs
chmod -R g+w  /home/dave/websites/albert/logs

restorecon -Rv /home/dave/websites

ls -lZ /home/dave/websites/default/

</code>

Note: this is to provide a default blackhole for IP sniffers and scanners so that (a) F2B can monitor those logfiles, and (b) website logfiles are free of casual intrusion attempts aimed at the server and not directed

===== C. Configure Fail2Ban =====

''yum install fail2ban''

<blockquote>

Note: F2B has become "refactored hell", meaning that most settings have been moved into "** paths-common.conf**  " containing global (distro-agnostic) default settings, overridden by "** paths-fedora.conf**  " (CentOS-specific differences) but customisations should be added to "** paths-overrides.local**  " rather than amending these files. This introduces a number of potential issues:

</blockquote>

  - knowing where to place customisations: the philosophy is that these should be added to new files (rather than existing) so that they are outside the scope of change impact during distro updates
  - knowing what the final configuration actually is: several global configurations are overridden by the **.d/**  files which themselves have ***common**  and ***.local**  files influencing behaviour. To ascetain exactly what Fail2Ban believes is the current configuration, run'' fail2ban-server -d''  or ''fail2ban-server –dp.''
  - action at a distance: the strong dependency chain means that a single change is potentially inherited across multiple configurations, increasing risk of regression during amendments.

Add the following defaults to a'' [DEFAULT]''section in **/etc/fail2ban/jail.d/ **  (note: this directory contains default jail configurations as well as jails for services, so is a mixture of generic and specific settings)

  * ''bantime = 60m''  to **10-bantime.conf**
  * ''findtime = 1h ''to **10-findtime.conf**
  * ''ignoreip = 127.0.0.1/8 ::1 81.187.254.92 142.4.214.142 78.129.208.24''  added to **10-ignoreip.conf**
  * ''maxretry = 3''  to **10-maxretry.conf**
<blockquote>

Note: it's possible to add all of the above into one config file (or even into a .local file) but the over-segregation was intended so that each filename was self-describing and could easily be disabled by renaming.

</blockquote>

  * added **20-email.conf**  containing email notification directives and default action for jails:

<code>
## -- this will be changed by canonical mapping in postfix
sender = root@albert.server
## -- destination - will be picked up by /etc/aliases
destemail = root

# -- changed MTA from "sendmail" to "mail".
# -- means it uses mail-whois-lines not sendmail-whois-lines
mta = mail

## -- the jail name is set to the name by default
## use "jailname = Annoying Portscan Sniffer" in a jail to change the name
## .. just for emails, not for chains, etc.
jailname = %(__name__)s

# ban & send an e-mail with whois report and relevant log lines
# to the destemail, but with a Jail name for forwarding
action_mwlj = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
  %(mta)s-whois-lines[name=%(jailname)s, sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]

</code>

  * added **50-ssh-jail.conf**  to this directory** (first jail) **containing:

<code>
## ------------- ssh sniffers ---------------
[sshd]

# To use more aggressive sshd modes set filter parameter "mode" in jail.local:
# normal (default), ddos, extra or aggressive (combines all).
# See "tests/files/logs/sshd" or "filter.d/sshd.conf" for usage example and details.
#mode  = normal

enabled = true
## -- this is a different port now
port    = 22022
logpath = %(sshd_log)s
backend = %(sshd_backend)s

## -- just for testing
#bantime  = 1m

## -- 24-hour bantime... see if reports work
bantime = 24h

## -- this is used for emails instead of <name>
## -- if "jailname" is missing, defaults to <name>
jailname = SSH Sniffer

## -- email, whois, logs... and a custom subject line.
action =%(action_mwlj)s

[selinux-ssh]

enabled = false
port     = ssh
logpath  = %(auditd_log)s


</file>
  * added **60-apache-jail.conf** for the apache sniffers: (note: this needs updating...)

<code>#
# HTTP servers
#

[DEFAULT]
## -- these are different paths here
## -- sadly, even though it takes these different locations...
## F2B and SEinux don't play well together.
bitbucket_error_log = /home/dave/websites/default/logs/error.log
bitbucket_access_log = /home/dave/websites/default/logs/access.log

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-botsearch]
## -- this is used for phpmyadmin sniffers and the like
##
port     = http,https

## -- works: it's whitelisted
#logpath  = %(apache_error_log)s

## -- let's see if a different location works
#logpath  = %(bitbucket_error_log)s
logpath  = /home/dave/websites/default/logs/error.log
## -- issues with sodding python2 and SELinux
## *** seems you can't have a different path to the error log.
## .. .now you can, thanks to Maarten's SELinux policy fix

bantime = 48h

## -- to enable and send notifications
jailname = Apache Sniffer
action =%(action_mwlj)s

enabled = true

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-badbots]
# Ban hosts which agent identifies spammer robots crawling the web
# for email addresses. The mail outputs are buffered.
port     = http,https
logpath  = %(apache_access_log)s
bantime  = 48h
maxretry = 1

jailname = Apache Bot Checker
action =%(action_mwlj)s
enabled = true

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-shellshock]

port    = http,https
logpath = %(apache_error_log)s
maxretry = 1

jailname = Apache Shellshock scanner
action =%(action_mwlj)s
#enabled = true

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-auth]

port     = http,https
logpath  = %(apache_error_log)s

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-noscript]

port     = http,https
logpath  = %(apache_error_log)s

[apache-overflows]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-nohome]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-fakegooglebot]

port     = http,https
logpath  = %(apache_access_log)s
maxretry = 1
ignorecommand = %(ignorecommands_dir)s/apache-fakegooglebot <ip>

[apache-modsecurity]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

</code>

Add ''blocktype = DROP''  in **firewallcmd-common.conf**, meaning that the default action will to be adding a DROP rule.

Test with SSH connections to an unnamed account (''ssh nothing@albert'') and watch logfiles: **/var/log/secure**  and **/var/log/fail2ban.log**

Some other work done:

Add the following to **/usr/local/bin/server_timezone**  - this is simply to help with emailed reports

<code>
#!/bin/sh
##
## -- this is used simply because Fail2Ban has issues with any command containing a percent
echo UTC$(/bin/date +%z)

</code>

==== Action: more verbose emailed reports ====

Add the following to **/etc/fail2ban/action.d/mail-whois-common.local**:

<code>
[DEFAULT]
## change this to be the more verbose version
## however, not using it at the moment
#_whois = whois.arin <ip> || echo "missing whois program"

## -- seems F2B can't cope with an embedded % in the command
#_test_var = `whoami`
#_test_var_new = `(date +%z)`
#_server_timezone = `date +%z`
##
## -- none of those worked...
##

## references /usr/local/bin/server_timezone
_server_timezone = `server_timezone`

</code>

The **/etc/fail2ban/action.d/mail-whois-lines.conf**  action differs to include a more information, including details of IP ownership for reporting:
<code>

# Fail2Ban configuration file
#
# Author: Cyril Jaquier
# Modified-By: Dave - altered subject and included whois details.
#
[INCLUDES]

before = mail-whois-common.conf
   helpers-common.conf

[Definition]

# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
#
actionstart = printf %%b "Jail \"<name>\" started.\n
               Using logfile: <logpath>\n
               "|mail -s "[Fail2Ban] <name>: started on <fq-hostname>" <dest>

# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
#
actionstop = printf %%b "Jail \"<name>\" has stopped.
        "|mail -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest>

# bypass ban/unban for restored tickets
norestored = 1

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionban = printf %%b "============= Fail2Ban report ====================\n
            Server Timezone: %(_server_timezone)s
            IP Address blocked: <ip>
            Number of attempts: <failures>
            Jail name: '<name>'
            \n
            +++++++++++++ logfile analysis ++++++++++++++++++++\n
            `%(_grep_logs)s`\n\n
            ------------- WHOIS information -------------------\n
            WHOIS information about <ip> is:\n
            `%(_whois_command)s`\n\n
            "|mail -s "[Fail2Ban] detected <ip> running <name>" <dest>

# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionunban =

[Init]

# Default name of the chain
#
name = default

# Path to the log files which contain relevant lines for the abuser IP
#
logpath = /dev/null

# Number of log lines to include in the email
#
grepmax = 1000
greplimit = tail -n 20
#grepopts = -m 1000 -a
grepopts = -m <grepmax>


</file>

For this action to be used, we define a new action in **/etc/fail2ban/jail.d/20-email.conf**
<code>


[DEFAULT]

# -- sorts out sender and destination email addresses

## -- this will be changed by canonical mapping in postfix
sender = root@albert.server

## -- destination - will be picked up by /etc/aliases
destemail = root

# -- changed it from "sendmail" to "mail".
# -- means it uses mail-whois-lines not sendmail-whois-lines

mta = mail

## -- the jail name is set to the name by default
jailname = %(__name__)s

## use "jailname = Annoying Portscan Sniffer" in a jail to change the name
## .. just for emails, not for chains, etc.

# ban & send an e-mail with whois report and relevant log lines
# to the destemail, but with a Jail name for forwarding

action_mwlj = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
   %(mta)s-whois-lines[name=%(jailname)s, sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]

</code>

This now means that specific jails can make use of a different action (**action_mwlj**  = Action, mail, whois, log, jailname) and also specify a custom name in the mail:

==== /etc/fail2ban/jail.d/50-ssh-jail.conf ====

<code>
## ------------- ssh sniffers ---------------

[sshd]

# To use more aggressive sshd modes set filter parameter "mode" in jail.local:
# normal (default), ddos, extra or aggressive (combines all).
# See "tests/files/logs/sshd" or "filter.d/sshd.conf" for usage example and details.
#mode   = normal

enabled = true
## -- this is a different port now
port    = 22022
logpath = %(sshd_log)s
backend = %(sshd_backend)s

## -- just for testing
#bantime  = 1m

## -- 24-hour bantime... see if reports work
bantime = 24h

## -- this is used for emails instead of <name>
## -- if "jailname" is missing, defaults to <name>
jailname = SSH Sniffer

## -- email, whois, logs... and a custom subject line.
action = %(action_mwlj)s

</code>

Note: when testing, to manually remove an IP that's been banned use: ''fail2ban-client set sshd unbanip IP.ADD.RS.HERE''

Also added some work to blocking apache sniffers on ALBERT (probably not needed on HAL):

==== /etc/fail2ban/filter.d/apache-botsearch.local: ====

<file>
[Definition]

# Webroot represents the webroot on which all other files are based
webroot = /var/www/html/

</file>

Note: SELinux prevented F2B reading logfiles stored in non-default locations (specifically webserver logs stored outside of /var/log/httpd) thanks to SELinux's policy on Python. Maarten created a new SELinux policy as a workaround:

<file>
module custom_fail2ban 1.1.0;

require {
  type fail2ban_t;
        type user_home_t;
        type user_home_dir_t;
        class dir { search };
}

#============= fail2ban_t ==============
allow fail2ban_t user_home_dir_t:dir { search };
allow fail2ban_t user_home_t:dir { search }; <font inherit/inherit;;inherit;;inherit></font>

</file>

Compile this **Type Enforcing **file into a **Policy module**: ''checkmodule -M -m -o custom.mod custom.te''

Then build a **Policy Package **from this module: ''semodule_package -o custom.pp -m custom.mod''

Then install this package: ''semodule -i custom.pp''

==== /etc/fail2ban/jail.d/60-apache-jail.conf: ====

<code>
#
# HTTP servers
#

[DEFAULT]
## -- these are different paths here
## -- sadly, even though it takes these different locations...
## F2B and SEinux don't play well together.
bitbucket_error_log = /home/dave/websites/default/logs/error.log
bitbucket_access_log = /home/dave/websites/default/logs/access.log

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-botsearch]
## -- this is used for phpmyadmin sniffers and the like
##
port     = http,https

## -- works: it's whitelisted
logpath  = %(apache_error_log)s

## -- let's see if a different location works
#logpath  = %(bitbucket_error_log)s
#logpath  = /home/dave/websites/default/logs/error.log
## -- issues with sodding python2 and SELinux
## *** seems you can't have a different path to the error log.

bantime = 48h

## -- to enable and send notifications
jailname = Apache Sniffer
action =%(action_mwlj)s

enabled = true

## ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[apache-badbots]
# Ban hosts which agent identifies spammer robots crawling the web
# for email addresses. The mail outputs are buffered.
port     = http,https
logpath  = %(apache_access_log)s
bantime  = 48h
maxretry = 1

jailname = Apache Bot Checker
action =%(action_mwlj)s
enabled = true

</code>

===== D. Configure MySQL =====

''yum install mysql mariadb-server php-mysql''

Start the service then set the root password:

<code>
SET PASSWORD FOR 'root'@'localhost' = PASSWORD('SomePassHere');

</code>

Create another account with DBA privileges:

<code>
CREATE USER dave_dba'@'localhost' IDENTIFIED BY '!NewPass!';
GRANT ALL PRIVILEGES ON *.* TO 'dave_dba'@'localhost' WITH GRANT OPTION;

</code>

Add this info to a .my.cnf file in the home directory if needed:

<code>
[client]
host=localhost
user=dave_dba
password=!NewPass!

</code>

===== E. Configuring email (Postfix) =====

The challenge here is running a MTA that only listens on localhost to forward mail off - not to accept mail. The following rules should apply:

  * mails sent FROM an account on albert (e.g.: **fred**) should originated from a real FQDN (**fred.on.albert@gmail.com**) - use **sender_canonical_maps**
  * mails sent TO an account on albert (e.g.: fred) should be forwarded to an off-server address (**fred.albert@tombstones.org.uk**) - use **/etc/aliases**

This ensures that:

  * albert accepts no mails from outside
  * albert can send mails off the server
  * mails sent to local accounts are forward off.

Solution: in **/etc/postfix/main.cf**, set the following:

  - configure a **myhostname **setting of a non-deliverable domain, e.g.: ''myhostname = albert.server''  - this ensures mails sent from a local login of **fred**  appear to originate from **fred@albert.server**
  - set ''mydomain = $myhostname''  and'' myorigin = $myhostname''  (else HELO will use albert.albert.server or albert.locahost)
  - add **albert.server**  to mydestination so that locally-sent mails (from **fred@albert.server**) are "accepted" by postfix
  - add **sender_canonical_maps = [[https://trac.x0blr.com/#data-tracwysiwyg-link=hash:/etc/postfix/outbound.sender_canonical|hash:/etc/postfix/outbound.sender_canonical]]**
  - configure **outbound.sender_canonical**  to map **fred@albert.server**  to **fred.on.albert@gmail.com**

- this ensures that the mail is sent from a "real" domain

  - any addresses not added to **outbound.sender_canonical **will not be rewritten (and thus identify them as fake)
  - run "**postmap outbound.sender_canonical**" to rebuild the DBM hash when changes are made

Lastly, to ensure any mails sent TO the local account are forwarded off, add entries to /etc/aliases in the form:
<code>

root:  security@tombstones.org.uk

</code>

(don't forget to run "newaliases" to rebuild aliases.db when changing)

To test:

  * ''mail -s"root sending to fred locally" fred < /etc/hosts''
  * ''mail -s"root sending off the server" fred@gmail.net < /etc/hosts''
Note: the sender alias appearing in the "From" mail header is the comment field in **/etc/passwd**: **usermod -c "Fred Blogs on Albert" fred**.

===== F. Logwatch =====

Install - no service to configure. Add new settings to ''/etc/logwatch/conf/logwatch.conf:''

<code>
## -- this is transformed into logwatch@albert.server -> albert-logwatch@tombstones
MailFrom = Logwatch
Detail = 3

</code>

Note: minor change to the perl script just to show "[Logwatch]" in the subject line.

===== LetsEncrypt certs: =====

Took quite a bit of effort:

1. apache + php + mod_ssl + php-process

2. create 2 vhosts (oe being default), don't forget SELinux

3. Fix permissions on /var/lib/letsencrypt/, install certbot (install snapd first)

4. run certbot to build initial cert

==== To manage certificates: ====

<code>
## renew the (already-installed) certificate for albert
certbot renew --cert-name albert.tombstones.org.uk

## force the renewal if it's not yet expired
certbot renew --cert-name albert.tombstones.org.uk --force-renewal

</code>

To perform post-processing, add a script to'' /etc/letsencrypt/renewal-hooks/''  (then check it works!) and pass the path to the "–manual-cleanup-hook" option, e.g.:

<code>
certbot renew --cert-name albert.tombstones.org.uk --manual-cleanup-hook post/example-script.sh

</code>

(nb: path is relative to /etc/letsencrypt/renewal-hooks/)

See: [[https://certbot.eff.org/docs/using.html|https://certbot.eff.org/docs/using.html]]

===== Gitea work. =====

To restore the puppet configs, the following steps were taken:

  - rebuild configs.tombstones Gitea
  - add a "deploy" account to this site
  - create a API token to the "deploy" account
  - (re)create repos on Gitea
  - push existing puppet manifests back to the repo to keep it in synch

For the last part, **kickstarts / common / post.cfg **performs the following...

1. create a separate set of root keys, using:

<code>
 ssh-keygen -q -b 4096 -t rsa -f /root/.ssh/id_rsa_deploy -N "" -C"deploy@$(hostname -s)"

</code>

2. Add an entry to /root/.ssh/config:

<file>
Host deploy
  Hostname config.tombstones.org.uk
  Port 22022
  StrictHostKeyChecking no
  IdentityFile /root/.ssh/id_rsa_deploy

</file>

3. Register these keys with the "deploy" account:

<file>
curl -X POST "https://config.tombstones.org.uk:23000/api/v1/user/keys" \
-k \
-H "accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: token 2b2182bbbb7e52b3193c4c9718c6e96c372f8156" \
-d "{  \"key\": \"$(cat /root/.ssh/id_rsa_deploy.pub)\",  \"read_only\": true,  \"title\": \"$(hostname -s)-deploy-$(date +'%s')\"}"

</file>

4. This means that puppet can check out a repo with:

<code>
git clone git@deploy:tombstones/puppet-common.git

</code>

From there, git on each VM needs to be instructed to switch to this repo instead. To examine the URLs, use:

<code>
tupper:/var/lib/puppet/manifests/puppet-content ## git remote -v
origin    git@deploy:tombstones/puppet-content.git (fetch)
origin    git@deploy:tombstones/puppet-content.git (push)

</code>

To switch a URL over, use **git remote set-url**:

<code>
tupper:/root/puppet-manifests/puppet-common ## git remote -v
origin  ssh://git@config.tombstones.org.uk:22022/tombstones/puppet-common.git (fetch)
origin  ssh://git@config.tombstones.org.uk:22022/tombstones/puppet-common.git (push)

tupper:/root/puppet-manifests/puppet-common ## git remote set-url origin ssh://git@deploy:tombstones/puppet-common.git
tupper:/root/puppet-manifests/puppet-common ## git remote -v
origin    ssh://git@deploy:tombstones/puppet-common.git (fetch)
origin    ssh://git@deploy:tombstones/puppet-common.git (push)

</code>

Using the .ssh/config aliases, the URL is much shorter (but will make use of the "config" aliases, which may need updating). However, this also means that the keys have been registered for FETCH (read-only) operations; it cannot be used to PUSH updates back to gitea.