Puppet configuration

The following are steps in demystifying/reverse-engineering how puppet works on the VMs:

1. Puppet is installed and updates to the puppet config files (from gitea) is via crontab:

##
## Puppet removed from configs here as no longer used.

# HEADER: This file was autogenerated at 2021-03-15 05:48:52 +1100 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: repofix
*/5 * * * * /usr/local/bin/repofix &> /dev/null
# Puppet Name: puppet-apply-reboot
@reboot /opt/puppetlabs/bin/puppet apply /var/lib/puppet/manifests/puppet-common/00_puppet.pp &> /tmp/puppet-apply-reboot-priority.log && /opt/puppetlabs/bin/puppet apply /var/lib/puppet/manifests &> /tmp/puppet-apply-reboot.log
# Puppet Name: puppet-apply
*/15 * * * * /opt/puppetlabs/bin/puppet apply /var/lib/puppet/manifests &> /tmp/puppet-apply

There are three cronjobs:

1. puppet-apply-reboot - intended when there's a server reboot
2. repofix - pulls down files from gitea into /var/lib/puppet/manifests
3. puppet-apply, which calls puppet to apply new configs found in: /var/lib/puppet/manifests.

So the two regular ongoing jobs of interest are:

• repofix - keeps the server configs in synch with gitea (pulls them down, overwrites local files)
• puppet-apply - enact the instructions found in the puppet manifest files

Hence, to test a puppet config, add a puppet config to a directory then point the4 puppet command directly at that, e.g.:

puppet apply /root/test/new.pp

In /var/lib/puppet/manifests is the puppet-common directory, which contains the following files:

-rw-r--r--. 1 root root 2.4K Mar 15  2021 00_puppet.pp
-rw-r--r--. 1 root root  780 Mar 15  2021 01_packages.pp
-rw-r--r--. 1 root root  451 Mar 15  2021 02_rootpw.pp
-rw-r--r--. 1 root root  765 Mar 15  2021 03_fail2ban.pp
-rw-r--r--. 1 root root 1.8K Mar 15  2021 04_hardening.pp
-rw-r--r--. 1 root root  309 Mar 15  2021 06_pam.pp
-rw-r--r--. 1 root root  473 Mar 15  2021 07_hosts.pp
-rw-r--r--. 1 root root 2.2K Mar 15  2021 09_ssl.pp
-rw-r--r--. 1 root root 1.4K Mar 15  2021 10_email.pp
-rw-r--r--. 1 root root 3.2K Mar 15  2021 10_ssh.pp
-rw-r--r--. 1 root root  410 Mar 15  2021 20_iptables.pp
-rw-r--r--. 1 root root  284 Mar 15  2021 30_sudoers.pp
-rw-r--r--. 1 root root  526 Mar 15  2021 40_rkhunter.pp
-rw-r--r--. 1 root root  384 Mar 15  2021 60_firewalld.pp
-rw-r--r--. 1 root root  847 Mar 15  2021 61_firewalld-ipsets.pp
-rw-r--r--. 1 root root  139 Mar 15  2021 62_firewalld-direct-chains.pp
-rw-r--r--. 1 root root  574 Mar 15  2021 63_firewalld-services.pp
-rw-r--r--. 1 root root  356 Mar 15  2021 64_firewalld-direct-rules.pp
-rw-r--r--. 1 root root  413 Mar 15  2021 64_firewalld-rules.pp
-rw-r--r--. 1 root root  156 Mar 15  2021 69_firewalld-reload.pp
-rw-r--r--. 1 root root 2.6K Mar 15  2021 90_users.pp
drwxr-xr-x. 8 root root  198 Oct  4 06:20 .git/
-rw-r--r--. 1 root root   28 Mar 15  2021 README.md

The "repofix" script looks like:

#!/bin/bash

set -u

REPOBASE=/var/lib/puppet/manifests
RUNDATE=$(date +'%Y-%m-%d-%H-%M-%S')
LOGPATH=/var/log/repofix
LOGFILE=${LOGPATH}/${RUNDATE}.log

[ ! -d $LOGPATH ] && mkdir -p ${LOGPATH}

echo "RUN for ${RUNDATE}" &> $LOGFILE

for REPO in $(ls -d $REPOBASE/*)
do
  cd ${REPO}
  if [[ !  $(git status | tail -n1) == "nothing to commit, working directory clean" ]]
  then
    echo "need to fix $REPO, showing diff on next line" &>> ${LOGFILE}
    git diff &>> ${LOGFILE}
    git fetch origin &> /dev/null
    git reset --hard origin/master &> /dev/null
  fi
  git pull &> /dev/null
done

echo "END RUN" &>> ${LOGFILE}

This is run by root and uses an SSH key found in /root/.ssh/id_rsa_deploy.pub. To make use of this, the following remediation also needed doing:

1. create a gitea user called deploy
2. add id_rsa_deploy.pub as a valid key to this account
3. amended ~/.ssh/config so that the "deploy" hostname used "deploy" as an account (only for ssh purposes - the username is actually "git"
4. tested with "git status" and "git pull" to check connectivity (outstanding: still some permissions to fix)

Also had to create tombstones/puppet-common repo in gitea so that the git pull commands worked (still some fixing required).

The contents of puppet-common from albert have been committed to gitea (from fender, not albert) but there is still some issue with git on albert pulling them down - to be fixed. Once that's done, albert will begin using files checked into gitea.