The Ansible Edda

Ansible playbooks for provisioning The Nine Worlds.

Secrets vault

• Encrypt with: ansible-vault encrypt vault.yml
• Decrypt with: ansible-vault decrypt secrets.yml
• Encrypt all vault.yml in a directory with: ansible-vault encrypt directory/**/vault.yml
• Decrypt all vault.yml in a directory with: ansible-vault decrypt directory/**/vault.yml
• Run a playbook with ansible-playbook --vault-id @prompt playbook.yml

The Nine Worlds

The main entrypoint for The Nine Worlds is main.yml.

Keyring integration

Keyring integration requires python3-keyring to be installed.

To set the keyring password run:

./vault-keyring-client.py --set [--vault-id <vault-id>]

If --vault-id is not specified, the password will be stored under ansible.

To use the password from the keyring invoke playbooks with:

ansible-playbook --vault-id @vault-keyring-client.py ...

Production and testing

The inventory files are split into production and testing.

To run the main.yml playbook on production hosts:

ansible-playbook main.yml -i production

To run the main.yml playbook on production hosts:

ansible-playbook main.yml -i testing

Testing virtual machines

The scripts for starting, stopping, and reverting the testing virtual machines is located in scripts/testing/vmgr.py.

Playbooks

The Ansible Edda playbook is composed of smaller playbooks. To run a single playbook, invoke the relevant playbook directly from the playbook directory. For example, to run the system playbook, run:

ansible-playbook playbooks/system.yml

Alternatively you can use its tag as well:

ansible-playbook main.yml --tags "system"

Roles

Playbooks are composed of roles defined in the roles directory, playbooks/roles.

To play only a specific role, e.g. system/base in the playbook system, run:

ansible-playbook playbooks/system.yml --tags "system:base"

Or from the main playbook:

ansible-playbook main.yml --tags "system:base"

Role sub-tasks

Some roles are split into smaller groups of tasks. This can be checked by looking at the tasks/main.yml file of a role, e.g. playbooks/roles/system/base/tasks/main.yml.

To play only a particular group within a role, e.g. sshd in base of system, run:

ansible-playbook playbooks/system.yml --tags "system:base:sshd"

Or from the main playbook:

ansible-playbook main.yml --tags "system:base:sshd"

Testing backups

Before testing the backups, you may want to shut yggdrasil down for extra confidence that it is not being accessed/modified during this process. It is easy to access yggdrasil by accident if /etc/hosts is not modified in the test VM, something that is easy to forget.

Baldur on Scaleway

1. Create baldur by running:

   python scripts/scaleway/baldur.py create --volume-size <size-in-GB>
   

   Pick a volume size that's larger than what yggdrasil estimates for rpool/var/lib/yggdrasil/data.
2. When done destroy baldur by running:

   python scripts/scaleway/baldur.py delete
   

Baldur on Yggdrasil

1. Create a VM on yggdrasil and install the same OS that is running on yggdrasil.

• Install the OS on a zvol on rpool.
• If the same VM is to be used for testing, a GUI is helpful.
• Prepare a zvol on hpool of size that's larger than what yggdrasil estimates for rpool/var/lib/the-nine-worlds/data and mount at /var/lib/the-nine-worlds/data.
• Create non-root user wojtek with sudo privileges.

2. Configure SSH to use yggdrasil as a jump server.
3. Set refreserv=0 on the zvols to make snapshots take less space.
   • zfs set refreserv=0 tank/home/ahrens
4. Use ZFS for snapshots/roolback of the zvols.
   • zfs snapshot tank/home/ahrens@friday
   • zfs rollback tank/home/ahrens@friday
5. Service testing can then be done directly from the VM. To achieve that /etc/hosts needs to be set to directly point at the right proxy server, e.g., 10.66.3.8, not localhost.

Test

1. Provision baldur by running

   ansible-playbook --vault-id @vault-keyring-client.py -i inventory/baldur_production playbooks/baldur.yml
   

2. Restore all the backups by ssh'ing into baldur and running (as root):

   /usr/local/sbin/restic-batch --config-dir /etc/the-nine-worlds/restic-batch.d restore
   

3. Once restore has completed, chown -R <user>:<user> all the restored directories in /var/lib/the-nine-worlds/data. Restic restores the UID information of the host from which the backup was performed which may not match that of the new target machine. Note that permissions and ownership are restored as a second step once all the content is restored. Therefore, the files will list root as owner during the restoration.
4. Start all the pod services with:

   ansible-playbook --vault-id @vault-keyring-client.py -i inventory/baldur_production playbooks/services_start.yml
   

   Give them some time to download all the images and start.
5. Once the CPU returns to idling check the state of all the pod services and their veth interfaces. If necessary restart the affected pod. Sometimes they fail to start (presumably due to issues related to limited CPU and RAM).
6. Boot into a test VM. Ideally, one installed onto a virtual disk since the live system might not have enough space. A VM is used to make sure that none of the services on the host workstation connect to baldur by accident.
7. Modify /etc/hosts in the VM to point at baldur for all relevant domains.
8. Test each service manually one by one. Use the Flagfox add-on to verify that you are indeed connecting to baldur.
   • Some containers fail to start up if the database takes too long to come online. In that case restart the container.
   • Some containers fail to start up if they cannot make DNS queries. Note that 192.168.0.0/16 is blocked by firewall rules. If /etc/the-nine-worlds/resolv.conf points at a DNS resolved at such an address all DNS queries will fail. Simply update resolv.conf to e.g. 1.1.1.1.
9. Stop all the pod services with:

   ansible-playbook --vault-id @vault-keyring-client.py -i inventory/baldur_production playbooks/services_stop.yml
   

Music organisation

The playbooks/music.yml playbook sets up tools and configuration for organising music. The process is manual though. The steps for adding a new CD.

All steps below are to be executed as the music user.

Note on tagging

• For live albums add "YYYY-MM-DD at Venue, City, Country" in the "Subtitle" tag.
• For remasters use original release tags and add "YYYY Remaster" in the "Subtitle" tag.

Ripping a CD

1. Use a CD ripper and rip the CD to /var/lib/yggdrasil/home/music/rip using flac encoding.
2. Samba has been set up to give Windows access to the above directory. Therefore, CD rippers available only for Windows can also be used, e.g. dBpoweramp.

Import new music

1. Run beet import /var/lib/yggdrasil/home/music/rip. This will move the music files to /var/lib/yggdrasil/data/music/collection.
2. Run beet convert -a <match>, where <match> is used to narrow down to new music only. This will convert the flac files into mp3 files for sharing via Nextcloud.
3. Run nextcloud-upload /var/tmp/music/mp3/<artist> for every artist to upload to Nextcloud.
4. Remove the /var/tmp/music/mp3/<artist> directory.

Collections

Every track has a compilation tag at track-level as well as at album-level (at least in Beets). To label the album as a compilation for sorting purposes, run beet modify -a <album> comp=True.

Archive music

From rip

1. Run beet --config .config/beets/archive.yaml import --move /var/lib/yggdrasil/home/music/rip. This will move the music files to /var/lib/yggdrasil/data/music/archive.

From collection

1. Run beet --config .config/beets/archive.yaml import /var/lib/yggdrasil/data/music/collection/<artist>/<album>. This will copy the music files to /var/lib/yggdrasil/data/music/archive.
2. Run beet remove -d -a "album:<album>". This will remove the music files from the collection.