225 lines
8.3 KiB
Markdown
225 lines
8.3 KiB
Markdown
# The Ansible Edda
|
|
|
|
Ansible playbooks for provisioning The Nine Worlds.
|
|
|
|
## Secrets vault
|
|
|
|
- Encrypt with: ```ansible-vault encrypt vault.yml```
|
|
- Decrypt with: ```ansible-vault decrypt secrets.yml```
|
|
- Encrypt all `vault.yml` in a directory with: ```ansible-vault encrypt directory/**/vault.yml```
|
|
- Decrypt all `vault.yml` in a directory with: ```ansible-vault decrypt directory/**/vault.yml```
|
|
- Run a playbook with ```ansible-playbook --vault-id @prompt playbook.yml```
|
|
|
|
## The Nine Worlds
|
|
|
|
The main entrypoint for The Nine Worlds is [`main.yml`](main.yml).
|
|
|
|
### Keyring integration
|
|
|
|
Keyring integration requires `python3-keyring` to be installed.
|
|
|
|
To set the keyring password run:
|
|
|
|
``` sh
|
|
./vault-keyring-client.py --set [--vault-id <vault-id>]
|
|
```
|
|
|
|
If `--vault-id` is not specified, the password will be stored under `ansible`.
|
|
|
|
To use the password from the keyring invoke playbooks with:
|
|
|
|
``` sh
|
|
ansible-playbook --vault-id @vault-keyring-client.py ...
|
|
```
|
|
|
|
### Production and testing
|
|
|
|
The inventory files are split into [`production`](production) and [`testing`](testing).
|
|
|
|
To run the `main.yml` playbook on production hosts:
|
|
``` sh
|
|
ansible-playbook main.yml -i production
|
|
```
|
|
|
|
To run the `main.yml` playbook on production hosts:
|
|
``` sh
|
|
ansible-playbook main.yml -i testing
|
|
```
|
|
|
|
### Testing virtual machines
|
|
|
|
The scripts for starting, stopping, and reverting the testing virtual machines is located in
|
|
`scripts/testing/vmgr.py`.
|
|
|
|
### Playbooks
|
|
|
|
The Ansible Edda playbook is composed of smaller [`playbooks`](playbooks). To run a single playbook,
|
|
invoke the relevant playbook directly from the playbook directory. For example, to run the
|
|
[`system`](system) playbook, run:
|
|
|
|
``` sh
|
|
ansible-playbook playbooks/system.yml
|
|
```
|
|
|
|
Alternatively you can use its tag as well:
|
|
|
|
``` sh
|
|
ansible-playbook main.yml --tags "system"
|
|
```
|
|
|
|
### Roles
|
|
|
|
Playbooks are composed of roles defined in the `roles` directory,
|
|
[`playbooks/roles`](playbooks/roles).
|
|
|
|
To play only a specific role, e.g. `system/base` in the playbook `system`, run:
|
|
|
|
``` sh
|
|
ansible-playbook playbooks/system.yml --tags "system:base"
|
|
```
|
|
|
|
Or from the main playbook:
|
|
|
|
``` sh
|
|
ansible-playbook main.yml --tags "system:base"
|
|
```
|
|
|
|
### Role sub-tasks
|
|
|
|
Some roles are split into smaller groups of tasks. This can be checked by looking at the
|
|
`tasks/main.yml` file of a role, e.g.
|
|
[`playbooks/roles/system/base/tasks/main.yml`](playbooks/roles/system/base/tasks/main.yml).
|
|
|
|
To play only a particular group within a role, e.g. `sshd` in `base` of `system`, run:
|
|
|
|
``` sh
|
|
ansible-playbook playbooks/system.yml --tags "system:base:sshd"
|
|
```
|
|
|
|
Or from the main playbook:
|
|
|
|
``` sh
|
|
ansible-playbook main.yml --tags "system:base:sshd"
|
|
```
|
|
|
|
## Testing backups
|
|
|
|
Before testing the backups, you may want to shut `yggdrasil` down for extra confidence that it is
|
|
not being accessed/modified during this process. It is easy to access `yggdrasil` by accident if
|
|
`/etc/hosts` is not modified in the test VM, something that is easy to forget.
|
|
|
|
### Baldur on Scaleway
|
|
|
|
1. Create `baldur` by running:
|
|
```sh
|
|
python scripts/scaleway/baldur.py create --volume-size <size-in-GB>
|
|
```
|
|
Pick a volume size that's larger than what `yggdrasil` estimates for
|
|
`rpool/var/lib/yggdrasil/data`.
|
|
2. When done destroy `baldur` by running:
|
|
```sh
|
|
python scripts/scaleway/baldur.py delete
|
|
```
|
|
|
|
### Baldur on Yggdrasil
|
|
|
|
1. Create a VM on `yggdrasil` and install the same OS that is running on `yggdrasil`.
|
|
- Install the OS on a zvol on `rpool`.
|
|
- If the same VM is to be used for testing, a GUI is helpful.
|
|
- Prepare a zvol on `hpool` of size that's larger than what `yggdrasil` estimates for
|
|
`rpool/var/lib/the-nine-worlds/data` and mount at `/var/lib/the-nine-worlds/data`.
|
|
- Create non-root user `wojtek` with `sudo` privileges.
|
|
2. Configure SSH to use `yggdrasil` as a jump server.
|
|
3. Set `refreserv=0` on the zvols to make snapshots take less space.
|
|
- `zfs set refreserv=0 tank/home/ahrens`
|
|
4. Use ZFS for snapshots/roolback of the zvols.
|
|
- `zfs snapshot tank/home/ahrens@friday`
|
|
- `zfs rollback tank/home/ahrens@friday`
|
|
5. Service testing can then be done directly from the VM. To achieve that `/etc/hosts` needs to be
|
|
set to directly point at the right proxy server, e.g., `10.66.3.8`, not `localhost`.
|
|
|
|
### Test
|
|
|
|
1. Provision `baldur` by running
|
|
```sh
|
|
ansible-playbook --vault-id @vault-keyring-client.py -i inventory/baldur_production playbooks/baldur.yml
|
|
```
|
|
2. Restore all the backups by ssh'ing into `baldur` and running (as root):
|
|
```sh
|
|
/usr/local/sbin/restic-batch --config-dir /etc/the-nine-worlds/restic-batch.d restore
|
|
```
|
|
3. Once restore has completed, `chown -R <user>:<user>` all the restored directories in
|
|
`/var/lib/the-nine-worlds/data`. Restic restores the UID information of the host from which the
|
|
backup was performed which may not match that of the new target machine. Note that permissions
|
|
and ownership are restored as a second step once all the content is restored. Therefore, the
|
|
files will list `root` as owner during the restoration.
|
|
4. Start all the pod services with:
|
|
```sh
|
|
ansible-playbook --vault-id @vault-keyring-client.py -i inventory/baldur_production playbooks/services_start.yml
|
|
```
|
|
Give them some time to download all the images and start.
|
|
5. Once the CPU returns to idling check the state of all the pod services and their `veth`
|
|
interfaces. If necessary restart the affected pod. Sometimes they fail to start (presumably due
|
|
to issues related to limited CPU and RAM).
|
|
6. Boot into a test VM. Ideally, one installed onto a virtual disk since the live system might not
|
|
have enough space. A VM is used to make sure that none of the services on the host workstation
|
|
connect to `baldur` by accident.
|
|
7. Modify `/etc/hosts` in the VM to point at `baldur` for all relevant domains.
|
|
8. Test each service manually one by one. Use the Flagfox add-on to verify that you are indeed
|
|
connecting to `baldur`.
|
|
- Some containers fail to start up if the database takes too long to come online. In that case
|
|
restart the container.
|
|
- Some containers fail to start up if they cannot make DNS queries. Note that `192.168.0.0/16` is
|
|
blocked by firewall rules. If `/etc/the-nine-worlds/resolv.conf` points at a DNS resolved at
|
|
such an address all DNS queries will fail. Simply update `resolv.conf` to e.g. `1.1.1.1`.
|
|
9. Stop all the pod services with:
|
|
```sh
|
|
ansible-playbook --vault-id @vault-keyring-client.py -i inventory/baldur_production playbooks/services_stop.yml
|
|
```
|
|
|
|
## Music organisation
|
|
|
|
The `playbooks/music.yml` playbook sets up tools and configuration for organising music. The process
|
|
is manual though. The steps for adding a new CD.
|
|
|
|
All steps below are to be executed as the `music` user.
|
|
|
|
### Note on tagging
|
|
|
|
* For live albums add "YYYY-MM-DD at Venue, City, Country" in the "Subtitle" tag.
|
|
* For remasters use original release tags and add "YYYY Remaster" in the "Subtitle" tag.
|
|
|
|
### Ripping a CD
|
|
|
|
1. Use a CD ripper and rip the CD to `/var/lib/yggdrasil/home/music/rip` using flac encoding.
|
|
2. Samba has been set up to give Windows access to the above directory. Therefore, CD rippers
|
|
available only for Windows can also be used, e.g. dBpoweramp.
|
|
|
|
### Import new music
|
|
|
|
1. Run `beet import /var/lib/yggdrasil/home/music/rip`. This will move the music files to
|
|
`/var/lib/yggdrasil/data/music/collection`.
|
|
2. Run `beet convert -a <match>`, where `<match>` is used to narrow down to new music only. This
|
|
will convert the flac files into mp3 files for sharing via Nextcloud.
|
|
3. Run `nextcloud-upload /var/tmp/music/mp3/<artist>` for every artist to upload to Nextcloud.
|
|
4. Remove the `/var/tmp/music/mp3/<artist>` directory.
|
|
|
|
#### Collections
|
|
|
|
Every track has a `compilation` tag at track-level as well as at album-level (at least in Beets). To
|
|
label the album as a compilation for sorting purposes, run `beet modify -a <album> comp=True`.
|
|
|
|
### Archive music
|
|
|
|
#### From rip
|
|
|
|
1. Run `beet --config .config/beets/archive.yaml import --move /var/lib/yggdrasil/home/music/rip`.
|
|
This will move the music files to `/var/lib/yggdrasil/data/music/archive`.
|
|
|
|
#### From collection
|
|
|
|
1. Run `beet --config .config/beets/archive.yaml import
|
|
/var/lib/yggdrasil/data/music/collection/<artist>/<album>`. This will copy the music files to
|
|
`/var/lib/yggdrasil/data/music/archive`.
|
|
2. Run `beet remove -d -a "album:<album>"`. This will remove the music files from the collection.
|