xamxam/docs/deployment.md

# Deployment

Server setup, deployment, backups, and rollback procedures for XAMXAM.

---

## Overview

- Production host: `xamxam` (over SSH), app root `/var/www/xamxam/`
- Files are pushed with `rsync` — **there is no git repo on the remote**
- Web / FPM user: `www-data`, app group: `xamxam`
- The **DocumentRoot** is `app/public/`, but on the server the code lives flat
  under `/var/www/xamxam/` (Composer autoload path adjusted to `src/`)
- SQLite database at `/var/www/xamxam/storage/xamxam.db`

Deployment is orchestrated through the `justfile` (`deploy` group).

---

## One-time server setup

From a machine with `ssh` access to the `xamxam` host, **first** bootstrap the
role/user/dirs with `scripts/setup-server.sh` (creates the `xamxam` group, adds
`www-data` and the deploy user to it, creates `/var/www/xamxam` and the
cache/log/backup dirs), then run the full remote provisioning once:

```bash
# 1. role/user/dir bootstrap (as root on the host):
rsync scripts/setup-server.sh xamxam:/tmp/setup-server.sh
ssh -t xamxam "sudo DEPLOY_USER=\$USER bash /tmp/setup-server.sh"

# 2. full provisioning (env/APP_KEY + deploy + nginx + backup + cron + logrotate):
just provision-server
```

> **Fresh box caveat:** `provision-server` assumes `setup-server.sh` already ran.
> On a truly empty `/var/www/xamxam` the deploy's rsync fails with
> `Permission denied` on `storage/` until the setgid group-writable tree exists.
> `scripts/deploy-server.sh` (via `just deploy-nginx`) is what normalises
> ownership/perms on each deploy.

This chains (each step is also runnable individually):

1. `scripts/provision-server-env.sh` — ensure the server has an `APP_KEY` in
   `/var/www/xamxam/.env` (idempotent; see below).
2. `just deploy` — code + Composer deps + migrations + `deploy-nginx`
   (install + apply the nginx config and fix permissions) + env + permissions check.
3. `just deploy-backup` — install the backup script + cron jobs.
4. `just deploy-cleanup-cron`, `just deploy-logrotate` — install the orphaned-
   draft cleanup cron and log rotation.

It finishes by telling you what's left to do in `/admin/account` (set the admin
password — a fresh DB starts unauthenticated — and configure SMTP/PeerTube
credentials and Nextcloud sync).

> The abandoned-upload GC cron (`just deploy-tmp-cleanup-cron`) is **not** part
> of `provision-server`; install it separately if you want hourly GC of
> abandoned FilePond uploads.

If you'd rather do it step by step (e.g. you already provisioned nginx):

```bash
just deploy
just deploy-nginx
```

For a full initial rollout *without* the env/nginx steps (backup + cleanup cron
only), when those are already handled:

```bash
just deploy-all-first        # deploy + deploy-backup + deploy-cleanup-cron + deploy-tmp-cleanup-cron + deploy-logrotate
```

### Server `APP_KEY` — idempotent, never overwrites

`just provision-server` (via `scripts/provision-server-env.sh`) ensures the
server has an `APP_KEY` in `/var/www/xamxam/.env`. It is **idempotent** and will
**never overwrite an existing key**:

- If `/var/www/xamxam/.env` already has an `APP_KEY=…` → it is left untouched
  and the script prints that the key already exists (so encrypted credentials
  stay decryptable). It exits `0`.
- If the file exists but has no `APP_KEY` → a fresh key is appended, existing
  lines untouched.
- If the file is absent → it is created with a fresh key.

Ownership (`www-data:xamxam`) and permissions (`640`) are normalised after any
write. `deploy-env` also refuses to overwrite a remote `.env` that already has
`APP_KEY`.

If you ever rotate `APP_KEY`, re-encrypt the SMTP password with
`just reencrypt-password <new_base64_key>` and push the new key via
`just deploy-env`.

---

## Deploying

| Command | Purpose |
|---------|---------|
| `just deploy` | Full deploy: build + code + Composer deps + migrations + env + permissions check |
| `just deploy-code` | rsync app files only (no Composer, no nginx, no migrations) |
| `just deploy-deps` | Sync composer.{json,lock} → server, then `composer install`/`dump-autoload` |
| `just deploy-migrate` | Run pending DB migrations on the server |
| `just deploy-env` | Upload `app/.env` (only if the remote `.env` is absent — never overwrites a key) |
| `just deploy-nginx` | Upload + apply + reload nginx config |
| `just deploy-permissions` | Restore `www-data:xamxam` ownership + 2775/664 perms (via `scripts/fix-permissions.sh`, needs sudo) |
| `just deploy-db` | Push local `xamxam.db` → remote (**refuses** if a remote DB already exists) |
| `just deploy-verify-permissions` | Check ownership / permissions on the server |

> ℹ️ **First deploy?** After `just deploy`, run `just deploy-backup`,
> `just deploy-cleanup-cron` and `just deploy-tmp-cleanup-cron` to install the
> backup/cleanup cron jobs, and
> `just deploy-logrotate` to install log rotation. A single `just deploy-all-first`
> chains all of these together.
> The app log directory `/var/log/xamxam/` is provisioned automatically by
> `scripts/deploy-server.sh` on every `just deploy-nginx` (which `just deploy`
> includes), so no separate step is needed for it. To migrate logs written by an
> older build, run `just migrate-log-names --apply` once.

### Environment file & re-encryption

`app/.env` holds secrets (e.g. `APP_KEY`). `deploy-env` will **not** overwrite a
remote `.env` that already has an `APP_KEY`. If you rotate `APP_KEY`, re-encrypt
the SMTP password and push the new key:

```bash
just reencrypt-password <new_base64_key>   # runs scripts/reencrypt-smtp-password.php on server
just deploy-env
```

---

## Backups

SQLite backups are taken with a WAL-safe hot backup (`sqlite3 .backup`) then
gzipped to `/var/backups/xamxam/` on the server.

| Command | Purpose |
|---------|---------|
| `just deploy-backup` | Install backup script + cron jobs (one-shot) |
| `just deploy-backup-script` | Install `/usr/local/bin/backup-sqlite.sh` |
| `just deploy-backup-cron` | Install `/etc/cron.d/xamxam-backup` (hourly 30d + daily 90d) + dirs/log |
| `just deploy-check-backup-log` | Tail `/var/log/xamxam-backup-YYYY-MM-DD.log` |
| `just deploy-list-backups` | List backups on the server |
| `just trigger-backup` | Run the backup script now |
| `just test-restore <path.gz>` | Fetch + decompress + verify a remote snapshot |

Retention: hourly backups kept 30 days, nightly (02:00) backups kept 90 days.
Backup files: `/var/backups/xamxam/db-<timestamp>.db.gz`.

Off-site copies are synced daily to Nextcloud WebDAV — see
[nextcloud-sync.md](nextcloud-sync.md).

Draft cleanup is handled by a separate cron (`/etc/cron.d/xamxam-cleanup`),
installed via `just deploy-cleanup-cron`, logging to
`/var/log/xamxam-cleanup-YYYY-MM-DD.log`. Verify with `just deploy-check-cleanup-log`.

Abandoned FilePond uploads (staging dirs never linked to a published TFE) are
garbage-collected hourly by `/etc/cron.d/xamxam-tmp-cleanup`, installed via
`just deploy-tmp-cleanup-cron`, logging to
`/var/log/xamxam-tmp-cleanup-YYYY-MM-DD.log`. Verify with
`just deploy-check-tmp-cleanup-log`.

Because abandoned uploads are removed automatically, the admin cleanup page
(`/admin/cleanup.php`, titled *Nettoyage*) now only surfaces the manually-managed
**Corbeille** (trashed files, with per-row restore/delete and bulk actions). The
manual "Téléversements abandonnés" table was removed from the UI; those files
are purged by the cron above.

---

## Permissions model

Ownership and permissions are applied by `scripts/deploy-server.sh` (run via
`just deploy-nginx`, which `just deploy` includes) and verified by
`deploy-verify-permissions`:

- Ownership: `www-data:xamxam`
- Directories: **2775** (setgid — new files inherit the `xamxam` group)
- Regular files: **664**
- `storage/xamxam.db` and other `*.db`: **660**
- `app/.env`: **640**

`just deploy-code` only rsyncs code (as the SSH/deploy user); it does **not**
set ownership itself. It therefore finishes by running `just deploy-permissions`
(`scripts/fix-permissions.sh` via sudo) so a standalone `deploy-code` can never
leave the site broken. Ownership is normalised by `deploy-server.sh`
(`chown -R www-data:xamxam /var/www/xamxam`, setgid dirs, locked DBs) too, via
`just deploy-nginx`.

> **Why this matters (HTTP 500):** php-fpm runs as `www-data`. After a plain
> `rsync -az` (run as the deploy user, no `--chown`), synced files/dirs are
> owned by the deploy user, and top-level dirs such as `storage/` lose the
> `www-data` group-write bit. The app opens the SQLite DB in **WAL mode**, so
> www-data must be able to create `xamxam.db-wal` / `xamxam.db-shm` **in**
> `storage/`; without write access there the request fatals → **500**. If this
> ever recurs (e.g. a manual rsync that skipped `deploy-permissions`), run
> `just deploy-permissions` (`sudo chown -R www-data:xamxam` + 2775/664) to
> restore it.

The nginx/`deploy-server.sh` step (`just deploy-nginx`, or
`sudo DEPLOY_USER=$USER bash /tmp/deploy-server.sh` via `just deploy-script`)
fixes permissions and installs the nginx config.

---

## Storage migration

If paths moved (e.g. legacy upload locations), a one-off migration script can
rewrite stored paths on the server:

```bash
just deploy-migrate-storage             # apply
just deploy-migrate-storage --dry-run   # dry-run only
```

---

## Rollback

Because there is no repo on the remote, rollback means restoring files and/or
restoring the database from a backup.

### Restore the database

```bash
ssh xamxam
sudo systemctl stop nginx
cp /var/backups/xamxam/db-<timestamp>.db.gz /tmp/restore.db.gz
gunzip -c /tmp/restore.db.gz > /var/www/xamxam/storage/xamxam.db
chown www-data:xamxam /var/www/xamxam/storage/xamxam.db
chmod 660 /var/www/xamxam/storage/xamxam.db
sudo systemctl start nginx
```

### Restore application files

Re-run `just deploy` from a good local state. In jj you can jump back to a
previous commit and redeploy:

```bash
jj log                          # find a known-good change
jj edit <previous-change-id>    # work from that revision
just deploy
```

---

## Verify after deploy

```bash
curl -I https://xamxam.erg.be/            # expect 200
curl -I https://xamxam.erg.be/admin/      # expect 200
curl -I https://xamxam.erg.be/storage/    # expect 404 (blocked)
just deploy-verify-permissions            # expect "All permissions OK"
```

---

## Troubleshooting

- **502 Bad Gateway**
  ```bash
  sudo systemctl status php8.4-fpm
  sudo systemctl restart php8.4-fpm
  ```
- **Nginx config error**
  ```bash
  sudo nginx -t
  ```
- **`deploy-db` refuses to run**
  The remote DB already exists. Remove it manually only if you intend to
  overwrite production data.
- **Composer did not pick up new classes**
  `deploy-deps` runs `composer dump-autoload` when the lock checksum is
  unchanged; if that still fails, force a reinstall.
- **Backup/schedule not running**
  Confirm the cron files are installed (`/etc/cron.d/xamxam-backup`,
  `/etc/cron.d/xamxam-cleanup`, `/etc/cron.d/xamxam-tmp-cleanup`) and the log
  files are writable by `www-data`.

---

## Selected commands reference

| Command | Purpose |
|---------|---------|
| `just deploy` | Full deploy (build + code + deps + migrate + env + perms) |
| `just deploy-nginx` | Apply nginx config |
| `just deploy-backup` | Install backup script + cron |
| `just deploy-cleanup-cron` | Install orphaned-draft cleanup cron |
| `just deploy-tmp-cleanup-cron` | Install abandoned-upload GC cron |
| `just deploy-list-backups` | List server backups |
| `just trigger-backup` | Run backup now |
| `just reencrypt-password <key>` | Re-encrypt SMTP password after key rotation |
| `just test-restore <path.gz>` | Verify a snapshot |