# Deployment Server setup, deployment, backups, and rollback procedures for XAMXAM. --- ## Overview - Production host: `xamxam` (over SSH), app root `/var/www/xamxam/` - Files are pushed with `rsync` — **there is no git repo on the remote** - Web / FPM user: `www-data`, app group: `xamxam` - The **DocumentRoot** is `app/public/`, but on the server the code lives flat under `/var/www/xamxam/` (Composer autoload path adjusted to `src/`) - SQLite database at `/var/www/xamxam/storage/xamxam.db` Deployment is orchestrated through the `justfile` (`deploy` group). --- ## One-time server setup From a machine with `ssh` access to the `xamxam` host, **first** bootstrap the role/user/dirs with `scripts/setup-server.sh` (creates the `xamxam` group, adds `www-data` and the deploy user to it, creates `/var/www/xamxam` and the cache/log/backup dirs), then run the full remote provisioning once: ```bash # 1. role/user/dir bootstrap (as root on the host): rsync scripts/setup-server.sh xamxam:/tmp/setup-server.sh ssh -t xamxam "sudo DEPLOY_USER=\$USER bash /tmp/setup-server.sh" # 2. full provisioning (env/APP_KEY + deploy + nginx + backup + cron + logrotate): just provision-server ``` > **Fresh box caveat:** `provision-server` assumes `setup-server.sh` already ran. > On a truly empty `/var/www/xamxam` the deploy's rsync fails with > `Permission denied` on `storage/` until the setgid group-writable tree exists. > `scripts/deploy-server.sh` (via `just deploy-nginx`) is what normalises > ownership/perms on each deploy. This chains (each step is also runnable individually): 1. `scripts/provision-server-env.sh` — ensure the server has an `APP_KEY` in `/var/www/xamxam/.env` (idempotent; see below). 2. `just deploy` — code + Composer deps + migrations + `deploy-nginx` (install + apply the nginx config and fix permissions) + env + permissions check. 3. `just deploy-backup` — install the backup script + cron jobs. 4. `just deploy-cleanup-cron`, `just deploy-logrotate` — install the orphaned- draft cleanup cron and log rotation. It finishes by telling you what's left to do in `/admin/account` (set the admin password — a fresh DB starts unauthenticated — and configure SMTP/PeerTube credentials and Nextcloud sync). > The abandoned-upload GC cron (`just deploy-tmp-cleanup-cron`) is **not** part > of `provision-server`; install it separately if you want hourly GC of > abandoned FilePond uploads. If you'd rather do it step by step (e.g. you already provisioned nginx): ```bash just deploy just deploy-nginx ``` For a full initial rollout *without* the env/nginx steps (backup + cleanup cron only), when those are already handled: ```bash just deploy-all-first # deploy + deploy-backup + deploy-cleanup-cron + deploy-tmp-cleanup-cron + deploy-logrotate ``` ### Server `APP_KEY` — idempotent, never overwrites `just provision-server` (via `scripts/provision-server-env.sh`) ensures the server has an `APP_KEY` in `/var/www/xamxam/.env`. It is **idempotent** and will **never overwrite an existing key**: - If `/var/www/xamxam/.env` already has an `APP_KEY=…` → it is left untouched and the script prints that the key already exists (so encrypted credentials stay decryptable). It exits `0`. - If the file exists but has no `APP_KEY` → a fresh key is appended, existing lines untouched. - If the file is absent → it is created with a fresh key. Ownership (`www-data:xamxam`) and permissions (`640`) are normalised after any write. `deploy-env` also refuses to overwrite a remote `.env` that already has `APP_KEY`. If you ever rotate `APP_KEY`, re-encrypt the SMTP password with `just reencrypt-password ` and push the new key via `just deploy-env`. --- ## Deploying | Command | Purpose | |---------|---------| | `just deploy` | Full deploy: build + code + Composer deps + migrations + env + permissions check | | `just deploy-code` | rsync app files only (no Composer, no nginx, no migrations) | | `just deploy-deps` | Sync composer.{json,lock} → server, then `composer install`/`dump-autoload` | | `just deploy-migrate` | Run pending DB migrations on the server | | `just deploy-env` | Upload `app/.env` (only if the remote `.env` is absent — never overwrites a key) | | `just deploy-nginx` | Upload + apply + reload nginx config | | `just deploy-permissions` | Restore `www-data:xamxam` ownership + 2775/664 perms (via `scripts/fix-permissions.sh`, needs sudo) | | `just deploy-db` | Push local `xamxam.db` → remote (**refuses** if a remote DB already exists) | | `just deploy-verify-permissions` | Check ownership / permissions on the server | > ℹ️ **First deploy?** After `just deploy`, run `just deploy-backup`, > `just deploy-cleanup-cron` and `just deploy-tmp-cleanup-cron` to install the > backup/cleanup cron jobs, and > `just deploy-logrotate` to install log rotation. A single `just deploy-all-first` > chains all of these together. > The app log directory `/var/log/xamxam/` is provisioned automatically by > `scripts/deploy-server.sh` on every `just deploy-nginx` (which `just deploy` > includes), so no separate step is needed for it. To migrate logs written by an > older build, run `just migrate-log-names --apply` once. ### Environment file & re-encryption `app/.env` holds secrets (e.g. `APP_KEY`). `deploy-env` will **not** overwrite a remote `.env` that already has an `APP_KEY`. If you rotate `APP_KEY`, re-encrypt the SMTP password and push the new key: ```bash just reencrypt-password # runs scripts/reencrypt-smtp-password.php on server just deploy-env ``` --- ## Backups SQLite backups are taken with a WAL-safe hot backup (`sqlite3 .backup`) then gzipped to `/var/backups/xamxam/` on the server. | Command | Purpose | |---------|---------| | `just deploy-backup` | Install backup script + cron jobs (one-shot) | | `just deploy-backup-script` | Install `/usr/local/bin/backup-sqlite.sh` | | `just deploy-backup-cron` | Install `/etc/cron.d/xamxam-backup` (hourly 30d + daily 90d) + dirs/log | | `just deploy-check-backup-log` | Tail `/var/log/xamxam-backup-YYYY-MM-DD.log` | | `just deploy-list-backups` | List backups on the server | | `just trigger-backup` | Run the backup script now | | `just test-restore ` | Fetch + decompress + verify a remote snapshot | Retention: hourly backups kept 30 days, nightly (02:00) backups kept 90 days. Backup files: `/var/backups/xamxam/db-.db.gz`. Off-site copies are synced daily to Nextcloud WebDAV — see [nextcloud-sync.md](nextcloud-sync.md). Draft cleanup is handled by a separate cron (`/etc/cron.d/xamxam-cleanup`), installed via `just deploy-cleanup-cron`, logging to `/var/log/xamxam-cleanup-YYYY-MM-DD.log`. Verify with `just deploy-check-cleanup-log`. Abandoned FilePond uploads (staging dirs never linked to a published TFE) are garbage-collected hourly by `/etc/cron.d/xamxam-tmp-cleanup`, installed via `just deploy-tmp-cleanup-cron`, logging to `/var/log/xamxam-tmp-cleanup-YYYY-MM-DD.log`. Verify with `just deploy-check-tmp-cleanup-log`. Because abandoned uploads are removed automatically, the admin cleanup page (`/admin/cleanup.php`, titled *Nettoyage*) now only surfaces the manually-managed **Corbeille** (trashed files, with per-row restore/delete and bulk actions). The manual "Téléversements abandonnés" table was removed from the UI; those files are purged by the cron above. --- ## Permissions model Ownership and permissions are applied by `scripts/deploy-server.sh` (run via `just deploy-nginx`, which `just deploy` includes) and verified by `deploy-verify-permissions`: - Ownership: `www-data:xamxam` - Directories: **2775** (setgid — new files inherit the `xamxam` group) - Regular files: **664** - `storage/xamxam.db` and other `*.db`: **660** - `app/.env`: **640** `just deploy-code` only rsyncs code (as the SSH/deploy user); it does **not** set ownership itself. It therefore finishes by running `just deploy-permissions` (`scripts/fix-permissions.sh` via sudo) so a standalone `deploy-code` can never leave the site broken. Ownership is normalised by `deploy-server.sh` (`chown -R www-data:xamxam /var/www/xamxam`, setgid dirs, locked DBs) too, via `just deploy-nginx`. > **Why this matters (HTTP 500):** php-fpm runs as `www-data`. After a plain > `rsync -az` (run as the deploy user, no `--chown`), synced files/dirs are > owned by the deploy user, and top-level dirs such as `storage/` lose the > `www-data` group-write bit. The app opens the SQLite DB in **WAL mode**, so > www-data must be able to create `xamxam.db-wal` / `xamxam.db-shm` **in** > `storage/`; without write access there the request fatals → **500**. If this > ever recurs (e.g. a manual rsync that skipped `deploy-permissions`), run > `just deploy-permissions` (`sudo chown -R www-data:xamxam` + 2775/664) to > restore it. The nginx/`deploy-server.sh` step (`just deploy-nginx`, or `sudo DEPLOY_USER=$USER bash /tmp/deploy-server.sh` via `just deploy-script`) fixes permissions and installs the nginx config. --- ## Storage migration If paths moved (e.g. legacy upload locations), a one-off migration script can rewrite stored paths on the server: ```bash just deploy-migrate-storage # apply just deploy-migrate-storage --dry-run # dry-run only ``` --- ## Rollback Because there is no repo on the remote, rollback means restoring files and/or restoring the database from a backup. ### Restore the database ```bash ssh xamxam sudo systemctl stop nginx cp /var/backups/xamxam/db-.db.gz /tmp/restore.db.gz gunzip -c /tmp/restore.db.gz > /var/www/xamxam/storage/xamxam.db chown www-data:xamxam /var/www/xamxam/storage/xamxam.db chmod 660 /var/www/xamxam/storage/xamxam.db sudo systemctl start nginx ``` ### Restore application files Re-run `just deploy` from a good local state. In jj you can jump back to a previous commit and redeploy: ```bash jj log # find a known-good change jj edit # work from that revision just deploy ``` --- ## Verify after deploy ```bash curl -I https://xamxam.erg.be/ # expect 200 curl -I https://xamxam.erg.be/admin/ # expect 200 curl -I https://xamxam.erg.be/storage/ # expect 404 (blocked) just deploy-verify-permissions # expect "All permissions OK" ``` --- ## Troubleshooting - **502 Bad Gateway** ```bash sudo systemctl status php8.4-fpm sudo systemctl restart php8.4-fpm ``` - **Nginx config error** ```bash sudo nginx -t ``` - **`deploy-db` refuses to run** The remote DB already exists. Remove it manually only if you intend to overwrite production data. - **Composer did not pick up new classes** `deploy-deps` runs `composer dump-autoload` when the lock checksum is unchanged; if that still fails, force a reinstall. - **Backup/schedule not running** Confirm the cron files are installed (`/etc/cron.d/xamxam-backup`, `/etc/cron.d/xamxam-cleanup`, `/etc/cron.d/xamxam-tmp-cleanup`) and the log files are writable by `www-data`. --- ## Selected commands reference | Command | Purpose | |---------|---------| | `just deploy` | Full deploy (build + code + deps + migrate + env + perms) | | `just deploy-nginx` | Apply nginx config | | `just deploy-backup` | Install backup script + cron | | `just deploy-cleanup-cron` | Install orphaned-draft cleanup cron | | `just deploy-tmp-cleanup-cron` | Install abandoned-upload GC cron | | `just deploy-list-backups` | List server backups | | `just trigger-backup` | Run backup now | | `just reencrypt-password ` | Re-encrypt SMTP password after key rotation | | `just test-restore ` | Verify a snapshot |