mirror of
https://codeberg.org/PostERG/xamxam.git
synced 2026-09-25 01:53:03 +02:00
admin: backup logs via parameters.php, nextcloud secondary backup
- surface backup/cleanup cron logs + backup freshness status - email xamxam@erg.be when SQLite backups go stale (backup watchdog) - sync SQLite snapshots to Nextcloud WebDAV + remote-freshness watchdog - precise retention pruning, manual sync in check recipe, and Nextcloud-sync docs
This commit is contained in:
Pontoporeia
@@ -20,6 +20,7 @@ This directory mixes **current reference** docs, **proposals/plans**, and
|
||||
| [database.md](database.md) | SQLite schema, migrations, tables, common ops |
|
||||
| [search.md](search.md) | `/search` and `/repertoire` behaviour |
|
||||
| [export.md](export.md) | CSV / DB / files export + full restore procedure |
|
||||
| [nextcloud-sync.md](nextcloud-sync.md) | Off-site SQLite snapshot sync to Nextcloud WebDAV |
|
||||
| [import.md](import.md) | CSV import format + behaviour |
|
||||
| [security.md](security.md) | Current security posture |
|
||||
| [file-uploads.md](file-uploads.md) | Upload surfaces, types, storage layout |
|
||||
|
||||
+9
-19
@@ -162,28 +162,18 @@
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Remote Sync *(for later)*
|
||||
## Phase 5 — Remote Sync *(implemented — see [nextcloud-sync.md](nextcloud-sync.md))*
|
||||
|
||||
**Goal:** Push backups off the VM to a remote destination so a disk failure or VM loss doesn't take your history with it.
|
||||
|
||||
- [ ] Choose a remote destination (Backblaze B2, S3, SFTP, etc.)
|
||||
- [ ] Install and configure rclone:
|
||||
```bash
|
||||
apt install rclone
|
||||
rclone config # set up a remote, name it "mybackups"
|
||||
```
|
||||
- [ ] Add remote sync to the backup script after the `gzip` step:
|
||||
```bash
|
||||
rclone copy "$BACKUP_FILE" mybackups:myapp-backups/
|
||||
```
|
||||
- [ ] Enable versioning on the remote bucket (B2/S3) so even remote overwrites are recoverable
|
||||
- [ ] Test a full restore from remote:
|
||||
```bash
|
||||
rclone copy mybackups:myapp-backups/db-<timestamp>.db.gz /tmp/
|
||||
gunzip /tmp/db-<timestamp>.db.gz
|
||||
sqlite3 /tmp/db-<timestamp>.db ".tables"
|
||||
```
|
||||
- [ ] (Optional) Set up a separate cron to prune remote copies older than 6 months
|
||||
Implemented via a PHP WebDAV sync to Nextcloud (`cloud.erg.school`), reusing the SMTP credentials — see [nextcloud-sync.md](nextcloud-sync.md) for the full reference. The checklist below is superseded by that doc.
|
||||
|
||||
- [x] Choose a remote destination — **Nextcloud WebDAV** (`/XAMXAM-BCK`)
|
||||
- [x] Transport — **PHP `curl`**, not rclone (reuses SMTP credentials, no extra binary)
|
||||
- [x] Add remote sync separate from the backup script — `scripts/nextcloud-sync.php` (daily cron, 20 min after the 02:00 snapshot)
|
||||
- [x] Remote retention — keep last 7 snapshots (`REMOTE_KEEP`)
|
||||
- [x] Test a full restore from remote — restore procedure documented in [nextcloud-sync.md](nextcloud-sync.md)
|
||||
- [x] Monitoring — `backup-watchdog.php` alerts on stale/missing remote copy
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -95,6 +95,9 @@ gzipped to `/var/backups/xamxam/` on the server.
|
||||
Retention: hourly backups kept 30 days, nightly (02:00) backups kept 90 days.
|
||||
Backup files: `/var/backups/xamxam/db-<timestamp>.db.gz`.
|
||||
|
||||
Off-site copies are synced daily to Nextcloud WebDAV — see
|
||||
[nextcloud-sync.md](nextcloud-sync.md).
|
||||
|
||||
Draft cleanup is handled by a separate cron (`/etc/cron.d/xamxam-cleanup`),
|
||||
installed via `just deploy-cleanup-cron`, logging to
|
||||
`/var/log/xamxam-cleanup-YYYY-MM-DD.log`. Verify with `just deploy-check-cleanup-log`.
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
# Nextcloud off-site backup sync
|
||||
|
||||
XAMXAM pushes its latest SQLite snapshot to a Nextcloud folder so that a
|
||||
server/disk failure doesn't take the backup history with it. This is the
|
||||
implementation of *Phase 5 — Remote Sync* from [backup-plan.md](backup-plan.md).
|
||||
|
||||
## What it does
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Destination** | `https://cloud.erg.school/remote.php/dav/files/xamxam%40erg.be/XAMXAM-BCK` |
|
||||
| **Protocol** | WebDAV (HTTP `PUT` / `PROPFIND` / `DELETE`) |
|
||||
| **Credentials** | Reused from `smtp_settings` (`xamxam@erg.be`), decrypted via `Crypto` |
|
||||
| **Transport** | PHP `curl` extension — no `rclone`, no shell `curl` |
|
||||
| **What is uploaded** | The newest `db-*.db.gz` snapshot from `/var/backups/xamxam` |
|
||||
| **Remote filename** | `xamxam-db-<timestamp>.db.gz` (timestamped, not overwritten) |
|
||||
| **Remote retention** | Last **7** snapshots (configurable via `REMOTE_KEEP`) |
|
||||
| **Cadence** | Daily, 20 min after the 02:00 snapshot |
|
||||
|
||||
The uploads are **SQLite snapshots only** — the CSV and files-ZIP exports
|
||||
(see [export.md](export.md)) are *not* pushed. This keeps Nextcloud usage
|
||||
minimal; the full export set remains available on-demand from the admin panel.
|
||||
|
||||
### Sizing
|
||||
|
||||
As of writing, the live DB is ~2.5 MB and compresses to ~475 KB. Seven daily
|
||||
snapshots are therefore ~3.3 MB — a negligible fraction of the 10 GB Nextcloud
|
||||
quota for `xamxam@erg.be`. If more history is desired, raise `REMOTE_KEEP` in
|
||||
the cron line; there is ample headroom.
|
||||
|
||||
## Script: `scripts/nextcloud-sync.php`
|
||||
|
||||
CLI script (deployed to `/usr/local/bin/nextcloud-sync.php`), run by cron.
|
||||
|
||||
Flow:
|
||||
1. Resolve `APP_ROOT` (`/var/www/xamxam` in prod, `app/` in dev) and load the
|
||||
composer autoloader.
|
||||
2. No-op `DatabaseMigrations` (read-only — the sync must never mutate the DB).
|
||||
3. Load SMTP credentials via `SmtpRelay::getSettings()` (decrypts the password).
|
||||
4. Find the newest `db-*.db.gz` in `/var/backups/xamxam`.
|
||||
5. `PUT` it to the WebDAV path (streaming upload, memory-safe).
|
||||
6. `PROPFIND` the uploaded file and verify its size matches the local file
|
||||
(integrity check — a failed sync is detected, not assumed).
|
||||
7. `PROPFIND` the folder, list `xamxam-db-*.db.gz`, and `DELETE` the oldest
|
||||
beyond `REMOTE_KEEP` (`7` by default).
|
||||
|
||||
Exit code `0` on success, `1` on any failure (logged to
|
||||
`/var/log/xamxam-backup-YYYY-MM-DD.log`). A sync failure never blocks the
|
||||
local backup — the two jobs are separate cron entries.
|
||||
|
||||
### Environment variables
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
|----------|---------|---------|
|
||||
| `REMOTE_KEEP` | `7` | Number of remote snapshots to retain |
|
||||
|
||||
## Cron wiring
|
||||
|
||||
In `/etc/cron.d/xamxam-backup` (deployed from `deploy/xamxam-backup.cron`):
|
||||
|
||||
```
|
||||
# Daily snapshot at 2am — kept 90 days
|
||||
0 2 * * * www-data RETENTION_DAYS=90 /usr/local/bin/backup-sqlite.sh >> /var/log/xamxam-backup-$(date +\%Y-\%m-\%d).log 2>&1
|
||||
|
||||
# Push the latest snapshot to Nextcloud WebDAV (daily, after the 2am snapshot)
|
||||
20 2 * * * www-data php /usr/local/bin/nextcloud-sync.php >> /var/log/xamxam-backup-$(date +\%Y-\%m-\%d).log 2>&1
|
||||
```
|
||||
|
||||
The sync runs as `www-data`, so it shares the file permissions of the
|
||||
backup script (`/var/backups/xamxam` is `www-data:www-data`). The `.env`
|
||||
holding the crypto key is `640 www-data:xamxam` — readable by `www-data`, so
|
||||
`Crypto::decrypt` works from the CLI.
|
||||
|
||||
## Monitoring
|
||||
|
||||
The backup watchdog ([`scripts/backup-watchdog.php`](../scripts/backup-watchdog.php))
|
||||
now checks **both** the local snapshots and the Nextcloud copy:
|
||||
|
||||
- **Local** — newest `db-*.db.gz` older than 2 h → alert.
|
||||
- **Remote** — newest `xamxam-db-*.db.gz` older than 48 h (or none found) → alert.
|
||||
|
||||
Both feed the same anti-flood email to `xamxam@erg.be` (or the configured
|
||||
`notify_email`). A single alert can report either or both issues.
|
||||
|
||||
## Manual verification
|
||||
|
||||
```bash
|
||||
# Read the backup folder (expect 207 Multi-Status)
|
||||
curl -sS -u 'xamxam@erg.be:<password>' -X PROPFIND --head \
|
||||
'https://cloud.erg.school/remote.php/dav/files/xamxam%40erg.be/XAMXAM-BCK' \
|
||||
-o /dev/null -w '%{http_code}\n'
|
||||
|
||||
# Upload a test file (expect 201), then delete it (expect 204)
|
||||
curl -sS -u 'xamxam@erg.be:<password>' -X PUT --data-binary test \
|
||||
'https://cloud.erg.school/remote.php/dav/files/xamxam%40erg.be/XAMXAM-BCK/_write_test.txt' \
|
||||
-o /dev/null -w '%{http_code}\n'
|
||||
curl -sS -u 'xamxam@erg.be:<password>' -X DELETE \
|
||||
'https://cloud.erg.school/remote.php/dav/files/xamxam%40erg.be/XAMXAM-BCK/_write_test.txt' \
|
||||
-o /dev/null -w '%{http_code}\n'
|
||||
|
||||
# Run the sync manually (server only)
|
||||
sudo -u www-data php /usr/local/bin/nextcloud-sync.php
|
||||
```
|
||||
|
||||
## Restoring from Nextcloud
|
||||
|
||||
The remote snapshot is the same `.db.gz` the local backup produces, so
|
||||
restoring follows the standard procedure in [deployment.md](deployment.md):
|
||||
|
||||
```bash
|
||||
# Download the snapshot
|
||||
curl -sS -u 'xamxam@erg.be:<password>' -O \
|
||||
'https://cloud.erg.school/remote.php/dav/files/xamxam%40erg.be/XAMXAM-BCK/xamxam-db-<timestamp>.db.gz'
|
||||
|
||||
# Restore
|
||||
sudo systemctl stop nginx
|
||||
gunzip -c xamxam-db-<timestamp>.db.gz > /var/www/xamxam/storage/xamxam.db
|
||||
chown www-data:www-data /var/www/xamxam/storage/xamxam.db
|
||||
chmod 660 /var/www/xamxam/storage/xamxam.db
|
||||
sudo systemctl start nginx
|
||||
```
|
||||
|
||||
## Security notes
|
||||
|
||||
- Credentials are read from the DB (`smtp_settings`) and **never** logged or
|
||||
echoed; only the resulting success/failure messages are written to the log.
|
||||
- The Nextcloud password is the same LDAP/SSO-backed `xamxam@erg.be`
|
||||
credential used for SMTP — the WebDAV Basic-auth endpoint accepts it directly
|
||||
(verified live: `PUT` → 201, `PROPFIND` → 207, `DELETE` → 204).
|
||||
- `CURLOPT_SSL_VERIFYPEER` / `CURLOPT_SSL_VERIFYHOST` are enabled (certificate
|
||||
chain is validated).
|
||||
- The sync is one-way and additive-then-pruned: it never deletes a snapshot
|
||||
until a newer one has been confirmed uploaded.
|
||||
Reference in New Issue
Block a user