filepond: implement async server-ID upload architecture with nested queue support + PeerTube integration

Replace `storeAsFile:true` with a full async FilePond round-trip pipeline using opaque server-side file IDs.

* Added 4 new PHP endpoints under `/admin/actions/filepond/`:

  * `process.php` — upload/process single file and return opaque `file_id`
  * `revert.php` — delete pending tmp uploads before form submit
  * `load.php` — stream existing files by DB ID for FilePond preload
  * `remove.php` — soft-delete `thesis_files` rows
* `process.php` improvements:

  * accept arbitrary FilePond field names instead of hardcoded `file`
  * support PHP-nested multi-file queue inputs (`queue_file[tfe][]`)
  * explicit unwrapping of nested `$_FILES` structures
  * add `audio/mp3` to audio + `peertube_audio` MIME whitelists
  * immediate upload of `peertube_*` files to PeerTube, returning `peertube:{uuid}` IDs
  * extensive `error_log()` instrumentation for request, CSRF, MIME, upload, and save stages
* `revert.php` now accepts `peertube:` IDs without local cleanup
* `ThesisFileHandler`:

  * add `handleFilePondQueueFiles()` + `handleFilePondSingleFile()`
  * process async uploads from `storage/tmp/filepond/` via opaque `file_id`
  * inline handling of `peertube:{uuid}` IDs with direct `thesis_files` insertion
  * remove obsolete deferred PeerTube queue-processing flow
* `ThesisCreateController` + `ThesisEditController`:

  * gate async path behind `filepond_mode=1`
  * preserve legacy multipart flow as fallback
* `file-upload-filepond.js`:

  * remove `storeAsFile:true`
  * add `buildServerConfig()` for async endpoint wiring
  * fix `syncOrderInput()` to use `serverId`
  * add `onprocessfile` hook
  * add `fileValidateSizeFilterItem` for per-extension size caps
  * preload existing uploads via `data-existing-files` + `server.load`
  * replace static `INPUT_ID_TO_TYPE` map with `data-queue-type`
  * add extensive `console.log()` debugging across upload pipeline stages
* `upload-progress.js`:

  * block form submission while uploads are pending
  * update `collectFileNames()` to read processed FilePond items
* Templates/layout:

  * add `data-queue-type`
  * add `data-existing-files`
  * add global CSRF meta tag outside admin-only context
  * add `filepond_mode` hidden input
  * add CSRF token/meta support for partage pages
  * move website URL field below file upload block
* `.gitignore`: exclude `storage/tmp/` from version control
This commit is contained in:
Pontoporeia
2026-05-11 20:11:31 +02:00
parent b56d073210
commit 2e9ebfc684
18 changed files with 1342 additions and 261 deletions

130
TODO.md
View File

@@ -1,123 +1,9 @@
# TODO
# FilePond Server-ID Refactor
## Move Restrictions d'accès aux fichiers to acces.php
- [x] Remove fieldset from templates/admin/contenus.php
- [x] Add fieldset to templates/admin/acces.php
- [x] Load $siteSettings in admin/acces.php controller
- [x] Update redirect in settings.php for formulaire_restrictions → /admin/acces.php
## Fix PeerTube upload — Google-resumable protocol adherence
- [x] Use Location header from init response (not reconstruct URL from JSON body)
- [x] Switch chunk method from PUT → PATCH (Google-resumable variant)
- [x] Use actual file MIME type in chunk Content-Type (not application/octet-stream)
- [x] Ensure chunk size is multiple of 256 KB
- [x] Add PATCH/HEAD methods to httpRequest()
- [x] Add CURLOPT_HEADERFUNCTION to capture response headers
- [x] Disable CURLOPT_FOLLOWLOCATION to preserve Location header
- [x] Add cancelUpload() helper for Delete-on-error cleanup
- [x] PeerTube upload fixed — simple multipart POST /api/v1/videos/upload works
- [x] Upload progress: 0-25% browser upload, 25-99% server polling via /admin/actions/upload-progress.php
- [x] Decorelate formats from fichiers: no HTMX toggling; Site web/Vidéo/Audio always visible
- [x] Sticky formats fieldset inside parent container
- [x] Server-side progress: PeerTubeService writes to temp file, client polls progress endpoint
- [x] Fix cover deletion bug: !empty() instead of isset()
- [x] Remove old duplicate file list CSS; unified recap+edit file figure styling
- [x] Standardise uploaded/preexisting files styling: recap now matches edit (classes, icons, meta row, display_label)
- [x] Refactor licence fieldset: Libre→CC2r+licence, Interne→opt-in licence, Interdit→none
## HTMX Toast Feedback for Settings Checkboxes (contenus.php)
- [x] Add `hx-target` response divs to the three fieldsets in contenus.php
- [x] Update settings.php to return HTML toast on HTMX requests
## JS Refactoring — Extract inline scripts into app/ files
- [x] Remove overtype-webcomponent.min.js (unused)
- [x] Extract copyLogContent + fallbackCopy + tab-updater → app/admin-logs.js
- [x] Extract copyUrl → app/clipboard.js
- [x] Extract tag-search inline script → app/pill-search.js (generalized for tag + language)
- [x] Extract tfe.php access-request form → app/access-request.js
- [x] Update all templates to use new external JS files
## Production Error Fixes (2026-05-11 remote logs)
- [x] **413 Request Entity Too Large** — bumped `client_max_body_size` to 256M, PHP post/upload to 256M, timeouts to 300s
- [x] **Missing `v_smtp_active` view** on server — made all `CREATE VIEW` statements idempotent with `IF NOT EXISTS` in schema.sql
- [x] **`bars.svg` 404** — created `app/public/assets/img/bars.svg` (animated SVG spinner)
- [x] **Nginx rate limiting too aggressive** — increased admin zone to 300r/m, burst=30 to handle ~11 concurrent HTMX fragment requests on contenus.php page load
- [x] **Migration idempotency**`CREATE INDEX` / `CREATE TRIGGER` / `CREATE VIEW` now use `IF NOT EXISTS` in schema.sql and generate-schema.py; migrate.sh no longer fails on re-run
- [ ] **Database readonly** — intermittent permission issue after deploy (added deploy-nginx recipe; permissions should be fixed by --chown + deploy-server.sh)
- [x] **Upload progress bar not visible**`collectFileNames()` now also checks FilePond instances directly (not just `input.files`); `upload-progress.php` no longer requires admin auth (blocked partage form polling)
- [x] **Validation error messages hidden by generic fallback**`ErrorHandler::userMessage` doesn't handle plain `Exception`, so all `throw new Exception(...)` validation messages fall through to "Une erreur inattendue…". Fix: add `Exception` pass-through before the generic fallback.
- [x] **Validation messages use wrong field names** — "Nom/Prénom/Pseudo" → "Auteur·ice(s)", "Titre du mémoire" → "Titre du TFE" to match form labels. Updated autofocusFieldForError in both controllers.
## PeerTube Alternate Labels & FilePond Pools
- [x] Add `peertube_video_label` and `peertube_audio_label` columns (migration 029)
- [x] Update PeerTubeService getSettings/updateSettings for new fields
- [x] Add label fields to parametres.php admin form
- [x] Handle label saving in admin/actions/settings.php
- [x] Uncomment video/audio slots in fichiers-fragment.php with FilePond pools when PeerTube enabled
- [x] Register `peertube_video` / `peertube_audio` queue types in file-upload-filepond.js
- [x] Update handlePeerTubeUpload → handlePeerTubeQueueFiles in both create/edit controllers
- [x] When PeerTube active, restrict TFE pool to PDF/images/VTT/archives only (no video/audio)
- [x] Add HTMX swap attributes to Vidéo/Audio format checkboxes for live toggling
- [x] Store PeerTube uploads as `peertube_ids:{uuid}` in thesis_files.file_path
- [x] Create `templates/partials/peertube-embed.php` iframe embed template
- [x] Render PeerTube embeds in public thesis view (tfe.php)
- [x] Handle PeerTube files in admin recapitulatif.php and fichiers-fragment.php
- [x] Shared SMTP credentials — remove username/password from peertube_settings (migration 031)
- [x] PeerTubeService reads credentials from SmtpRelay
- [x] OAuth client_id/secret fetched on-demand and cached in-memory (no DB storage)
- [x] Resumable upload protocol (POST init + PUT chunks) in PeerTubeService::upload()
- [x] Admin recapitulatif: show real PeerTube watch links (public/unlisted only)
- [x] Optimize public thesis view: load PeerTube instance URL once before file loop
- [ ] Test end-to-end: activate PeerTube, set labels, submit form with video/audio files
## SQLite Backup & Data Integrity (docs/backup-plan.md)
### Phase 1 — WAL Mode
- [x] WAL mode already active (`PRAGMA journal_mode``wal`) — set in Database constructor
- [ ] Verify `-wal` and `-shm` sidecar files exist after writes
- [ ] Verify nginx/PHP write access to sidecar files on server
- [x] Add deploy-verify-permissions recipe that checks ownership, directory perms, file perms, and writability after rsync
- [x] deploy recipe now uploads and runs deploy-server.sh to fix permissions, then verifies them
- [x] deploy recipe now runs migrations (scripts/migrate.sh) after ensuring DB exists
- [x] fix migrate.sh to detect server vs local layout (no app/ subdir on server)
- [x] regenerate schema.sql from local DB via generate-schema.py (includes v_smtp_active, all 28 migrations)
- [x] fix generate-schema.py to include v_smtp_active (was explicitly excluded)
### Phase 2 — Audit Log
- [x] `admin_audit_log` table already exists (migration 009), `AdminLogger` already writes to it
- [x] Create the `audit_log` table for data-level audit (before/after row snapshots)
- [x] Create `Audit.php` helper class
- [x] Instrument all DELETE, UPDATE, INSERT operations on core tables (theses, tags, languages, thesis_files)
- [ ] Verify by triggering a test delete and querying `SELECT * FROM audit_log ORDER BY id DESC LIMIT 5`
### Phase 3 — Soft Deletes
- [x] Add `deleted_at` columns to `languages`, `tags`, `theses`
- [x] Rebuild views `v_theses_full` and `v_theses_public` with `deleted_at IS NULL` filters
- [x] Update `schema.sql` for fresh installs
- [x] Replace all hard DELETEs with soft deletes (`DELETE``UPDATE ... SET deleted_at = ...`)
- [x] Add `deleted_at IS NULL` to all SELECT queries touching these tables
- [x] Add admin "Corbeille" view for soft-deleted theses with Restore and Hard Delete actions
- [ ] Test each htmx-driven element (language search, tag search, repertoire filters) to confirm deleted entries don't appear
- [ ] Admin: add soft-deleted tags/languages view with restore option
### Phase 4 — Hourly Snapshots via Cronjob
- [x] Create `scripts/backup-sqlite.sh` (hot backup via `sqlite3 .backup`, gzip, retention pruning)
- [x] Test locally — backup created, restores correctly
- [x] Add `just backup-snapshot` command for local ad-hoc backups
- [x] Deploy backup script to server (`/usr/local/bin/backup-sqlite.sh`) — `just deploy-backup-script`
- [x] Create `/var/backups/xamxam/` directory on server — part of `just deploy-backup-cron`
- [x] Add cron jobs (hourly 30d + daily 90d) — `just deploy-backup-cron`
- [x] Test restore from production backup — `just test-restore <remote-gz-path>`
- [x] Manual backup trigger — `just trigger-backup`
- [x] Check backup log — `just deploy-check-backup-log`
- [x] List remote backups — `just deploy-list-backups`
- [x] One-shot deploy — `just deploy-backup` (script + cron)
### Phase 5 — Remote Sync *(for later)*
- [ ] (Deferred)
- [x] Step 1 — Build 4 PHP endpoints (process.php, revert.php, load.php, remove.php)
- [x] Step 2 — Update ThesisFileHandler to accept file_ids instead of $_FILES
- [x] Step 3 — Update file-upload-filepond.js (async server model + all fixes)
- [x] Step 4 — Update templates (data-queue-type on all inputs, data-existing-files in edit)
- [x] Step 5 — Update upload-progress.js (new collectFileNames, pending-uploads guard)
- [ ] Step 6 — QA / integration testing
- [ ] Step 7 — Cleanup: remove transition flags, remove INPUT_ID_TO_TYPE