diag: invalid_grant is SSO auth-method mismatch, not bad creds

- feat: creds-test.sh gum probe for SMTP vs PeerTube auth + PeerTubeService::probeAuth()
- feat: app-token.sh gum probe for long-lived PeerTube app token (client_credentials)
- docs: add copy-paste proof commands to demonstrate the SSO break to admins
This commit is contained in:
Pontoporeia
2026-08-24 11:33:34 +02:00
parent 5460041989
commit 7b6d79c133
8 changed files with 748 additions and 1 deletions
+289
View File
@@ -0,0 +1,289 @@
# PeerTube `invalid_grant` incident — diagnosis & ownership
**Date:** 2026-08-18
**Symptom reported:** uploads to PeerTube fail with
`✗ PeerTube auth failed (400): … "code":"invalid_grant"`, after the SSO admins
"updated the SSO for services such as PeerTube and the email server."
**Key premise (from the developer):** the credential set in use
(`xamxam@erg.be` + password) was provided from the organisation's **LDAP**, and
was previously the single credential that worked for both the mail server and
PeerTube — because both services were anchored to that same LDAP directory.
---
## 1. What actually changed (verified)
| # | Fact | Evidence | Verified |
|---|------|----------|----------|
| 1 | The stored credentials are **unchanged**. The SMTP username is still `xamxam@erg.be` on `mail.erg.school:587`. | `SELECT host, port, username FROM smtp_settings` | ✅ |
| 2 | SMTP **still authenticates** with those stored credentials, right now. | `creds-probe.php` → `smtp.ok = true` | ✅ |
| 3 | PeerTube **rejects the same stored credentials** at `POST /api/v1/users/token` with OAuth2 `password` grant → `invalid_grant`. | live `creds-probe.php` → `peertube.ok = false`, code `invalid_grant` | ✅ |
| 4 | Nothing is intercepting/redirecting the request at the HTTP layer. `videos.erg.be` serves PeerTube directly (`nginx`, `x-powered-by: PeerTube`), the token endpoint answers `400` with **0 redirects**. | `curl -w` trace of `/api/v1/users/token` and `GET /` | ✅ |
| 5 | The **long-lived app-token** path (`client_credentials` grant) is **rejected** for the built-in `local` OAuth client (`unsupported_grant_type`). It requires an admin-created application client. | `app-token.sh` live run | ✅ |
| 6 | The manual browser login reportedly now goes through **`portail.erg.be`** (SSO portal). | user observation (login page is a client-rendered SPA; no server-side SSO link in raw HTML) | ⚠️ observed, not server-verified |
**Conclusion of the investigation:** the credential was provided from LDAP and
worked everywhere *before* because both the mail server and PeerTube ultimately
verified against that same LDAP directory. The admins moved **both** onto
`portail.erg.school` SSO — but differently: the mail server kept its `PLAIN`/`LOGIN`
password fallback (so SMTP still works), while PeerTube *removed* its `password`
grant entirely (so the app's `password` grant now returns `invalid_grant` even
though the password is correct).
Nothing in this repo, and nothing about the credential, changed.
---
## 2. Root cause
The app authenticates to PeerTube with an **OAuth2 `password` grant**:
```
POST https://videos.erg.be/api/v1/users/token
grant_type=password
username=xamxam@erg.be
password=<the SMTP password>
```
This grant type relies on PeerTube being able to verify the `xamxam@erg.be`
password **itself** (either a local PeerTube password or a direct bind against
the account's identity directory — historically LDAP).
**Important context:** the single credential set (`xamxam@erg.be` + password)
was provided from the organisation's **LDAP**. Previously *both* the mail server
and PeerTube were anchored to that same LDAP directory, which is why one
credential worked everywhere and why the app was built to reuse it.
The admins moved PeerTube's authentication onto an **SSO / OpenID Connect
provider** (`portail.erg.be`). When PeerTube delegates login to an IdP instead of
the LDAP directory directly:
- the `password` grant can no longer bind to LDAP / verify a local password —
PeerTube now only accepts an SSO-issued identity;
- `POST /api/v1/users/token?grant_type=password` therefore returns
`invalid_grant` ("user credentials are invalid") — even though the very same
username/password still works for the mail server (which is also on SSO but
kept its `PLAIN`/`LOGIN` password fallback) and still works when you sign in
through the SSO web flow.
In other words: **the credential was never wrong, and still isn't.** What was
removed is the *path* the app used — the direct LDAP/password grant — in favour
of the SSO IdP. The app kept using the retired path.
---
## 3. Ownership
**This failure is admin-side, not a defect in this application — and not
fixable with the credential you were given.**
- The app's configuration (instance URL, channel, credentials) is unchanged.
- The stored password is still valid (proven by SMTP).
- The application made no code change that could cause this.
- You were handed a single LDAP credential that *previously* was sufficient
exactly because both services were LDAP-backed. The admins retired the
LDAP-direct path on PeerTube; they did not change your credential.
- Therefore there is no credential value you can type that will make the
`password` grant succeed — the missing piece is a *method* (local/LDAP
password grant), not a *password*.
**One honest caveat (not blame, but worth stating):** the app's *design* chose to
reuse the SMTP password as the PeerTube login and to use the `password` grant.
That coupling means an authentication change on the PeerTube/SSO side will always
surface here. That is a robustness gap on our side, but it is **not the trigger** —
the trigger was the admin-side SSO/LDAP change.
**What we need from the admins (either/or):**
1. **Re-enable the API `password` grant** for an account, or provision a
**local PeerTube account** (separate from SSO) whose password the app can use, **or**
2. **Create an application OAuth client** (PeerTube `create-client`) and hand us
`client_id`/`client_secret`, so the app can switch to the long-lived
`client_credentials` grant, **or**
3. **Expose the SSO IdP's OIDC endpoints**, so the app can authenticate against
`portail.erg.be` directly instead of PeerTube's local password grant.
---
## 3.5 Service topology (DNS + IdP discovery, verified)
| Host | IP | Reverse DNS | Identity |
|------|-----|-------------|----------|
| `videos.erg.be` | `194.78.61.186` | Belgacom static ADSL (on-prem PeerTube) | PeerTube app (`x-powered-by: PeerTube`) |
| `mail.erg.school` | `79.99.201.114` | `mail.erg.school` | **Mailcow** (`MCSESSID` cookie) |
| `portail.erg.school` | `79.99.201.119` | none | **LemonLDAP::NG SSO** (`trspan="authPortal"`, CAS + OIDC) |
Note: the developer referred to it as `portail.erg.be`, but the real host is
**`portail.erg.school`** (`portail.erg.be` does not resolve).
The SSO portal is a **LemonLDAP::NG** instance and doubles as an **OIDC provider**,
confirmed by its `.well-known/openid-configuration`:
```
issuer: https://portail.erg.school/
authorization_endpoint: https://portail.erg.school/oauth2/authorize
token_endpoint: https://portail.erg.school/oauth2/token
userinfo_endpoint: https://portail.erg.school/oauth2/userinfo
response_types_supported: ["code"]
grant_types_supported: ["authorization_code", "refresh_token"]
token_endpoint_auth_methods: ["client_secret_post", "client_secret_basic"]
```
**The decisive fact:** the IdP supports **only `authorization_code` + `refresh_token`**
and **only `response_type=code`**.
- ❌ No `password` grant (cannot exchange username/password headlessly).
- ❌ No `client_credentials` grant (so the "long-lived app token" idea is
**unsupported by this IdP** — PeerTube's own `local` client already rejected it).
- ❌ No dynamic client registration (`/oauth2/register` serves HTML, not an API).
- ❌ No device-authorization flow.
`authorization_code` is an **interactive browser** workflow (redirect to the
portal, human login, redirect back with a `code`, then exchange). A headless
backend upload job cannot complete it by itself.
### Why SMTP still works but PeerTube does not (both are SSO now)
`mail.erg.school` **is also on the new SSO** — verified from its post-STARTTLS
SMTP capabilities:
```
AUTH PLAIN LOGIN XOAUTH2 OAUTHBEARER PLAIN LOGIN XOAUTH2 OAUTHBEARER
```
The presence of `XOAUTH2` / `OAUTHBEARER` proves the mail server was wired up
for SSO/OAuth2 SMTP auth. **But it kept `PLAIN` and `LOGIN` alongside**; the app
authenticates with `AuthType = 'PLAIN'` (`SmtpRelay.php:229`), which still works
against the remaining LDAP/legacy passdb.
The two migrations were different in kind:
| Service | SSO added | Legacy password path | App's method | Outcome |
|---------|-----------|----------------------|--------------|---------|
| `mail.erg.school` | ✅ `XOAUTH2`/`OAUTHBEARER` | ✅ **kept** `PLAIN`/`LOGIN` | `PLAIN` | works |
| `videos.erg.be` | ✅ OIDC `authorization_code` | ❌ **removed** `password` grant | `password` grant | broken |
The mail migration was **additive** (SSO *alongside* password login); the PeerTube
migration was a **hard cutover** (SSO *replaced* the password grant). That one
difference is why the same credential works for mail and fails for PeerTube.
---
## 4. How to solve it
**Honest headline: there is no way to fix this with just the LDAP credential you
already hold, because the IdP offers no non-interactive grant type** (no
`password`, no `client_credentials`, no device flow). Every viable fix requires
an admin action first.
### Short-term — unblock (admin action, no code)
Have the admins restore a **local/LDAP-direct `password` grant** on PeerTube for
the `xamxam@erg.be` account (i.e. make PeerTube verify the password itself again), **or**
provision a dedicated local PeerTube account whose password the app can use.
This is the only option that needs *no* app code change.
### Mid-term — OIDC `authorization_code` + `refresh_token` (the SSO-first fix)
This is the **correct** path now that the IdP is known to be `portail.erg.school`
( LemonLDAP::NG OIDC, only `authorization_code`/`refresh_token`). It is **not**
headless-able in one shot — it needs a one-time human login in the browser to get
the first `refresh_token`, after which the app can keep refreshing indefinitely
without a human.
Concretely the admins must do **two small things**:
1. **Register an OIDC client** for this app on `portail.erg.school` (there is no
self-service `/oauth2/register`, so an admin creates it) and give us a
`client_id` + `client_secret`.
2. Give us a **one-time authorization** (the login on the portal) so we can
exchange the `code` for an `access_token` + a **`refresh_token`**.
Then we store the `refresh_token` (encrypted, like the SMTP password already is)
and change `PeerTubeService::obtainToken()` to:
- refresh via `POST https://portail.erg.school/oauth2/token` (`grant_type=refresh_token`), and
- use the resulting token the way it uses the current one.
This is genuinely SSO-aware and survives password rotations, but it still needs
an admin to register the client and one interactive login to seed the refresh
token.
### The earlier "long-lived app token" idea is now ruled out
`client_credentials` is **not supported** by this IdP (verified via discovery),
and PeerTube's own `local` client already rejected it. Drop it as an option; the
`scripts/app-token.sh` probe is kept only as a diagnostic for non-SSO PeerTube
deployments.
### How to demonstrate this to the admins (copy-paste commands)
Every claim in this report is reproducible with just `curl` — no credentials
leaked, no app code involved. Run these and paste the outputs.
**1. The IdP is the SSO (LemonLDAP OIDC), and it does NOT offer a `password` or
`client_credentials` grant.**
```bash
curl -sS https://portail.erg.school/.well-known/openid-configuration | jq '{issuer, grant_types_supported, response_types_supported, token_endpoint, authorization_endpoint}'
# → "grant_types_supported": ["authorization_code", "refresh_token"]
# (no "password", no "client_credentials")
```
**2. PeerTube itself rejects the old password grant** (this is the `invalid_grant`
the app sees — note: the credentials are *not* shown, only the server's reply).
```bash
# fetch the PeerTube local OAuth client (public endpoint)
curl -sS https://videos.erg.be/api/v1/oauth-clients/local | jq .
# ask PeerTube for a token via the password grant (it refuses)
curl -sS -X POST https://videos.erg.be/api/v1/users/token \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'grant_type=password&response_type=code&client_id=<client_id>&client_secret=<client_secret>&username=xamxam@erg.be&password=<PASSWORD>' \
| jq .
# → 400 { "code": "invalid_grant", "detail": "Invalid grant: user credentials are invalid" }
```
**3. The mail server is *also* on SSO, but kept the legacy `PLAIN`/`LOGIN`
password path — which is why SMTP still works.** Show its advertised AUTH
mechanisms after STARTTLS:
```bash
python3 - <<'EOF'
import smtplib, ssl
s = smtplib.SMTP("mail.erg.school", 587, timeout=20)
s.ehlo(); s.starttls(context=ssl.create_default_context())
code, _ = s.ehlo()
print("AUTH mechanisms:", s.esmtp_features.get("auth", "NONE"))
s.quit()
EOF
# → "AUTH ... PLAIN LOGIN XOAUTH2 OAUTHBEARER ..."
# XOAUTH2/OAUTHBEARER = SSO added; PLAIN/LOGIN = legacy kept
```
**4. The one-sentence version to put in an email to the admins:**
> PeerTube now authenticates only through `portail.erg.school` (OIDC), and its
> old `password` grant has been removed, so backend uploads fail with
> `invalid_grant`. The mail server kept its `PLAIN`/`LOGIN` password fallback,
> which is why SMTP still works. To restore uploads, either re-enable a
> password/LDAP grant for the `xamxam@erg.be` account on PeerTube, **or** register
> an OIDC client for this app and let us use `authorization_code` + `refresh_token`.
---
## 5. Tooling produced during this investigation
| Tool | Purpose |
|------|---------|
| `scripts/creds-probe.php` | Reads stored creds, probes SMTP AUTH + PeerTube password grant, prints JSON |
| `scripts/creds-test.sh` | gum UI wrapper; logs results (never the password); `just creds-test` |
| `scripts/app-token.sh` | gum probe for the long-lived `client_credentials` app-token path; `just app-token` |
| `PeerTubeService::probeAuth()` | Public helper isolating token issuance from channel resolution |
All live runs are reproducible:
```bash
just creds-test # proves SMTP ok, PeerTube invalid_grant
just app-token # proves client_credentials rejected for the local client
```