xamxam/docs/archive/peertube-sso-incident.md

# PeerTube `invalid_grant` incident — diagnosis & ownership

**Date:** 2026-08-18
**Symptom reported:** uploads to PeerTube fail with
`✗ PeerTube auth failed (400): … "code":"invalid_grant"`, after the SSO admins
"updated the SSO for services such as PeerTube and the email server."

**Key premise (from the developer):** the credential set in use
(`xamxam@erg.be` + password) was provided from the organisation's **LDAP**, and
was previously the single credential that worked for both the mail server and
PeerTube — because both services were anchored to that same LDAP directory.

---

## 1. What actually changed (verified)

| # | Fact | Evidence | Verified |
|---|------|----------|----------|
| 1 | The stored credentials are **unchanged**. The SMTP username is still `xamxam@erg.be` on `mail.erg.school:587`. | `SELECT host, port, username FROM smtp_settings` | ✅ |
| 2 | SMTP **still authenticates** with those stored credentials, right now. | `creds-probe.php` → `smtp.ok = true` | ✅ |
| 3 | PeerTube **rejects the same stored credentials** at `POST /api/v1/users/token` with OAuth2 `password` grant → `invalid_grant`. | live `creds-probe.php` → `peertube.ok = false`, code `invalid_grant` | ✅ |
| 4 | Nothing is intercepting/redirecting the request at the HTTP layer. `videos.erg.be` serves PeerTube directly (`nginx`, `x-powered-by: PeerTube`), the token endpoint answers `400` with **0 redirects**. | `curl -w` trace of `/api/v1/users/token` and `GET /` | ✅ |
| 5 | The **long-lived app-token** path (`client_credentials` grant) is **rejected** for the built-in `local` OAuth client (`unsupported_grant_type`). It requires an admin-created application client. | `app-token.sh` live run | ✅ |
| 6 | The manual browser login reportedly now goes through **`portail.erg.be`** (SSO portal). | user observation (login page is a client-rendered SPA; no server-side SSO link in raw HTML) | ⚠️ observed, not server-verified |

**Conclusion of the investigation:** the credential was provided from LDAP and
worked everywhere *before* because both the mail server and PeerTube ultimately
verified against that same LDAP directory. The admins moved **both** onto
`portail.erg.school` SSO — but differently: the mail server kept its `PLAIN`/`LOGIN`
password fallback (so SMTP still works), while PeerTube *removed* its `password`
grant entirely (so the app's `password` grant now returns `invalid_grant` even
though the password is correct).
Nothing in this repo, and nothing about the credential, changed.

---

## 2. Root cause

The app authenticates to PeerTube with an **OAuth2 `password` grant**:

```
POST https://videos.erg.be/api/v1/users/token
grant_type=password
username=xamxam@erg.be
password=<the SMTP password>
```

This grant type relies on PeerTube being able to verify the `xamxam@erg.be`
password **itself** (either a local PeerTube password or a direct bind against
the account's identity directory — historically LDAP).

**Important context:** the single credential set (`xamxam@erg.be` + password)
was provided from the organisation's **LDAP**. Previously *both* the mail server
and PeerTube were anchored to that same LDAP directory, which is why one
credential worked everywhere and why the app was built to reuse it.

The admins moved PeerTube's authentication onto an **SSO / OpenID Connect
provider** (`portail.erg.be`). When PeerTube delegates login to an IdP instead of
the LDAP directory directly:

- the `password` grant can no longer bind to LDAP / verify a local password —
  PeerTube now only accepts an SSO-issued identity;
- `POST /api/v1/users/token?grant_type=password` therefore returns
  `invalid_grant` ("user credentials are invalid") — even though the very same
  username/password still works for the mail server (which is also on SSO but
  kept its `PLAIN`/`LOGIN` password fallback) and still works when you sign in
  through the SSO web flow.

In other words: **the credential was never wrong, and still isn't.** What was
removed is the *path* the app used — the direct LDAP/password grant — in favour
of the SSO IdP. The app kept using the retired path.

---

## 3. Ownership

**This failure is admin-side, not a defect in this application — and not
fixable with the credential you were given.**

- The app's configuration (instance URL, channel, credentials) is unchanged.
- The stored password is still valid (proven by SMTP).
- The application made no code change that could cause this.
- You were handed a single LDAP credential that *previously* was sufficient
  exactly because both services were LDAP-backed. The admins retired the
  LDAP-direct path on PeerTube; they did not change your credential.
- Therefore there is no credential value you can type that will make the
  `password` grant succeed — the missing piece is a *method* (local/LDAP
  password grant), not a *password*.

**One honest caveat (not blame, but worth stating):** the app's *design* chose to
reuse the SMTP password as the PeerTube login and to use the `password` grant.
That coupling means an authentication change on the PeerTube/SSO side will always
surface here. That is a robustness gap on our side, but it is **not the trigger** —
the trigger was the admin-side SSO/LDAP change.

**What we need from the admins (either/or):**

1. **Re-enable the API `password` grant** for an account, or provision a
   **local PeerTube account** (separate from SSO) whose password the app can use, **or**
2. **Create an application OAuth client** (PeerTube `create-client`) and hand us
   `client_id`/`client_secret`, so the app can switch to the long-lived
   `client_credentials` grant, **or**
3. **Expose the SSO IdP's OIDC endpoints**, so the app can authenticate against
   `portail.erg.be` directly instead of PeerTube's local password grant.

---

## 3.5 Service topology (DNS + IdP discovery, verified)

| Host | IP | Reverse DNS | Identity |
|------|-----|-------------|----------|
| `videos.erg.be` | `194.78.61.186` | Belgacom static ADSL (on-prem PeerTube) | PeerTube app (`x-powered-by: PeerTube`) |
| `mail.erg.school` | `79.99.201.114` | `mail.erg.school` | **Mailcow** (`MCSESSID` cookie) |
| `portail.erg.school` | `79.99.201.119` | none | **LemonLDAP::NG SSO** (`trspan="authPortal"`, CAS + OIDC) |

Note: the developer referred to it as `portail.erg.be`, but the real host is
**`portail.erg.school`** (`portail.erg.be` does not resolve).

The SSO portal is a **LemonLDAP::NG** instance and doubles as an **OIDC provider**,
confirmed by its `.well-known/openid-configuration`:

```
issuer:                          https://portail.erg.school/
authorization_endpoint:          https://portail.erg.school/oauth2/authorize
token_endpoint:                  https://portail.erg.school/oauth2/token
userinfo_endpoint:               https://portail.erg.school/oauth2/userinfo
response_types_supported:        ["code"]
grant_types_supported:           ["authorization_code", "refresh_token"]
token_endpoint_auth_methods:     ["client_secret_post", "client_secret_basic"]
```

**The decisive fact:** the IdP supports **only `authorization_code` + `refresh_token`**
and **only `response_type=code`**.

- ❌ No `password` grant (cannot exchange username/password headlessly).
- ❌ No `client_credentials` grant (so the "long-lived app token" idea is
  **unsupported by this IdP** — PeerTube's own `local` client already rejected it).
- ❌ No dynamic client registration (`/oauth2/register` serves HTML, not an API).
- ❌ No device-authorization flow.

`authorization_code` is an **interactive browser** workflow (redirect to the
portal, human login, redirect back with a `code`, then exchange). A headless
backend upload job cannot complete it by itself.

### Why SMTP still works but PeerTube does not (both are SSO now)

`mail.erg.school` **is also on the new SSO** — verified from its post-STARTTLS
SMTP capabilities:

```
AUTH PLAIN LOGIN XOAUTH2 OAUTHBEARER PLAIN LOGIN XOAUTH2 OAUTHBEARER
```

The presence of `XOAUTH2` / `OAUTHBEARER` proves the mail server was wired up
for SSO/OAuth2 SMTP auth. **But it kept `PLAIN` and `LOGIN` alongside**; the app
authenticates with `AuthType = 'PLAIN'` (`SmtpRelay.php:229`), which still works
against the remaining LDAP/legacy passdb.

The two migrations were different in kind:

| Service | SSO added | Legacy password path | App's method | Outcome |
|---------|-----------|----------------------|--------------|---------|
| `mail.erg.school` | ✅ `XOAUTH2`/`OAUTHBEARER` | ✅ **kept** `PLAIN`/`LOGIN` | `PLAIN` | works |
| `videos.erg.be` | ✅ OIDC `authorization_code` | ❌ **removed** `password` grant | `password` grant | broken |

The mail migration was **additive** (SSO *alongside* password login); the PeerTube
migration was a **hard cutover** (SSO *replaced* the password grant). That one
difference is why the same credential works for mail and fails for PeerTube.

---

## 4. How to solve it

**Honest headline: there is no way to fix this with just the LDAP credential you
already hold, because the IdP offers no non-interactive grant type** (no
`password`, no `client_credentials`, no device flow). Every viable fix requires
an admin action first.

### Short-term — unblock (admin action, no code)

Have the admins restore a **local/LDAP-direct `password` grant** on PeerTube for
the `xamxam@erg.be` account (i.e. make PeerTube verify the password itself again), **or**
provision a dedicated local PeerTube account whose password the app can use.
This is the only option that needs *no* app code change.

### Mid-term — OIDC `authorization_code` + `refresh_token` (the SSO-first fix)

This is the **correct** path now that the IdP is known to be `portail.erg.school`
( LemonLDAP::NG OIDC, only `authorization_code`/`refresh_token`). It is **not**
headless-able in one shot — it needs a one-time human login in the browser to get
the first `refresh_token`, after which the app can keep refreshing indefinitely
without a human.

Concretely the admins must do **two small things**:

1. **Register an OIDC client** for this app on `portail.erg.school` (there is no
   self-service `/oauth2/register`, so an admin creates it) and give us a
   `client_id` + `client_secret`.
2. Give us a **one-time authorization** (the login on the portal) so we can
   exchange the `code` for an `access_token` + a **`refresh_token`**.

Then we store the `refresh_token` (encrypted, like the SMTP password already is)
and change `PeerTubeService::obtainToken()` to:
- refresh via `POST https://portail.erg.school/oauth2/token` (`grant_type=refresh_token`), and
- use the resulting token the way it uses the current one.

This is genuinely SSO-aware and survives password rotations, but it still needs
an admin to register the client and one interactive login to seed the refresh
token.

### The earlier "long-lived app token" idea is now ruled out

`client_credentials` is **not supported** by this IdP (verified via discovery),
and PeerTube's own `local` client already rejected it. Drop it as an option; the
`scripts/app-token.sh` probe is kept only as a diagnostic for non-SSO PeerTube
deployments.

### How to demonstrate this to the admins (copy-paste commands)

Every claim in this report is reproducible with just `curl` — no credentials
leaked, no app code involved. Run these and paste the outputs.

**1. The IdP is the SSO (LemonLDAP OIDC), and it does NOT offer a `password` or
`client_credentials` grant.**

```bash
curl -sS https://portail.erg.school/.well-known/openid-configuration | jq '{issuer, grant_types_supported, response_types_supported, token_endpoint, authorization_endpoint}'
# → "grant_types_supported": ["authorization_code", "refresh_token"]
#   (no "password", no "client_credentials")
```

**2. PeerTube itself rejects the old password grant** (this is the `invalid_grant`
the app sees — note: the credentials are *not* shown, only the server's reply).

```bash
# fetch the PeerTube local OAuth client (public endpoint)
curl -sS https://videos.erg.be/api/v1/oauth-clients/local | jq .

# ask PeerTube for a token via the password grant (it refuses)
curl -sS -X POST https://videos.erg.be/api/v1/users/token \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'grant_type=password&response_type=code&client_id=<client_id>&client_secret=<client_secret>&username=xamxam@erg.be&password=<PASSWORD>' \
  | jq .
# → 400 { "code": "invalid_grant", "detail": "Invalid grant: user credentials are invalid" }
```

**3. The mail server is *also* on SSO, but kept the legacy `PLAIN`/`LOGIN`
password path — which is why SMTP still works.** Show its advertised AUTH
mechanisms after STARTTLS:

```bash
python3 - <<'EOF'
import smtplib, ssl
s = smtplib.SMTP("mail.erg.school", 587, timeout=20)
s.ehlo(); s.starttls(context=ssl.create_default_context())
code, _ = s.ehlo()
print("AUTH mechanisms:", s.esmtp_features.get("auth", "NONE"))
s.quit()
EOF
# → "AUTH ... PLAIN LOGIN XOAUTH2 OAUTHBEARER ..."
#   XOAUTH2/OAUTHBEARER = SSO added; PLAIN/LOGIN = legacy kept
```

**4. The one-sentence version to put in an email to the admins:**

> PeerTube now authenticates only through `portail.erg.school` (OIDC), and its
> old `password` grant has been removed, so backend uploads fail with
> `invalid_grant`. The mail server kept its `PLAIN`/`LOGIN` password fallback,
> which is why SMTP still works. To restore uploads, either re-enable a
> password/LDAP grant for the `xamxam@erg.be` account on PeerTube, **or** register
> an OIDC client for this app and let us use `authorization_code` + `refresh_token`.

### The open question to put to the admins (the "❓")

Every fact above is **verifiable from the outside** with `sso-diagnose.sh`. The
one thing it cannot establish remotely is **what authentication information
LemonLDAP actually forwards to PeerTube after it authenticates a user**. That is
infrastructure-side configuration, and it is the one remaining unknown.

Three architectures are possible after the migration; the admins must decide
which one is the *intended* contract for an external app hitting the PeerTube API:

1. **Browser/user SSO** (PeerTube is an OIDC client of LemonLDAP) — the app
   would use the IdP's `authorization_code` + `refresh_token` (interactive, needs
   a one-time human login + a registered OIDC client).
2. **Machine-to-machine** (direct PeerTube API) — needs a non-interactive
   mechanism (a PeerTube service account / API token), because the IdP does **not**
   advertise `client_credentials`.
3. **Reverse-proxy SSO** (LemonLDAP authenticates, forwards identity headers,
   PeerTube trusts them) — needs LemonLDAP to inject the identity (e.g.
   `Auth-User`/`X-Remote-User`) **and** PeerTube configured to consume it.

`request-side identity header probe → SKIPPED` in the diagnostic means we have
**not yet confirmed** that (3) is even wired up — i.e. that PeerTube receives
*who* LemonLDAP authenticated. That is the next thing to pin down.

Copy-paste ask for the admins:

> `videos.erg.be` is now behind LemonLDAP::NG; its OIDC discovery advertises only
> `authorization_code` and `refresh_token` (no `password`, no
> `client_credentials`). Could you confirm what mechanism external applications
> are now supposed to use to authenticate to the PeerTube API — (1) an OIDC
> authorization-code flow through `portail.erg.school`, (2) a forwarded
> authenticated-user identity (and if so, which header), or (3) a separate
> non-interactive service account / API token? And can you verify that the
> LemonLDAP-protected `videos.erg.be` vhost forwards the authenticated identity
> to the PeerTube backend as intended?

### Responsibility boundary (no zero-blame claim)

The *trigger* is unambiguously admin/infrastructure-side: the IdP removed the
`password` grant the app used, while the credential itself stayed valid (SMTP
still authenticates). But the app is **not** blameless in design: it coupled
PeerTube auth to the SMTP password and the `password` grant, so an IdP-side change
surfaces here. And **if** the app must keep speaking to PeerTube, it *will* need
code changes — to implement whichever supported grant the admins specify. Those
changes cannot be written until the admins settle the new authentication contract,
which is the blocking unknown above.

---

## 5. Tooling produced during this investigation

| Tool | Purpose |
|------|---------|
| `scripts/creds-probe.php` | Reads stored creds, probes SMTP AUTH + PeerTube password grant, prints JSON |
| `scripts/creds-test.sh` | gum UI wrapper; logs results (never the password); `just creds-test` |
| `scripts/app-token.sh` | gum probe for the long-lived `client_credentials` app-token path; `just app-token` |
| `scripts/sso-diagnose.sh` | Verifies every claim above (DNS/rDNS, IdP discovery, PeerTube OAuth + password grant, SMTP AUTH, identity-header propagation) into `sso-diagnose.log`; `just sso-diagnose` |
| `scripts/echo-headers.php` | Request-side echo endpoint to reveal the identity header LemonLDAP injects; pair with `--echo <url>` |
| `PeerTubeService::probeAuth()` | Public helper isolating token issuance from channel resolution |

All live runs are reproducible:

```bash
just creds-test      # proves SMTP ok, PeerTube invalid_grant
just app-token       # proves client_credentials rejected for the local client
just sso-diagnose    # full reproducible report → sso-diagnose.log (no secrets)
# the one remaining unknown (who LemonLDAP forwards to PeerTube) needs
# scripts/echo-headers.php hosted behind the SAME LemonLDAP vhost:
#   bash scripts/sso-diagnose.sh --echo 'https://videos.erg.be/path/to/echo-headers.php'
```