thebeast: black screen + 720p drop on compositor transitions is a DP/DSC link retrain (not a mode mismatch)

[jasonboukheir]

Summary

Transitioning systemd-boot → gamescope (Steam) → SDDM/kwin greeter → Hyprland on
thebeast produces a black screen and a momentary drop to a low ("720p")
resolution at every step — even though all four stages are configured with
identical mode timings, bit depth, and colorspace via thebeast.displays.

Root cause (confirmed by measurement): a DisplayPort (DSC) link retrain on each
DRM-master handoff — not a mode mismatch. Matching display settings cannot fix
it. The only thing that removes a handoff blank is never releasing the link (one
persistent root compositor with the others nested).

Hardware: Samsung Odyssey G95SC, 5120×1440 @ 120 Hz, 10-bit HDR, DisplayPort
(DP-2), AMD RX 9070 XT (amdgpu, DCN/RDNA4). The link runs HBR2 ×4 with DSC.

Full design notes committed at hosts/thebeast/session/COMPOSITOR-TRANSITIONS.md.

Evidence

Captured during a round-trip Hyprland → greeter → gamescope → greeter → Hyprland
with drm.debug raised to log DPCD/AUX traffic.

1. The source GPU never emits a low resolution. The only non-empty CRTC mode
in the whole capture is mode: "5120x1440": 120 …; otherwise the CRTC is
enable=0 active=0 mode: "" (disabled). No 1280x720 anywhere. ⇒ the "720p" is
the monitor's own no-signal OSD during the retrain, not the GPU. (fb0 is
amdgpudrmfb, pinned by video=DP-2:5120x1440@120.)

2. The DP link is powered down to D3 and fully retrained at each handoff:

amdgpu …: AUX <- 0x600 = 02     # SET_POWER: link → D3 (powered DOWN)
amdgpu …: AUX <- 0x600 = 01     # SET_POWER: link → D0 (powered up)
amdgpu …: AUX <- 0x100 = 14     # LINK_BW_SET = HBR2 (5.4 Gbps/lane)
amdgpu …: AUX <- 0x101 = 84     # LANE_COUNT_SET = 4 lanes + enhanced framing
amdgpu …: AUX <- 0x160 = …      # DSC enable
amdgpu …: AUX <- 0x102 = 21→07→00   # training: clock-recovery, chan-eq, done
amdgpu …: AUX -> 0x202 = 77 77  # LANE_STATUS: CR + EQ + symbol lock achieved

3. Exit-then-start, never a handover. DRM-master holder timeline for
/dev/dri/card1: Hyprland → (gap) → kwin(greeter) → (gap) → gamescope-wl → (gap) → kwin(greeter) → (gap) → Hyprland. Each (gap) = CRTC disabled, link in D3.

Why it's fundamental

• Only one DRM master can own a CRTC at a time; the next client commits its own
  complete atomic state from scratch — nothing is inherited.
  (kernel KMS docs,
  mpv#6184)
• systemd-logind PauseDevices and revokes the old client's DRM fd on a
  session/VT switch, leaving a window where nothing drives the CRTC.
  (Sane Session-Switching)
• "Flicker-free boot" (Plymouth --retain-splash, i915 fastboot, simpledrm,
  amdgpu.seamless) only covers the boot chain and is i915-centric; Fedora
  notes AMD "will still see the monitor turn off briefly."
  (FlickerFreeBoot)
• "Seamless" display managers (GDM, SDDM+Plymouth) use timing-overlap +
  same-mode preservation, not a persistent-compositor handover.
  greetd/regreet are explicitly exit-then-start.
  (KDE thread,
  sddm#1013,
  regreet)
• uwsm only manages the systemd user session/environment; it never touches KMS
  and has no bearing on this.

Ruled out

• Mode mismatch / EDID fallback / fbcon restore — source CRTC only ever holds
  5120x1440@120 or active=0.
• uwsm — orthogonal to the DRM modeset.

Privilege constraint

A nested child is observable (input+output) by its parent compositor, so the
persistent root must be at least as trusted as its most privileged child.
gamer (Steam/gamescope) is unprivileged; jasonbk is privileged. Therefore
jasonbk must not nest under the gamer gamescope, and crossing into
jasonbk is a genuine privilege boundary = a separate DRM master = an
unavoidable retrain. That retrain is acceptable — it's the boundary we want to
be hard.

Decision (current)

Gaming-first, greeter retained. Retrain budget: boot (1) + gaming (0) +
gamescope↔desktop (2 each way). Keeping the standalone greeter means jasonbk's
processes don't start until authentication, at the cost of a second blank on the
occasional sysadmin trip.

Keep the unified thebeast.displays settings — they don't stop the retrain, but
they ensure one retrain per handoff instead of two.

Target architecture (aspirational)

Keep gamescope as the persistent root that owns the DP link; nest the gamer
desktop and greeter inside it (0 retrain); spend the one unavoidable retrain at
the jasonbk boundary.

boot ─retrain─► gamescope (gamer)            ← PERSISTENT ROOT, owns the DP link
                  ├─ Steam / games           nested → 0 retrain
                  ├─ gamer Hyprland (desktop) nested → 0 retrain
                  └─ greeter (launcher)       nested → 0 retrain
                         └─ login jasonbk ─retrain─► jasonbk Hyprland (separate master)
   exit jasonbk Hyprland ─retrain─► back to gamescope (gamer)

⚠️ Security caveat (nested greeter)

If the greeter renders nested inside the unprivileged gamer gamescope, the
gamer compositor (parent) can keylog the jasonbk password. Trade:
nested = 0 retrain but keyloggable; separate-master = protected but 1 retrain.
Must be a conscious choice (likely acceptable on a single-physical-user couch box).

Feasibility (proven vs unproven)

• Persistent gamescope root + nested games — proven.
• Hyprland nested inside gamescope — unproven here (gamescope hosts one
  fullscreen app; SteamOS "switch to desktop" actually exits gamescope, doesn't
  nest). Needs a spike.
• HDR + VRR through the nest — unproven/bleeding-edge; highest risk.
• Nested greeter — feasible (light greeter, e.g. greetd/regreet), modulo the
  security caveat.

TODO

• Spike: Hyprland nested in gamescope on the 9070 XT — latency/tearing/input,
  and does HDR/VRR survive the nest?
• Decide the greeter trade (nested+keyloggable vs separate-master+1 retrain).
• Wire gamer "switch to desktop" + greeter launcher to nest, not exit
  gamescope (Jovian/steamos-session-select defaults to exiting).
• Confirm jasonbk Hyprland retrains exactly once each way (no double retrain).