Add opt-in GitHub Copilot SDK transport for @theia/ai-copilot

[jdubois]

What it does

Project leads reported that the @theia/ai-copilot integration "works, but only older models (e.g. GPT-4o) are available to users." This PR adds an opt-in transport that routes the extension through the official Copilot CLI via @github/copilot-sdk, as an alternative to the existing direct-REST path, so model discovery and chat go through a first-party recognized Copilot integration.

The new path is gated behind a preference (ai-features.copilot.useSdk, default off), so existing behavior is unchanged unless a user explicitly enables it. Scope is deliberately narrow:

• Model discovery via CopilotClient.listModels() (returns a curated, deduplicated chat-model list with policy state).
• Single-turn streaming chat, with all agent tools disabled and permission requests rejected. The CLI owns its own tool loop, which is incompatible with Theia's "model is a function" contract where Theia drives the tool loop, so multi-turn tool use and structured output are intentionally not supported on this path.

Design notes:

• New node-side pieces: copilot-sdk-mappers.ts (pure, unit-tested model-id selection + message-to-prompt flattening; type-only SDK import), copilot-sdk-client-provider.ts (token-keyed CLI client lifecycle), and copilot-sdk-language-model.ts (bridges SDK session events to Theia stream parts, terminating on session.idle / session.error / cancellation).
• The manager branches model discovery and the concrete LanguageModel implementation on the useSdk flag; the frontend rebuilds the Copilot model set when the preference is toggled.

Honest caveat on the root cause (please read): A live experiment during development calls into question the simple "REST transport is limited to old models" framing. Calling the raw REST endpoint GET https://api.githubcopilot.com/models with a capable gho_ token already returns the full modern lineup (gpt-5.5, claude-opus-4.8, gemini-3.1-pro-preview, ...), in fact more entries than listModels(). The two paths differ mainly in curation (REST returns the raw catalog including embeddings and legacy IDs; the SDK returns a curated chat list). This strongly suggests the "only GPT-4o" symptom is tied to the entitlement of Theia's own OAuth App (clientId: Ov23liS2vINy9VOAweyv) rather than the REST-vs-SDK transport itself. The SDK likely helps in production because the @github/copilot CLI authenticates as a first-party integration (token exchange + integration headers), but this has not been confirmed against a token minted by Theia's OAuth App.

How to test

Offline verification (done):

• npx lerna run compile --scope @theia/ai-copilot
• npx lerna run lint --scope @theia/ai-copilot
• npx lerna run test --scope @theia/ai-copilot (29 passing, including 10 new copilot-sdk-mappers unit tests)

Manual / interactive:

1. Sign in via the Copilot status bar (existing flow).
2. Set "ai-features.copilot.useSdk": true in preferences.
3. Confirm the Copilot model list refreshes and includes current models, and that a single-turn chat request streams a response.
4. Toggle the preference off to confirm it reverts to the REST path.

Note: the live chat path could not be end-to-end tested in CI here (requires an authenticated CLI). The model-discovery comparison above was validated with a standalone listModels() run.

Follow-ups

• Confirm the actual root cause by minting a token from Theia's OAuth App and calling /models with it, to verify whether that app is under-entitled.
• Evaluate a lighter-weight alternative: keep the REST path but add the copilot_internal/v2/token exchange plus Copilot-Integration-Id / Editor-Version headers (and/or a registered integration id). This would avoid the heavy @github/copilot CLI dependency (native binaries for all platforms are added to the lockfile).
• If this path is pursued further: multi-turn support, tool calling, and structured output; and a shared per-user CLI client instead of one process per backend connection.
• This branch lives on a fork; the commit still needs a Signed-off-by line to satisfy the Eclipse Contributor Agreement (ECA) check.

Breaking changes

• This PR introduces breaking changes and requires careful review. If yes, the breaking changes section in the changelog has been updated.

N/A. The new behavior is fully opt-in and off by default.

Attribution

N/A

Review checklist

• As an author, I have thoroughly tested my changes and carefully followed the review guidelines
• User-facing text is internationalized using the nls service (for details, please see the Internationalization/Localization section in the Coding Guidelines)

Reminder for reviewers

• As a reviewer, I agree to behave in accordance with the review guidelines

[planger]

Thanks a lot for putting this together. I haven't gone through the details properly yet, but wanted to reply right away to say we really appreciate your thorough investigation and write-up.

We'll also keep looking into the OAuth App entitlement angle and follow up on that, since that caveat seems like it could point to the real root cause.

I'll come back with more substantive feedback once I've had a proper look. Thanks again!