Skip to content

Evaluate migrating tests/support/mcp.py to the official mcp PyPI SDK #74

Description

@WilliamBergamin

Background

The eval suite talks to the Slack MCP server (https://mcp.slack.com/mcp) through a hand-rolled, stdlib-only Streamable HTTP JSON-RPC client at tests/support/mcp.py (MCPClient). It's ~130 lines, synchronous, and dependency-free, which suits the sync pytest suite well.

This issue tracks evaluating whether to replace it with the official mcp PyPI SDK (streamablehttp_client + ClientSession).

Spun out of PR #70 review: the question came up while investigating a 401 Unauthorized failure. Worth noting up front — the 401 was an expired/invalid token, not a client bug, so the SDK migration would not have prevented it. That failure is being addressed separately (clearer 401/403 error message + token docs).

Trade-offs

For migrating:

  • Spec-compliant transport, maintained upstream (handles protocol/version drift, SSE edge cases, session lifecycle for us).
  • Less bespoke code to maintain.

Against / cost:

  • The SDK is async (asyncio); our pytest eval suite is synchronous, so we'd need an asyncio.run(...) wrapper around tool-listing.
  • Adds a runtime dependency (mcp, which pulls in httpx/anyio/pydantic) to a suite that is currently stdlib-only for this path.
  • Does not address token expiry — the root cause of the reported 401.

Suggested outcome

Prototype the tool-listing call (list_tools) against mcp's streamablehttp_client(url, headers={"Authorization": f"Bearer {token}"}) + ClientSession, wrapped for sync use, and compare against the current MCPClient on reliability and complexity. Decide keep-vs-migrate from there.

Co-Authored-By: Claude svc-devxp-claude@slack-corp.com

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions