Skip to content

Replace Twisted with aiohttp and native asyncio#454

Open
sandhose wants to merge 9 commits intoquenting/modern-tooling/rufffrom
quenting/modern-tooling/aiohttp
Open

Replace Twisted with aiohttp and native asyncio#454
sandhose wants to merge 9 commits intoquenting/modern-tooling/rufffrom
quenting/modern-tooling/aiohttp

Conversation

@sandhose
Copy link
Copy Markdown
Member

@sandhose sandhose commented Apr 2, 2026

Summary

  • Replace Twisted's HTTP server (Resource/Site/NOT_DONE_YET) with aiohttp.web
  • Replace Twisted's HTTP client (ProxyAgent/readBody/FileBodyProducer) with aiohttp.ClientSession
  • Replace Twisted reactor with asyncio.run() + aiohttp.web.AppRunner
  • Replace DeferredSemaphore with asyncio.Semaphore
  • Use aioapns native proxy_host/proxy_port instead of ProxyingEventLoopWrapper
  • Migrate tests from twisted.trial to pytest + pytest-asyncio + pytest-aiohttp
  • Add Pushkin.close() lifecycle method for proper session cleanup on shutdown
  • Add ApnsPushkin.close() to shut down the aioapns connection pool
  • Harden shutdown: wrap individual pushkin.close() in try/except
  • Fix PUSHGATEWAY_HTTP_RESPONSES_COUNTER to count 400 responses (early returns)
  • Delete Twisted proxy agent, CONNECT protocol, TLS context factory, and all related test helpers
  • Clean up stale mypy.ini sections referencing deleted files
  • Use CIMultiDict for DummyResponse.headers so .getall() works correctly

Known limitations

  • os.environ["HTTPS_PROXY"] is set as a global side-effect for the Google Auth session proxy. Pre-existing pattern, carried over.
  • No explicit SIGTERM handler — relies on asyncio.run() default behavior. May need attention for container deployments.
  • Proxy unit tests deleted with no replacement (integration test via Docker compose remains).
  • test_overlong_requests_are_rejected deleted — client_max_size enforces the limit but is not tested.

Test plan

  • uv run pytest tests/ — 55 tests pass
  • CI green: linting, type checking, unit tests, Docker build + proxy check

Part 3 of 7 in the repository modernisation series. Builds on #453.

@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch from 7ba52a0 to 8955880 Compare April 2, 2026 12:36
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch from a7c093c to 1bb8aa4 Compare April 2, 2026 12:36
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch from 8955880 to e5cab6f Compare April 2, 2026 16:53
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch from 1bb8aa4 to 5f86b8d Compare April 2, 2026 16:53
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch from e5cab6f to c89a35e Compare April 2, 2026 17:29
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch from 5f86b8d to 6a1be12 Compare April 2, 2026 17:29
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch from c89a35e to fcf8ce2 Compare April 2, 2026 17:39
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch from 65d3fa2 to cb6f287 Compare April 2, 2026 17:58
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch 2 times, most recently from 5403098 to d0cea1d Compare April 2, 2026 22:18
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch 2 times, most recently from 78aadaa to f96764e Compare April 2, 2026 22:38
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch from d0cea1d to e0a99f3 Compare April 2, 2026 22:38
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch from f96764e to e1c6876 Compare April 2, 2026 22:48
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch 5 times, most recently from d7e0207 to 5a1ae0f Compare April 3, 2026 10:21
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch from e1c6876 to 8713ae8 Compare April 3, 2026 10:21
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch from 5a1ae0f to da1e07e Compare April 3, 2026 10:22
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch from 8713ae8 to 06655a7 Compare April 3, 2026 10:22
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch 2 times, most recently from 1f402da to f5c3e91 Compare April 3, 2026 12:06
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch from 06655a7 to 9037622 Compare April 3, 2026 12:06
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch from f5c3e91 to 2d16192 Compare April 3, 2026 12:11
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch 2 times, most recently from 8da7963 to ea0f3c9 Compare April 3, 2026 13:24
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch from 2d16192 to 2f2a872 Compare April 3, 2026 13:24
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch from 2f2a872 to be6956e Compare April 3, 2026 13:28
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch from ea0f3c9 to 9d22a3d Compare April 3, 2026 13:28
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch from be6956e to e6b6eb8 Compare April 3, 2026 13:33
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch from 9d22a3d to 1b905ff Compare April 3, 2026 13:33
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch from e6b6eb8 to a01ee4d Compare April 3, 2026 13:40
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch 2 times, most recently from 4a2c52e to 5b6b2c5 Compare April 3, 2026 13:49
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch 2 times, most recently from 44fd78f to 571943d Compare April 3, 2026 14:00
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch from 5b6b2c5 to 2e5242f Compare April 3, 2026 14:00
@sandhose sandhose mentioned this pull request Apr 3, 2026
3 tasks
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch from 571943d to 38323bd Compare April 3, 2026 14:10
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch from 2e5242f to 59748b5 Compare April 3, 2026 14:21
sandhose and others added 9 commits April 3, 2026 16:29
Trial used to run tests in a temp directory, but pytest does not.
Use tempfile.mkdtemp() so generated cert files don't pollute the
project root.
Remove the Twisted dependency entirely, replacing it with:
- aiohttp.web for the HTTP server
- aiohttp.ClientSession for HTTP clients (GCM, WebPush)
- asyncio.Semaphore for concurrency limiting
- asyncio.sleep for delays
- aioapns native proxy_host/proxy_port for APNS proxy support
- pytest + pytest-asyncio + pytest-aiohttp for tests

This deletes the custom Twisted proxy agent, CONNECT protocol,
event loop wrapper, and TLS context factory — all replaced by
aiohttp's built-in proxy support and ssl.SSLContext defaults.
The retry tests (test_api_v1_retry, test_retry_on_5xx) were sleeping
for real (70s and 30s respectively) because asyncio.sleep replaced
the old twisted_sleep which used a fake reactor clock. Also reduce
the concurrency test sleep from 1s to 10ms.
Rename TestCredentials, TestGcmPushkin, and TestPushkin to
StubCredentials, StubGcmPushkin, and StubPushkin so pytest doesn't
try to collect them as test classes. Also filter the google-auth
AuthorizedSession deprecation warning.
Wire pytest's tmp_path through the base TestCase fixture so tests
can use it for temporary files. Migrate the GCM service account
file to use tmp_path instead of NamedTemporaryFile.
Close aiohttp.ClientSession instances properly on shutdown in GCM
and WebPush pushkins. Add Pushkin.close() base method and call it
from Sygnal._start()'s finally block.
- Remove mypy.ini sections referencing deleted files (proxyagent_twisted,
  twisted_test_helpers, asyncio_test_helpers, test_httpproxy_*)
- Use CIMultiDict for DummyResponse.headers so .getall() works if tests
  ever exercise the retry/5xx code path
- Wrap individual pushkin.close() calls in try/except during shutdown so
  one failure doesn't prevent cleanup of remaining pushkins

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Track status_code through early returns and increment
  PUSHGATEWAY_HTTP_RESPONSES_COUNTER in the finally block for
  requests that don't reach _handle_dispatch.
- Add close() to ApnsPushkin to shut down the aioapns connection
  pool on shutdown.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@sandhose sandhose force-pushed the quenting/modern-tooling/aiohttp branch from fb6c10c to 475082f Compare April 3, 2026 14:30
@sandhose sandhose force-pushed the quenting/modern-tooling/ruff branch from 59748b5 to 184a6e9 Compare April 3, 2026 14:30
@sandhose sandhose marked this pull request as ready for review April 3, 2026 14:47
@sandhose sandhose requested a review from a team as a code owner April 3, 2026 14:47
csr_filename = "server.csr"
cnf_filename = "server.%i.cnf" % (cert_file_count,)
cert_filename = "server.%i.crt" % (cert_file_count,)
tmpdir = tempfile.mkdtemp(prefix="sygnal-test-certs-")
Copy link
Copy Markdown
Contributor

@reivilibre reivilibre Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something I wound up with when trying to pytestify Synapse:

$ cat tests/conftest.py
import os
import tempfile
from collections.abc import Iterator

import pytest


@pytest.fixture(scope="session", autouse=True)
def _run_tests_in_temporary_working_directory() -> Iterator[None]:
    """Run pytest from a temporary working directory.

    The test suite contains many trial-style tests which write temp files relative
    to the current working directory. Running from a temporary directory keeps the
    repository worktree clean when running under pytest.
    """

    old_cwd = os.getcwd()
    with tempfile.TemporaryDirectory(prefix="synapse-pytest-") as tempdir:
        os.chdir(tempdir)
        try:
            yield
        finally:
            os.chdir(old_cwd)

I think it might be as elegant as it gets and also cleans up the dir?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants