Summary
On a no_std ESP32-S3 application using Embassy, TlsConnection::open().await can prevent every other task on the same executor from being polled for roughly 6.7-7.3 seconds while connecting to github.com.
The socket-facing TLS API is asynchronous, but certificate and handshake-signature verification are synchronous calls inside the async handshake state machine. On a small MCU, the resulting CPU-bound verification can therefore starve unrelated timers and network tasks long enough to cause WebSocket disconnects and other expired timeouts.
This is distinct from a slow or timed-out network request: in the instrumented reproduction below, DNS took 16 ms, TCP connect took 90 ms, and the HTTP response was read in 5 ms. The TLS handshake took 9,108 ms wall-clock time and contained a continuous 6,845 ms interval in which the executor did not poll a separate sentinel task.
Environment
embedded-tls 0.19.0- Features:
rustpki, rsa, p384, alloc, log; default features disabled
- Target:
xtensa-esp32s3-none-elf, ESP32-S3 at 240 MHz
- Runtime/network stack: Embassy /
embassy-net, no_std
- The TLS task and latency-sensitive HTTP/WebSocket/device-poll tasks share one
Embassy executor
- Optimized size build (
opt-level = "s"); debug assertions and overflow checks
are disabled for embedded-tls and p384
- Endpoint:
github.com:443, TLS 1.3 with a P-256 key share
- Served certificate chain at the time of testing contained ECDSA/SHA-256 and
ECDSA/SHA-384 signatures (Sectigo E36/E46 chain)
- Wi-Fi RSSI was -44 dBm, with zero Wi-Fi reconnects recorded
One diagnostic run used a local copy of the exact 0.19.0 crate with an optional RSA/SHA-256 verifier injection hook. That hook does not alter the ECDSA code path exercised by github.com, and it recorded no hardware-RSA operations during this request. Earlier runs using the normal software verifier showed the same 6.7-7.3-second executor gaps.
Reproduction
- Run a periodic Embassy task which records the elapsed time between successive polls. In this application the sentinel normally runs every 10 ms and reports any gap over 100 ms.
- Open a TCP socket to
github.com:443.
- Construct
TlsConnection with the rustpki CertVerifier, a valid clock and the appropriate CA certificate.
- Mark the current diagnostic activity as
tls-handshake, call tls.open(TlsContext::new(&config, provider)).await, and clear the marker when it returns.
- Keep unrelated periodic and network tasks active on the same executor.
The relevant application operation is equivalent to:
let mut tls = TlsConnection::new(socket, read_buffer, write_buffer);
let started = Instant::now();
set_activity("tls-handshake");
let result = tls.open(TlsContext::new(&config, provider)).await;
clear_activity();
log_handshake_time(started.elapsed());The diagnostic marker covers the whole tls.open() call, so it does not by itself identify one particular cryptographic primitive. It does establish that the executor gap occurs inside the TLS handshake rather than DNS, TCP connect, HTTP transmission, or waiting for an HTTP response.
Results
The problem has occurred repeatedly:
| Run |
Longest interval without polling the sentinel |
Result after handshake |
| 1 |
7,318 ms |
Server returned an HTTP error |
| 2 |
6,832 ms |
Server returned an HTTP error |
| 3 |
6,734 ms |
Request eventually timed out |
| 4, with phase timings |
6,845 ms |
Server returned an HTTP error |
Detailed timings from run 4:
DNS: 16 ms
TCP connect: 90 ms
TLS handshake: 9,108 ms
HTTP write: 942 ms
HTTP read: 5 ms
Total: 11,381 ms
Longest executor poll gap: 6,845 ms
Activity during gap: tls-handshake
Wi-Fi RSSI: -44 dBm
Wi-Fi reconnects: 0
Representative log cascade (from prototype embassy project):
ERROR - timeout waiting for poll()
WARN - STALL: executor blocked 6845 ms (activity tls-handshake)
WARN - esp_wifi_internal_tx returned error: 257
WARN - WS send timeout - closing
ERROR - serve error: WriteTimeout(TimeoutError)
WARN - OTA-pull: manifest fetch failed: the server returned an HTTP error
The missing OTA manifest explains the final non-2xx HTTP result, but not the handshake stall: a TLS connection and certificate verification are still performed before the 5 ms HTTP response read.
Why this appears to happen
At upstream commit 27874162c9d9105701d2b588092f39311e34bd93:
Therefore, although TlsConnection::open() can yield while waiting for socket I/O, it cannot yield while these verifier methods execute. An async timeout wrapped around tls.open() also cannot preempt a long verifier call, because the timeout future is not polled until control returns to the executor.
The observed 6,845 ms interval may include certificate parsing, chain traversal, ECDSA verification, key agreement, hashing, or other synchronous handshake work. More internal timing points would be required to assign the complete interval to one operation. The API-level inability to yield during verifier work is independent of that attribution.
Expected behavior
An application using the async API should be able to perform a standards-compliant, fully verified TLS handshake without preventing unrelated tasks on the same executor from running for several seconds.
Certificate verification must continue to fail closed. Disabling certificate or hostname verification is not an acceptable workaround.
Possible solutions
Would it be worth considering one or more of the following?
-
An async verifier interface. Add an async counterpart to TlsVerifier, so verify_certificate and verify_signature can await a hardware peripheral, another executor task, or work offloaded to another core. This could use async trait methods/RPITIT, associated future types, or a separate AsyncTlsVerifier trait to avoid breaking the blocking API.
-
A poll-based verifier state machine. Let verification return a pending state and resume through poll_verify. This avoids requiring allocation or boxed futures and may fit the crate's no_std goals.
-
Break native chain verification into cooperative steps. Yield between certificate signatures and before/after CertificateVerify. This would not make an individual monolithic RustCrypto operation interruptible, but it could reduce the longest executor gap for multi-certificate chains.
-
Extend crypto-provider injection to public-key verification. Issue #123 already tracks hardware-accelerated crypto. Allowing custom P-256/P-384/RSA certificate verifiers would reduce CPU time and, combined with an async verifier contract, permit genuinely non-blocking peripheral or multicore implementations.
-
Document the scheduling limitation. If cooperative verification is not currently practical, document that the async API performs CPU-bound cryptography synchronously and that socket/handshake timeouts cannot pre-empt it. This lets single-executor applications plan explicit isolation.
Merely making the verifier method async will not by itself make a direct call to RustCrypto cooperative; the implementation would still need incremental work, hardware async support, or offloading. The value of an async contract is that it would make those implementations possible without blocking the handshake API.
Related upstream work
#123 addresses throughput and replaceable crypto, whereas this report is also about scheduling/cooperativeness: even accelerated synchronous verification can still monopolize an executor, and an application cannot currently supply a verifier that awaits offloaded work.
I can provide the complete diagnostic log, phase-timing patch, certificate survey, or help test an async/poll-based verifier design on ESP32-S3 if useful?
Summary
On a
no_stdESP32-S3 application using Embassy,TlsConnection::open().awaitcan prevent every other task on the same executor from being polled for roughly 6.7-7.3 seconds while connecting togithub.com.The socket-facing TLS API is asynchronous, but certificate and handshake-signature verification are synchronous calls inside the async handshake state machine. On a small MCU, the resulting CPU-bound verification can therefore starve unrelated timers and network tasks long enough to cause WebSocket disconnects and other expired timeouts.
This is distinct from a slow or timed-out network request: in the instrumented reproduction below, DNS took 16 ms, TCP connect took 90 ms, and the HTTP response was read in 5 ms. The TLS handshake took 9,108 ms wall-clock time and contained a continuous 6,845 ms interval in which the executor did not poll a separate sentinel task.
Environment
embedded-tls0.19.0rustpki,rsa,p384,alloc,log; default features disabledxtensa-esp32s3-none-elf, ESP32-S3 at 240 MHzembassy-net,no_stdEmbassy executor
opt-level = "s"); debug assertions and overflow checksare disabled for
embedded-tlsandp384github.com:443, TLS 1.3 with a P-256 key shareECDSA/SHA-384 signatures (Sectigo E36/E46 chain)
One diagnostic run used a local copy of the exact 0.19.0 crate with an optional RSA/SHA-256 verifier injection hook. That hook does not alter the ECDSA code path exercised by
github.com, and it recorded no hardware-RSA operations during this request. Earlier runs using the normal software verifier showed the same 6.7-7.3-second executor gaps.Reproduction
github.com:443.TlsConnectionwith therustpkiCertVerifier, a valid clock and the appropriate CA certificate.tls-handshake, calltls.open(TlsContext::new(&config, provider)).await, and clear the marker when it returns.The relevant application operation is equivalent to:
The diagnostic marker covers the whole
tls.open()call, so it does not by itself identify one particular cryptographic primitive. It does establish that the executor gap occurs inside the TLS handshake rather than DNS, TCP connect, HTTP transmission, or waiting for an HTTP response.Results
The problem has occurred repeatedly:
Detailed timings from run 4:
Representative log cascade (from prototype embassy project):
The missing OTA manifest explains the final non-2xx HTTP result, but not the handshake stall: a TLS connection and certificate verification are still performed before the 5 ms HTTP response read.
Why this appears to happen
At upstream commit
27874162c9d9105701d2b588092f39311e34bd93:TlsVerifier::verify_certificateandverify_signatureare synchronous methods.process_server_verifyinvokes both methods directly, without an.awaitpoint.CertificateVerifymessage(source)
and while traversing the certificate chain
(source).
Therefore, although
TlsConnection::open()can yield while waiting for socket I/O, it cannot yield while these verifier methods execute. An async timeout wrapped aroundtls.open()also cannot preempt a long verifier call, because the timeout future is not polled until control returns to the executor.The observed 6,845 ms interval may include certificate parsing, chain traversal, ECDSA verification, key agreement, hashing, or other synchronous handshake work. More internal timing points would be required to assign the complete interval to one operation. The API-level inability to yield during verifier work is independent of that attribution.
Expected behavior
An application using the async API should be able to perform a standards-compliant, fully verified TLS handshake without preventing unrelated tasks on the same executor from running for several seconds.
Certificate verification must continue to fail closed. Disabling certificate or hostname verification is not an acceptable workaround.
Possible solutions
Would it be worth considering one or more of the following?
An async verifier interface. Add an async counterpart to
TlsVerifier, soverify_certificateandverify_signaturecan await a hardware peripheral, another executor task, or work offloaded to another core. This could use async trait methods/RPITIT, associated future types, or a separateAsyncTlsVerifiertrait to avoid breaking the blocking API.A poll-based verifier state machine. Let verification return a pending state and resume through
poll_verify. This avoids requiring allocation or boxed futures and may fit the crate'sno_stdgoals.Break native chain verification into cooperative steps. Yield between certificate signatures and before/after
CertificateVerify. This would not make an individual monolithic RustCrypto operation interruptible, but it could reduce the longest executor gap for multi-certificate chains.Extend crypto-provider injection to public-key verification. Issue #123 already tracks hardware-accelerated crypto. Allowing custom P-256/P-384/RSA certificate verifiers would reduce CPU time and, combined with an async verifier contract, permit genuinely non-blocking peripheral or multicore implementations.
Document the scheduling limitation. If cooperative verification is not currently practical, document that the async API performs CPU-bound cryptography synchronously and that socket/handshake timeouts cannot pre-empt it. This lets single-executor applications plan explicit isolation.
Merely making the verifier method
asyncwill not by itself make a direct call to RustCrypto cooperative; the implementation would still need incremental work, hardware async support, or offloading. The value of an async contract is that it would make those implementations possible without blocking the handshake API.Related upstream work
#123 addresses throughput and replaceable crypto, whereas this report is also about scheduling/cooperativeness: even accelerated synchronous verification can still monopolize an executor, and an application cannot currently supply a verifier that awaits offloaded work.
I can provide the complete diagnostic log, phase-timing patch, certificate survey, or help test an async/poll-based verifier design on ESP32-S3 if useful?