Skip to content

Decouple onboard estimator updates onto a background thread#118

Open
Yuming-Lee24 wants to merge 2 commits into
learnsyslab:mainfrom
Yuming-Lee24:decouple-estimator-thread
Open

Decouple onboard estimator updates onto a background thread#118
Yuming-Lee24 wants to merge 2 commits into
learnsyslab:mainfrom
Yuming-Lee24:decouple-estimator-thread

Conversation

@Yuming-Lee24

@Yuming-Lee24 Yuming-Lee24 commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Decouple the onboard estimator (external mocap pose) updates from the control loop by
streaming them from a dedicated background thread at a fixed frequency (default 30 Hz).

Previously the external pose was sent inside the environment step() and rate-limited there
(POS_UPDATE_FREQ). Because the send was coupled to the control loop, any stall or jitter in
the loop (heavy compute_control, GC pause, blocking I/O) directly delayed the estimator
refresh. When the loop stalled, the onboard Kalman filter stopped receiving fresh poses and had
to dead-reckon on IMU alone, causing drift and a visible correction once poses resumed.

What changed

  • Crazyflie now owns a persistent event loop running on a background thread and an
    _update_estimators task that sends the external pose every 1 / update_freq seconds,
    independent of the control loop.
  • connect() starts the updater at the end of the connection sequence, so the estimator
    already receives poses before reset() runs and the drone is armed. close() sends an
    emergency stop first, then signals the updater to stop and joins the thread cleanly (with a
    bounded timeout).
  • Once the updater thread is running, _run submits operations with
    run_coroutine_threadsafe instead of run_until_complete, so control commands and pose
    updates share the same loop safely.
  • Removed the now-redundant manual send_external_pose() calls in RealRaceCoreEnv.step()
    and in return_to_start(); the background thread keeps streaming poses in both cases.
  • A transient send failure is logged and swallowed so a single hiccup does not kill the updater.
  • update_freq is exposed as a constructor / from_radio argument (default 30 Hz).

How it was verified

The payoff of decoupling only shows up when the control loop stalls — under a healthy loop both
the old and new code send at a steady 30 Hz. To make the difference measurable, the send
interval was logged on every external-pose send and an artificial stall was injected into the
controller:

# temporary stall injected into compute_control
if random.random() < 0.1:
    time.sleep(0.3)
# temporary instrumentation added to _send_external_pose
t = self._loop.time()
last = getattr(self, "_last_pose_send_t", None)
gap = 0.0 if last is None else t - last
logger.warning(f"{self.drone_name} external-pose GAP: {gap * 1000:.0f} ms")
self._last_pose_send_t = t

Both the injected stall and the logging are debug-only and are not part of this PR.

Results

Real deployment on cf52, same injected stall (controller repeatedly exceeds the loop period
by ~0.28 s):

Condition External-pose gap during a control-loop stall
Before (coupled to step()) spikes to 322–341 ms — estimator starves for the full stall
After (background thread) stays 33–43 ms — steady ~30 Hz, unaffected by the stall

Sample logs (after, decoupled — note the gap stays ~33 ms despite the loop overrun):

WARNING __main__:Controller execution time exceeded loop frequency by 0.281s.
WARNING crazyflie:cf52 external-pose GAP: 34 ms
WARNING crazyflie:cf52 external-pose GAP: 33 ms

Sample logs (before, coupled — the gap tracks the stall):

WARNING __main__:Controller execution time exceeded loop frequency by 0.284s.
WARNING crazyflie:cf52 external-pose GAP: 322 ms
WARNING __main__:Controller execution time exceeded loop frequency by 0.282s.
WARNING crazyflie:cf52 external-pose GAP: 341 ms

Re-verification after moving the updater start to the end of connect()

The first version started the updater in reset(arm=True). After moving the start to the end
of connect() and reordering close() to emergency-stop first, the full deployment was
re-run on cf52 with the same injected stall and instrumentation:

  • External-pose gaps stay 32–34 ms while the control loop overruns by ~0.28 s — the
    decoupling is unaffected by the new start location.
  • Poses already stream at ~30 Hz right after connect() returns, i.e. before reset() and
    arming.
  • The background stream keeps running through return_to_start() and the shutdown sequence;
    close() emergency-stops first and then joins the updater cleanly (no send or shutdown
    errors logged, including on Ctrl-C).
WARNING:__main__:Controller execution time exceeded loop frequency by 0.282s.
WARNING:lsy_drone_racing.utils.crazyflie:cf52 external-pose GAP: 34 ms
WARNING:lsy_drone_racing.utils.crazyflie:cf52 external-pose GAP: 34 ms
WARNING:lsy_drone_racing.utils.crazyflie:cf52 external-pose GAP: 32 ms
WARNING:lsy_drone_racing.utils.crazyflie:cf52 external-pose GAP: 33 ms
...
WARNING:__main__:Controller execution time exceeded loop frequency by 0.281s.
WARNING:lsy_drone_racing.utils.crazyflie:cf52 external-pose GAP: 34 ms
WARNING:lsy_drone_racing.utils.crazyflie:cf52 external-pose GAP: 32 ms

Testing

  • ruff check . — passed
  • ruff format --check --diff . — passed
  • pixi run -e tests tests -v — 83 passed, 20 skipped (GPU-only)
  • Bench + real-flight deployment on cf52 as above, re-verified after moving the updater
    start to the end of connect()

@Yuming-Lee24
Yuming-Lee24 requested a review from ratheron July 8, 2026 13:46
@Yuming-Lee24 Yuming-Lee24 added bug Something isn't working Deployment labels Jul 8, 2026
Comment thread lsy_drone_racing/envs/real_race_env.py Outdated
Comment thread lsy_drone_racing/utils/crazyflie.py Outdated
Comment thread lsy_drone_racing/utils/crazyflie.py Outdated
Comment thread lsy_drone_racing/utils/crazyflie.py Outdated
Comment thread lsy_drone_racing/utils/crazyflie.py Outdated
Comment thread lsy_drone_racing/utils/crazyflie.py
@ratheron

Copy link
Copy Markdown
Collaborator

The new version is also tested in real? Then we can merge it

@Yuming-Lee24

Copy link
Copy Markdown
Collaborator Author

The new version is also tested in real? Then we can merge it

Not yet, I will do it as soon as possible.

@Yuming-Lee24

Copy link
Copy Markdown
Collaborator Author

The new version is also tested in real? Then we can merge it

I have tested in real, all good, ready to merge

@amacati
amacati requested a review from ratheron July 18, 2026 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working Deployment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants