Skip to content

Update celery forwarder to use greenlets instead of processes #689

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

dmchoiboi
Copy link
Collaborator

Pull Request Summary

What is this PR changing? Why is this change being made? Any caveats you'd like to highlight? Link any relevant documents, links, or screenshots here if applicable.

Celery forwarder is io-bound; move to greenlets to enable higher concurrency

Test Plan and Usage Guide

How did you validate that your PR works correctly? How do you run or demo the code? Provide enough detail so a reviewer can reasonably reproduce the testing procedure. Paste example command line invocations if applicable.

@dmchoiboi dmchoiboi requested review from njd-scale and tiffzhao5 March 1, 2025 08:11
@dmchoiboi dmchoiboi force-pushed the dmchoi/fix-celery-forwarder branch from 0728de3 to 659f088 Compare March 1, 2025 08:22
@dmchoiboi dmchoiboi force-pushed the dmchoi/fix-celery-forwarder branch from 659f088 to 1f06017 Compare March 1, 2025 08:36
Copy link
Contributor

@tiffzhao5 tiffzhao5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious how much higher the concurrency is through greenlets and the tests you ran to confirm that?

Copy link

socket-security bot commented Mar 4, 2025

Updated and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package New capabilities Transitives Size Publisher
pypi/[email protected]3.1.1 Transitive: eval, network +763 556 MB JaredKuolt, ambroff, bob, ...6 more

View full report↗︎

Have feedback? Participate in our User Experience Survey 📊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants