Skip to content

[Fix] Avoid double-free in overlap_loop if Req is aborted while finishing - #111

Open
MisakaVan wants to merge 1 commit into
sgl-project:mainfrom
MisakaVan:fix-overlap-loop-race-condition
Open

[Fix] Avoid double-free in overlap_loop if Req is aborted while finishing#111
MisakaVan wants to merge 1 commit into
sgl-project:mainfrom
MisakaVan:fix-overlap-loop-race-condition

Conversation

@MisakaVan

@MisakaVan MisakaVan commented Mar 25, 2026

Copy link
Copy Markdown
Contributor

Fix double-free (and what appears to be a UAF) race condition in Scheduler.overlap_loop.

Problem

    def overlap_loop(self, last_data: ForwardData | None) -> ForwardData | None:
        """
        The main loop of overlapping scheduling and execution.

        It will overlap the execution of current batch and processing of last batch's results,
        which can effectively hide CPU latency and improve GPU utilization.
        """
        blocking = not (
            last_data is not None  # don't block if we have a batch to be processed
            or self.prefill_manager.runnable
            or self.decode_manager.runnable
        )
        for msg in self.receive_msg(blocking=blocking):
            self._process_one_msg(msg)

        forward_input = self._schedule_next_batch()
        ongoing_data = None
        if forward_input is not None:
            with self.engine_stream_ctx:  # run the batch in the engine's stream
                self.engine.stream.wait_stream(self.stream)
                ongoing_data = (forward_input, self._forward(forward_input))

        self._process_last_data(last_data)
        return ongoing_data

The _process_one_msg may abort a Req that is in-flight (in last_data), thus freeing its corresponding resource (the table_idx). Then such race conditions may happen:

  • If the Req is finishing (e.g. just decoded an EOS token), it will be double-freed in self._process_last_data(last_data).
  • If forward_input = self._schedule_next_batch() assigns a new prefill Req/ChunkedReq, it may be assigned the same table_idx just freed, while the original Req is still in-flight (to be handled by _process_last_data). While this is actually not a UAF condition since _process_last_data only calls cpu-side Req.append_host, it still creates unstable semantic where two living Req objects are sharing the same table_idx and makes the code vulnerable.

Fix

The fix defers aborted-request's resource freeing until after _process_last_data (which populates finished_reqs and finishes the previous in-flight batch). _process_one_msg just removes the Req from the managers so the request won't be scheduled for the next batch.
To-be-aborted reqs are queued in pending_aborts. The freeing procedure skips any reqs just finished or in-flight.

Test

This commit also contains a testcase of the double-free scenario. The testcase fails without the fix.

…aneously

In overlap_loop, when a request is aborted via AbortBackendMsg and also
finishes (EOS) in the same iteration, _free_req_resources was called twice:
once in the abort path and once in the finish path. The abort path did not
add the req to finished_reqs, so the finish path did not skip it.

The fix defers abort resource freeing: instead of calling _free_req_resources
immediately in the abort path, we queue the req in pending_aborts and free
it in _free_pending_aborts which runs after _process_last_data populates
finished_reqs, so already-finished reqs are properly skipped.
@MisakaVan

Copy link
Copy Markdown
Contributor Author

cc @DarkSharpness

@DarkSharpness DarkSharpness added the bugfix Fixes incorrect behavior, runtime errors, or regressions. label May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Fixes incorrect behavior, runtime errors, or regressions.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants