test: check request cancellation before backend dispatch

[pskiran1]

What does the PR do?

This PR expands request-cancellation integration coverage for non-batched identity and TensorRT models.

Adds queued-cancellation and response-cache assertions to the scheduler tests.
Creates rate-limited TensorRT and resource-holder models for backend-dispatch coverage.
Adds TensorRT cancellation, normal queued execution, and unaffected-request cases.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

triton-inference-server/core#522
triton-inference-server/tensorrt_backend#127

Where should the reviewer start?

Test plan:

• CI Pipeline ID:

Caveats:

Background

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: #xxx

[greptile-apps]

Greptile Summary

This PR adds integration coverage for cancelling queued inference requests before backend dispatch and improves cancellation synchronization and diagnostics.

• Adds shared helpers for callback synchronization, metrics polling, execution counts, pending-request state, and cancellation confirmation.
• Covers queued cancellation and response-cache behavior for non-batched identity models.
• Adds TensorRT rate-limiter coverage proving that a cancelled request is skipped while an adjacent live request executes.
• Logs request-scoped cancellation results after the cancellation call completes.

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding findings or new actionable defects.

The previous findings were manually resolved, and the current implementation uses bounded pending-state polling and logs cancellation only after the core cancellation call completes. The sole change since the previous review is formatting-only and does not alter behavior.

Important Files Changed

Filename	Overview
src/grpc/infer_handler.h	Adds request-scoped logging after core cancellation succeeds or fails and releases cancellation errors correctly.
qa/L0_request_cancellation/cancellation_test_utils.py	Introduces bounded synchronization and metrics helpers shared by cancellation integration tests.
qa/L0_request_cancellation/scheduler_test.py	Adds queued non-batched cancellation and response-cache assertions while replacing fixed cancellation sleeps with callback synchronization.
qa/L0_request_cancellation/trt_cancellation_test.py	Verifies that rate-limited TensorRT cancellation skips only the cancelled request and allows adjacent live work to execute.
qa/L0_request_cancellation/test.sh	Builds the required identity and TensorRT model repositories and runs the new cache and rate-limiter scenarios.

Sequence Diagram

sequenceDiagram
  participant Client
  participant gRPC as gRPC handler
  participant Limiter as Rate limiter
  participant Backend
  Client->>gRPC: Submit inference request
  gRPC->>Limiter: Queue request
  Client->>gRPC: Cancel request
  gRPC->>Limiter: Issue core cancellation
  Limiter-->>gRPC: Cancelled response
  gRPC-->>Client: CANCELLED
  Note over Limiter,Backend: Cancelled request is not dispatched

Loading

_{Reviews (13): Last reviewed commit: "Fix pre-commit" | Re-trigger Greptile}

[whoisj]

lgtm