[Feature]: Concurrent request dispatch for P2pNcclConnector

[simondanielsson]

🚀 The feature, motivation and pitch

The P2pNcclConnector supports sending the request to the P and D instances simultaneously. However, the vllm-router currently awaits the response from P before sending the request to D (like the toy proxy), which is a waste of TTFT as the P2pNcclConnectort already supports async transfer. We should allow also running the router in concurrent dispatch mode to save a full HTTP roundtrip and scheduling overhead on the D side.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[simondanielsson]

#167