[Serve] [Docs] Async io + Ray Serve best practices

[abrarsheikh]

https://anyscale-ray--58909.com.readthedocs.build/en/58909/serve/advanced-guides/asyncio-best-practices.html

[marwan116]

Thank you so much for putting this guide together, really clarifies how to think of asynchronous execution with Ray Serve.

I left some comments which I am curious about.

[marwan116]

Should we explicitly encourage Ray core usage ? For example remote actors require proper autoscaling and observability tooling to help scale and debug when things go wrong. Can we instead advocate for composition in this case ?

Here is what cursor suggested as an alternative section:

### Using serve composition for true parallelism

With serve composition, you can split CPU-heavy work into separate deployments that run in independent processes:

```python
@serve.deployment
class HeavyCompute:
    def compute(self, x):
        # Heavy compute runs in its own deployment process.
        return x * x


@serve.deployment
class ParallelProcessor:
    def __init__(self, compute_handle):
        self._compute_handle = compute_handle

    async def __call__(self, request):
        values = [1, 2, 3, 4]
        # Call the composed deployment in parallel.
        refs = [self._compute_handle.compute.remote(v) for v in values]
        results = await asyncio.gather(*[ref for ref in refs])
        return {"results": results}


# Compose the deployments.
app = ParallelProcessor.bind(HeavyCompute.bind())

[abrarsheikh]

The cluster should still be able to autoscale even if ray tasks are created from serve application. But good point about observability. Creating another deployment and offloading the work there has the same problem because then that deployment becomes blocking, so we just kick the can forward to a downstream deployment.

I have seen users use this pattern especially in OSS. So I am inclined to keep this but will mark this a advanced usage pattern, call out the observablity pitfall and not mention it repeatedly in the document.

[marwan116]

Sounds good to me about calling it an advanced pattern and leaving it to a dedicated section.

Yes I agree remote tasks should scale just fine. In my mind with composition they can set target and max requests to a low number close to 1 and use serve's queue based replica scaling to parallelize the compute but I recognize this probably adds more performance overhead than directly submitting tasks.

As for my comment on auto scaling - I was referring to remote actors given the document originally stated "use remote tasks or actors" which would require an auto scaling actor pool abstraction...