Some connected issues - retry mechanism?

[tommasocarraro-sony]

Hi Frankie!

I removed some redundant issues for the entities_api repo. As promised there, I am opening this issue here, as I think it is the right place.

First of all, my current setup:

1. Front-end app: ChainLit
2. LLM: DeepSeekV3 through Hyperbolic provider
3. Middleware layer: Entities API
4. External tools: MySQL database (for strict filtering), vector store (for exploratory queries), recommender system (to rank retrieved items based on the current user preferences)

While using this setting, I am encountering some problems:

1. Sometimes, the LLM fails to generate a valid JSON for the tool call. In such cases, the backend logs "JSON parsing failed even after fixes". The problem is that the conversation gets stopped, without showing any error message to the final user (me or the user using the front-end app). For this problem, I suggest streaming the error from the assistant, instead of only logging it. An example of the streamed message could be: "Sorry, I am not able to perform any tool call as the generated JSON is invalid, I will try now to adjust it..."

I know this is not something for which projectdavid or entities_api is designed. What I am suggesting here is to raise the exceptions (instead of "return False" or "break") so they can be managed in the front-end, if someone would like to. The example I provided up here could be a way of managing the exception in the front-end.

2. Sometimes, after the assistant properly generates the JSON file, the run goes into queued status and stays stuck there. In these cases, the timeout is then reached, and an exception is raised. Even in this case, the assistant does not stream the issue to the user, so it is unclear what is happening without looking at the logs.

3. Sometimes, the JSON answer returned by the Hyperbolic provider contains some errors. Because of this, the stream is interrupted. Even in this case, the user does not know the reason without looking at the logs.

4. There are some cases in which the Docker container containing FastAPI restarts. In these cases, it is possible to get two types of exceptions:

   • peer closed connection without sending a complete message body
   • server disconnected without sending a response

My objective: I would like to make the conversation flow as smooth as possible and prevent the user from writing adjustments prompts when problems occur.

I think all these problems can be solved in the front-end by catching all these exceptions from the backend. If any of these exceptions occur, it is enough to sleep (time.sleep()) for some seconds and then restart the conversation by sending the assistant the message the user sent when everything got interrupted by these problems.

What do you think? Are you also facing these kinds of issues?

[tommasocarraro-sony]

There are some cases in which the Docker container containing FastAPI restarts. In these cases, it is possible to get two types of exceptions:

• peer closed connection without sending a complete message body
• server disconnected without sending a response

For this part, I did not solve but at least I found the issue after many, many trials. I discovered Uvicorn was exiting with exit code 137. I discovered that it was a memory issue. The more prompts you launch, the more the RAM gets filled. If the memory limit of Docker is reached, Docker restarts the container. I do not get where all this RAM is used and why it is not cleaned after each prompt. For the moment, I have fixed it by increasing it. However, this will still be a limit, as I can still reach the limit, depending on the length of the conversation.