Add REST plugin for user-defined policies

[Nadine-H]

Resolves #2630.

Key notes:

• The REST plugin is a builtin plugin, unlike how other plugins are defined and loaded in the dstack system.
• The REST plugin has to be enabled through the dstack server config if the user wishes to hook up a plugin server.

[Nadine-H]

Realized that setting up this plugin as an external package is unnecessarily complicating the installation process. I'm going to change it so that it's more builtin the server code.

[r4victor]

Realized that setting up this plugin as an external package is unnecessarily complicating the installation process. I'm going to change it so that it's more builtin the server code.

A good place for a built-in server plugins would be to put them in src/dstack/_internal/server/services/plugins/builtin/rest_plugin

May require renaming src/dstack/_internal/server/services/plugins.py to src/dstack/_internal/server/services/plugins/__init__.py

Let me know if you need help with refactoring.

[r4victor]

I'll create an openapi spec doc

@Nadine-H, Do you think of writing the spec manually or generating it automatically? Automatic generation is highly preferable because spec models are regularly updated with new properties and it won't be practical to maintain manually written spec.

[Nadine-H]

I'll create an openapi spec doc

@Nadine-H, Do you think of writing the spec manually or generating it automatically? Automatic generation is highly preferable because spec models are regularly updated with new properties and it won't be practical to maintain manually written spec.

Automatic generation for sure. I set up a local FastAPI server to test the rest plugin and will share the openai.json with you shortly.

[dakota-shop]

How does the Plugin reject a Run from being applied? Do we want some exceptions (HTTP Status Codes) to be propagated to Dstack? Or should there be an errors element of the response body?

I'm leaning toward the latter since if we're monitoring the plugin server, it might be difficult to distinguish between "the plugin server thought this request was invalid" and "the plugin server could not answer the question".

[Nadine-H]

Sorry, I misread your comment when you shared it internally. Yes - I agree on the latter option, that way the user knows if the action is on their end or if it's a system issue (a bug in dstack or plugin server)

[Nadine-H]

Added a pydantic model for the plugin server response that includes an error field here. Expectation is if request is rejected, dstack should a 200 response from plugin server with an error.

[Nadine-H]

Adding documentation to the guides as well

[colinjc]

@r4victor Should async policies be a thing, or are policies always running in an executor?