bug: bentoml.bentos.build changes the cwd when passing build_ctx

[fwindolf]

Describe the bug

After running bento.bentoml.build and passing a build_ctx argument the cwd is changed to the build_ctx directory.

I would have expected to return to cwd set before that after finishing the build.

To reproduce

build_ctx = tempfile.TemporaryDirectory(prefix=f"bento_{MODEL_NAME}__")

cwd = os.getcwd()

bento = bentoml.bentos.build(..., build_ctx=build_ctx)

assert cwd == os.getcwd(), "Bentoml changed cwd"

Expected behavior

No response

Environment

bentoml: 1.0.12
python: 3.9.10
platform: ubuntu (WSL2)

[aarnphm]

I believe this is not a bug, as we are aware of this, from this PR #2737.

@bojiang iirc the reason for changing the cwd to make the imports order from service.py works correctly.

We will have to try importing the service when creating the bento, therefore it is considered as a standalone_load.

Maybe we want to fix this behaviour. Probably want to do something under

BentoML/src/bentoml/bentos.py

Line 268 in 931dedc

def build(

try:
	build_config = ...
finally:
	restore_cwd

But this would be a breaking change?

cc @sauyon for thoughts on this as well.

[fwindolf]

Maybe some more context: I use wandb and build the bento directly after training it. This leads to wandb failing on finish with an error, due to the cwd being set to the build_ctx directory, which is deleted after building.

As a user, this seems odd as there is no indication in build that the cwd is changed.

[sauyon]

Er, this looks like a bug to me; the point of the standalone_load change was to prevent this behavior.

[sauyon]

We've discussed this internally and have decided this is indeed a bug that we'd like to fix, but since we now want only one Bento to be loaded per process to work around some limitations of the way we're (ab)using Python to load services, we want to wrap every build in its own process; contribution here is welcome!

There would be two changes and it should be pretty straightforward:

1. Switch the BentoML CLI (

   BentoML/src/bentoml_cli/bentos.py

   Line 268 in 213b584

   def build(build_ctx: str, bentofile: str, version: str) -> None: # type: ignore (not accessed)

   ) from using the bentoml.build API to using the internal Bento.create API.
2. Switch bentoml.build to use subprocess to call out to the bentoml CLI; see bentoml.serve (

   BentoML/src/bentoml/bentos.py

   Line 428 in 213b584

   def serve(

   ) for an example of this.