Setup to use Pixi

[castelao]

What changed?

Added minimalist configuration to use pixi as an alternative to conda.

How does it make Bristlemouth better?

While pixi can install packages from conda-forge, it also tracks the full environment for the four configured platforms. Check pixi.lock (not supposed to edit it manually).

For OSX, inside the root directory of this repository, one would install and use it like:

curl -fsSL https://pixi.sh/install.sh | sh
pixi shell -e dev

Where should reviewers focus?

• Assumed two environments, the basic (equivalent to environment_ci.yml) and dev (the extended environment.yml). Did I miss any dependencies? I tried to keep the same version where it was pinned.
• Followed the same group of PyPI dependencies.
• It doesn't limit the current use with conda, but add support to also use pixi.

Checklist

• Add or update unit tests for changed code
• Ensure all submodules up to date. If this PR relies on changes in submodules, merge those PRs first, then point this PR at/after the merge commit
• Ensure code is formatted correctly with clang-format. If there are large formatting changes, they should happen in a separate whitespace-only commit on this PR after all approvals.

[matt001k]

Hello @castelao and thank you for contributing!
After doing some initial research on Pixi I like the idea of it,
but have a hard time following somethings in particular:

1. Why is it necessary to include the lock file? Is this not something that is auto-generated when setting up the project?
2. What are the benefits that Pixi has to offer that Conda might not have? I noticed you mentioned it tracks the full environment for the four configured platforms, what does this necessarily mean? Will it be easier for new developers to onboard their tooling?

My only concern is adding a lot of extra noise to the repository and potentially confusing new developers!
But if this is able to:

1. Onboard new developers quicker
2. Automate installing tools for new developers for future initiatives (such as through an arduino like IDE)

then I am totally invested!

Looking forward to your response and thank you again!

[castelao]

Hi @matt001k ,

1. Correct, with the pixi.toml the pixi.lock is automatically created. But by keeping the pixi.lock we can guarantee that everyone is using the same versions, so it is easier to reproduce and debug. For instance, a dependency on mylib>=3.14.15,<3.15.0 could be satisfied by 3.14.15 as well as 3.14.159. By running tests with pixi run --locked .... we define precisely which versions to use (as defined in pixi.lock).

2. The pixi.lock track all the platforms defined in channels, independent of the platform I'm running local. And when I update the dependencies, it update for the other platforms as well. I've been using pixi even on my Rust projects, in particular the ones with a Python interface, so I can guarantee that all developers, and deployments, on the three OS are in sync. Check the pixi.lock and you'll see the versions and builds for each one. The file indeed grows large, but I think it is worth.

Other advantages are to keep multiple environments in a single file, instead of using multiple environments.yml, and be able to define tasks (almost like a Makefile) for compiling, testing, etc. Here is an example that runs python tests but first compile the Rust model, which I use in the Github Actions. BTW, this is a wave ray-tracing model!

I dropped conda almost two years ago in favor of pixi and don't regret. I think it is cleaner and more robust, but it's a choice. If you're curious, I suggest trying it yourself before re-directing all the community.

[matt001k]

Ahh ok I can really see the benefits of the lock file, this ensures if there were ever updates to the toml file, someone does not have to go through the process of running: conda env update -f environment.yml. I find this to be a huge benefit!
I also really like the tasks pixi offers, this is huge in the realm of automation and repeatability!
With the additional features pixi has to offer, I like the idea of adding this along side conda and potentially migrating to it in the future.
I am going to take some time to review this PR do some extensive testing using the setup you have created!

[matt001k]

Could you also add some instructions to the README.md?

Maybe a new section that is labelled Install pixi - alternative or something along the lines of that?

[matt001k]

Another question, I noticed that Pixi is directory based and not system based. There are some users who build their firmware externally to this repo. Is there an easy way to activate the shell/run pixi from an external directory?

[towynlin]

Agreed that a little assistance in the README would be helpful. I had never heard of pixi @castelao — thanks for introducing me to it! I just tried it in a fresh clone, and pixi just did the right thing immediately! This is often not the experience of newcomers to Bristlemouth using conda. I want to try it on a computer that I'm more confident is actually fresh with no dev tools setup and see how it works. Windows users especially often have trouble onboarding.

[castelao]

Agreed that a little assistance in the README would be helpful. I had never heard of pixi @castelao — thanks for introducing me to it! I just tried it in a fresh clone, and pixi just did the right thing immediately! This is often not the experience of newcomers to Bristlemouth using conda. I want to try it on a computer that I'm more confident is actually fresh with no dev tools setup and see how it works. Windows users especially often have trouble onboarding.

@towynlin , I'm glad to hear that! I work with a few developers using windows and we have been having the same experience. It just work. I do appreciate the fact that we can track the full dependency tree for everyone in the team, independent of the platform. And when we do some update, everyone move together, including the tests.

I'm curious. Did you try with a Windows machine?

[castelao]

Another question, I noticed that Pixi is directory based and not system based. There are some users who build their firmware externally to this repo. Is there an easy way to activate the shell/run pixi from an external directory?

Good question. There are two options:

1. Activate the environment by running pixi shell -e dev inside the bm_protocol (i.e. inside the repository). With the environment activated, one can cd <anywhere> and use that environment. The point is that the definitions of that environment go with that repo.
2. Do a global install, which I do not recommend. That is the way conda operates, so we end up re-using the environments, such as one pytorch environment for all projects that are based on pytorch, which will break eventually.

[castelao]

@matt001k @towynlin , I'm sorry for the delay. It was a busy time here.

I think I covered all your requests. Did I miss anything? The documentation is a little strange because the conda instructions are in multiple places and I didn't want to change it too much.

[towynlin]

I love it and really appreciate the contribution @castelao. Thank you!

I tested on a newly spun up Windows VM. As prerequisites I had to install Git and the Microsoft Visual Studio C++ build tools, as well as pixi of course. After that I could do pixi shell -e dev. 🎉

Despite that success, I was not able to build any of the presets, but the failures are more to do with other parts of our code, nothing to do with pixi. We have more work to do to continue smoothing the path to Bristlemouth development on Windows.

Any further thoughts/objections/requests @matt001k ? If not, please merge.

[castelao]

@towynlin , do you think we could bypass the need for Microsoft Visual Studio C++ build tools by installing the equivalent tools through conda-forge? Something like this ?

[towynlin]

If you have the time to reproduce and test, that sounds like potentially a great option!

[matt001k]

No objections here! Awesome work. I am excited to explore pixi more! Thank you.

[towynlin]

@castelao Let us know whether you want us to (A) merge this as-is, and let the potential "compilers" dependency work be a separate PR, or (B) you want to test out adding the "compilers" dependency in this PR.

[castelao]

@towynlin , thanks for checking. It seems like this PR achieved its goal. Let's merge it as it is and I'll try the compilers solution in a following PR. This is another busy week for me, so it will take me a few days to be able to try.