Copier templates with additional Python dependencies

[sisp]

Copier templates sometimes require additional Python packages, e.g. copier-templates-extensions (+ e.g. additional packages needed to implement custom local Jinja extensions that are loaded via copier_templates_extensions.TemplateExtensionLoader) or external Jinja extensions. In those cases, a developer needs to install Copier and also the required additional packages manually. IMO, the UX isn't good and it's error-prone. Instead, I'd prefer a way to generate a new project or update an existing project without manual installation of additional packages.

Currently, I don't think there's a good way to do it because the template is an external resource that in essence consists only of folders and files some of which happen to contain Jinja template syntax. But I have an idea how to extend Copier and equip a template with information about additional Python packages:

copier.yml could be extended to have a new field for specifying dependencies, e.g.:

_dependencies:
  copier-templates-extensions: ">=0.3.0"
  # ...

Copier would offer a new CLI subcommand, e.g. copier install $src, to install the dependencies in a Copier-managed virtual env for this template and version (e.g. like Poetry or pre-commit manage their own virtual envs). To allow Copier to use those dependencies and to avoid dependency conflicts, the same version of Copier would need to be installed again in this virtual env along with the additional dependencies, and a call of, e.g., copier copy $src $dst would transparently delegate the call to the copier executable installed in the virtual env. Essentially, the original copier executable (e.g. Copier installed via pipx) would only be a kind of orchestrator. Alternatively, the virtual env could be prepended to Copier's PYTHONPATH, but I fear this might be error-prone when different (possibly incompatible) versions of the same package are installed in both the virtual env and in the, e.g., pipx-managed virtual env in which Copier is installed.

It might be conceivable to make the copier install $src step optional. When a Copier template with dependencies shall be used but copier install $src wasn't run before, Copier could automatically create the virtual env and install the dependencies, similar to how pre-commit works.

I'm curious about others' opinions. 🙇

/cc @yajo @pawamoy

[pawamoy]

There's a way with pipx:

pipx run --pip-args="$deps" copier copy $src $dst

Though it's not automated for users: they do have to copy/paste this command from a template's documentation for example.

[pawamoy]

Yeah you're right, edited my comment :)

[pawamoy]

Maybe pipx could be reused. Copier could depend on it. It would first fetch the template to get the dependencies, then run the pipx command in a subprocess. That becomes weird haha. And the template would be cloned twice. We'd really need a cache for templates clones.

[sisp]

Could pipx be used as a library instead of via subprocess call?

[pawamoy]

Just checked their code base, seems like it. At a glance, it actually looks like one of cleanest CLI / Python interface I've seen in the ecosystem. And I've seen a lot, and I'm very critical about this. Amazing work @cs01 and @itsayellow!

[sgarfinkel]

The worst part about this is to run a template update you also need to copy the dependencies. If you go this route it would be really nice to just run standard copier commands once you’ve laid down your initial template.

[sgarfinkel]

What about just letting folks ship a bootstrap python script that copier runs before executing anything? That way users can do whatever they want outside of just installing dependencies. If that API was well defined, you could even inject a helper object that bootstrap scripts could use to install these dependencies in a "copier-aware" way.

[sisp]

Add a --bootstrap flag that opts-in to running a root bootstrap.py script or something like that. This would also be handy for me where I'd ideally like to preseed some data at startup.

@sgarfinkel I think finer control over tasks (#240) might address your use case.

[pawamoy]

Ha, yeah, that's true 🤔

[sisp]

For my taste, a bootstrap script would be too broad. I tend to prefer features with a clear purpose and scope because it's easier for users to benefit from them in the right way.

[yajo]

Regarding security risk: Copier tasks (and migrations) are also a massive security risk.

I'd really love to have a way to make Copier more secure in this regard. Something like a safe mode, with a way to mark a template as trusted, and a way to warn about the specific risks ( tasks, migrations and extensions ATM).

[sisp]

Perhaps some kind of filesystem isolation/jail while generating/updating a project, along the lines of os.chroot but cross-platform (if possible) and without needing root. We should discuss this in a dedicated issue.

[yajo]

Copier is kinda similar to Ansible. We even include the extension with similar filters. Both are configured with yaml, written on Python, both rely on jinja a lot, both can be extensible. Ansible inventend Ansible Galaxy, which is a way to install ansible content. Some docs here.

If you ever used Ansible's requirements.yml, you'll have noticed it's full of problems. I mean: it would have been extremely better to package Ansible content as pip-installable packages. There you'd have complete venv isolation, dependency management, etc. for free. They have some business model around Ansible Galaxy, so ok for them... but honestly, if you need packages, just use packaging tools.

We have enough of those tools: Nix, poetry, pip, pipx... Let's not add this burden to Copier please... 😅

My solution is Nix, just like always when we talk about packaging matters. 🤷🏼‍♂️

Check out this example. There, I provide a version of Copier based on upstream but including an extension. It's all done in a short flake.nix file (it's longer because I put a lot of comments 😆). It's very easy to create and maintain. Cool thing is, it will work when 10 years have passed and Copier has turned on a SaaS AI-based robot, Python 5 has broken Linux compatibility, and Jinja has switched markers for emojis 😋. Just joking, but you get the point.

Copier could automatically create the virtual env and install the dependencies, similar to how pre-commit works.

I'm actually bitten no less than once a week by this design of pre-commit. When it happens publicly on some specific projects, I keep a list updated here just for fun: OCA/oca-addons-repo-template#175. But it happens privately and on other kinds of projects too!

As you can read there, I've invented an alternative called precommix which solves the problem with Nix. Others have invented pre-commit-hooks.nix which is better on some things, worse in some others (and we use it here in Copier BTW).

So, if I'm fleeing away of this broken system, please don't ask me to invent a similar one in Copier. 😅

[sisp]

TBH, I'm not yet convinced by the Nix approach (I know you're a huge fan). But I appreciate your effort for compiling the example! This is very helpful for the discussion.

Specifically in your example, I think the UX is suboptimal:

1. The Nix command requires the URL twice. This is mainly a bit inconvenient, but I might be able to live with it.

2. Specifying a Git tag isn't straightforward anymore because I'd need to pass the -r, --vcs-ref flag to Copier and use the matching Git URL, otherwise I might be installing a possibly mismatching version of Copier plus dependencies for the template. For instance:

   nix run git+https://git.example.com/my-copier-template.git?ref=refs/tags/v1.2.3 -- -r v1.2.3 git+https://git.example.com/my-copier-template.git ./result

   (I haven't checked whether that ?ref=... syntax is correct, just googled quickly.)

General concerns of mine regarding Nix:

1. It doesn't work natively on Windows, only via WSL. I personally can live with it, but there are many Windows users out there. Just saying ... 😕
2. I noticed that Nix is based on the idea of a distribution of packages. This means, there is only a single (typically the latest) version available for each package. There are some "workarounds", e.g. there is python37, python38, etc., but allowing to choose an arbitrary package version doesn't seem to align with the Nix philosophy. See also No way to install/use a specific package version? NixOS/nixpkgs#9682. Disclaimer: I'm new to Nix, so please correct me if I'm wrong. The problem I see is with the fact that I can't choose the version of a package individually but only the version of the package index. There might be good reasons why want to choose a particular package version and not the latest, e.g. a bug in the package and I need to wait for a fix before I can use it.

Regarding Copier now:

I think it's mostly Copier's responsibility to ensure a smooth UX including taking care of installing additional dependencies needed to run a template. I don't think it must mean that Copier actually installs the packages itself, but requiring a tool like Nix, which is way outside the Python ecosystem, isn't good UX in my opinion.

If you ever used Ansible's requirements.yml, you'll have noticed it's full of problems. I mean: it would have been extremely better to package Ansible content as pip-installable packages.

Might this be way forward? I've actually been thinking about a workflow in which a Copier template is a Python package published in a PyPI like pypi.org. Then, any standard package manager could take care of the extra dependencies and all packages. The missing piece for me is the implicit usage of the template that's bundled in the Python package. Specifically, I imagine - like in your Nix example - that Copier and additional dependencies would be dependencies of the package. A user would install the package and run

copier [MAIN_SWITCHES] copy [SUB_SWITCHES] destination_path

without the positional argument template_src because this would be given implicitly by the Python package which contains the template. Also, I imagine the _src field in the answers file would need to state that the template source is not the Git URL but rather the Python package, e.g.:

_src: my-copier-template
# or simply
_src: null
...

Regarding pre-commit issues:

Isn't the problem that pre-commit typically installs packages itself from some URLs without a locked dependency tree? I never understood why one should use pre-commit like this. Instead, I'd always install the packages via Poetry (for Python), NPM/Yarn/pNPM/... for Node.js, etc. (where I have locking) and configure the hooks as "local" hooks with language: system. This way, I can also benefit from Dependabot / Renovate Bot to update also my hook packages. Is there any problem left in your experience with pre-commit when following this approach?

[yajo]

1. The Nix command requires the URL twice. This is mainly a bit inconvenient, but I might be able to live with it.

2. Specifying a Git tag isn't straightforward anymore because I'd need to pass the -r, --vcs-ref flag to Copier and use the matching Git URL, otherwise I might be installing a possibly mismatching version of Copier plus dependencies for the template.

It's just a way to run nix packages without installing on path. Kinda like pipx run.

But you can do nix profile install git+https://gist.github.com/3f935292a326ff22192b4b497a8b6d86.git to get a copier binary into your $PATH, or nix shell git+https://gist.github.com/3f935292a326ff22192b4b497a8b6d86.git to get it only temporarily.

Also I'm needing the full URL because I'm using a gist. There's a shortcut for github.com. I install Copier currently using just nix profile install github:copier-org/copier. There are other shortcuts too for other providers.

In any case, we can compare that to docker, which is more known for reproducible installs. Imagine you did docker run example.com/my-copier-template:v1.2.3 copier -r v1.2.3 copy https://example.com/my-copier-template.git. Yes, you repeat twice the URL, but one is for getting the package, and other is for running it.

Hopefully, fixing #1081 should let us include the repetitive part into the app entry point, making this problem disappear.

I noticed that Nix is based on the idea of a distribution of packages. This means, there is only a single (typically the latest) version available for each package. There are some "workarounds", e.g. there is python37, python38, etc., but allowing to choose an arbitrary package version doesn't seem to align with the Nix philosophy. See also No way to install/use a specific package version? NixOS/nixpkgs#9682. Disclaimer: I'm new to Nix, so please correct me if I'm wrong. The problem I see is with the fact that I can't choose the version of a package individually but only the version of the package index. There might be good reasons why want to choose a particular package version and not the latest, e.g. a bug in the package and I need to wait for a fix before I can use it.

That's the case when the package is shipped separately, like in Nixpkgs, which is the distro repo. However that's actually cool when compared to other distros!

Let's put an example and compare with apt. Let's say your app needs Python 3.1 because of reasons. How would you install Python 3.1 on Ubuntu 22.04? You'd probably have to enable an apt repo of an old Ubuntu version where it was available, hope that binary packages are still available, and then install python3 from there. But what would possibly happen?

1. Your modern Python 3 package would be replaced.
2. dependencies of that old package would be also old and replace your modern ones.
3. if you ever get to installing it, you'd probably break your OS in many unpredictable ways.

On Nix, you can have a different shell on each project, bringing the officially suggested versions of everything. You can overlay specific packages from specific commits of repos. You leverage the binary cache if available for speed, but if it's lost, you just compile it automatically, and it will work just like it did 10 years ago. Yes, just like with Ubuntu, you need to know first on which version of the distro (commit, for nixpkgs) the python3 package had version 3.1, but that's just because you're using a repo, and that's how repos work.

However, when the package is shipped with the app (like copier), the app tag matches the package tag. Thus this problem (if it really is) disappears.

Actually, last week it happened to me. I have copier installed from master, where currently we have breaking changes. I tried to update a template that got broken by those. What did I do?

nix shell github:copier-org/copier/v7.1.0
copier update 
exit

Beautiful.

Might this be way forward? I've actually been thinking about a workflow in which a Copier template is a Python package published in a PyPI like pypi.org. Then, any standard package manager could take care of the extra dependencies and all packages.

You can't predict the package manager used to install Copier. It could be nix, pip, pipx, Poetry, pdm, conda, apt, pdm, dnf, chocolatey... you name it. It could be a system-wide install, a vitrualenv or a pdm-like flat list of packages. You could be root or not. And there's Windows!

The jinja2 extensions must be installed in the same environment that Copier. Keep that in mind.

Besides, the fact we just use git as template repos is a feature! For instance, yeoman uses npm packages for the same reasons you're saying and, you know what? It's terrible if you're not into the npm world.

What if your template is for a npm package? How do you install npm as a Python dependency? And if it is java? Or haskell?

Sorry... but this is a route I wouldn't want to go through. 😅

To me, the problem is solved with nix. For others, it might be solved with docker, or just with some instructions on the README telling the user what to install. But let's not become the next pre-commit please. 😁