An idiomatic approach to supporting derived variables

[sisp]

Background

Creating internal/private/computed/derived variables (typically) from answers of the questionnaire is a recurring topic and request (#229, #545, #629, #678 (comment), #700, #716, #1195 (comment)) which is partially solved by Jinja import/include (see test cases for some inspiration), but this approach is slightly unintuitive and can only derive string variables. Alternatively, the context hook extension can be used, but it requires installing the copier-templates-extensions package which makes a Copier template unsafe (for good reason because arbitrary code can be executed) even when no other Jinja extensions, migrations or tasks are used.

I think I've come up with a more idiomatic solution to this problem.

Solution idea

We could add a new setting to the copier.yml file, e.g. _jinja_extra_context, where derived variables are defined:

name:
  type: str

_jinja_extra_context:
  slug: "{{ name|lower|replace(' ', '-') }}"

Then, we could extend the current render context by a new field, e.g. _copier_extra, whose value is a Mapping with a custom implementation of the __getitem__ method which renders the template of the requested derived variable. Here is a minimal and isolated example of the setup:

from collections.abc import Mapping
from typing import Any, Callable, Iterator

from jinja2 import Environment


class ComputedContext(Mapping[str, str]):
    def __init__(
        self,
        computed: Mapping[str, str],
        context_factory: Callable[[], Mapping[str, Any]],
        env: Environment,
    ) -> None:
        self._computed = computed
        self._context_factory = context_factory
        self._env = env

    def __getitem__(self, key: str) -> str:
        template = self._env.from_string(self._computed[key])
        return template.render(self._context_factory())

    def __iter__(self) -> Iterator[str]:
        yield from iter(self._computed)

    def __len__(self) -> int:
        return len(self._computed)


def create_context(
    env: Environment, context: Mapping[str, Any], computed: Mapping[str, str]
) -> Mapping[str, Any]:
    new_context: Mapping[str, Any] = {
        **context,
        "_copier_extra": ComputedContext(computed, lambda: new_context, env),
    }
    return new_context


env = Environment()
context = create_context(
    env,
    {"name": "The Name"},
    {"slug": "{{ name|lower|replace(' ', '-') }}"},
)
template = env.from_string("{{ name }} -> {{ _copier_extra.slug }}")
result = template.render(context)
assert result == "The Name -> the-name"

Native types

In addition, it would be possible to extend this approach to support derived non-string variables through Jinja native types (although this has some limitations), e.g. by allowing a custom YAML tag like !!jinja-native to inform Copier that those derived variables shall be rendered using a native environment:

number:
  type: float

_jinja_extra_context:
  squared: !!jinja-native "{{ number ** 2 }}"

Additional context

I originally tried to avoid a new context key like _copier_extra and "magically" add (lazily evaluated) derived variables to the root context, and for this I experimented with lazy-object-proxy, but it doesn't work because Jinja does a shallow copy of the provided context in the render() method call which triggers the evaluation of the proxied object. This leads to eager evaluation of all derived variables which isn't possible when using derived variables in the middle of the questionnaire where some (actually unused) derived variables depend on answers to questions which haven't been answered yet.

[lhupfeldt]

I just made #1205 without seeing this.
I do not know know enough about the copier internals to really comment here, there is a lot of implemetation details.
when: false actually worked perfectly for me. I was also considering before mkaing #1205 whether hidden variables should be in their own section, but I think having them interpersed with prompted variables is more logical because both should be able to be computed / default based on the other. Keeping them together allows declaring them in order of use, making the config easier to maintain without scrolling.
But no big deal if it causes technical issues.

[lhupfeldt]

If I understand your solution, it will run after all questions have been answered.
But my suggestion will allow defaults for questions to be based on temporary hidden variables.
I have had no need for complex types, but I can see how it could be beneficial.
I think the two suggestions solves different problems.
Right now I'm really looking for something similar to the abandoned when: false feature, but obviously well defined and always calculated, not stored in the answers file (though I did not notice that error before).

[carlsylvia]

We actually abandoned the use of when=false to provide calculated values. This approach did not work for us because the values were stored in the answers file rather than being re-calculated each time the template was run. We moved to the use of the Copier Template Extensions, but unfortunately that approach now requires the use of the new --UNSAFE flag, which I am glad to see it being updated to --trust. My question is whether it makes sense to have the dynamically calculated functionality within the template (using either when=false or _var), or using the Copier Template Extensions which achieves the same goal.

[carlsylvia]

I like the idea of extending the use of the special _var semantics. It seems like it would be a very clean and intuitive solution. When we first started using Copier I incorrectly assumed that was how we would create "private/internal" variables, then moved to the when=false approach, and finally the Copier Template Extensions. Whatever the final solution is, I would want to ensure that the value is re-calculated each time the template is applied, as that was the issue we ran into with the when=false approach.

[sisp]

If I understand your solution, it will run after all questions have been answered.

@lhupfeldt No, you would define all derived variables separately, but they would be evaluated lazily upon usage. This way, you only need to make sure to use a derived variable only after all the variables it depends on are available. This also transitive derived variables, i.e. a derived variable can depend on another derived variable. Compared to a question with when: false even if the value were properly re-calculated, my suggested approach is not limited to deriving variables of type str/float/int/bool/json/yaml but (when using Jinja native types) any Python object.

[lhupfeldt]

If your solution allows dependencies between question defaults and variables in both directions, then it handles all scenarios that when: false did, fixes the missing recalculation bug and allows advanced, types. I don't see how it will be simpler to implement than an ordered definition though. But I'm just looking forward to the feature :)

[yajo]

Hi everybody! Thanks for this thread, I think this is really needed.

This feature is asked so often that I'm losing more time and energy explaining why this isn't supported than the time I'd lose supporting it.

To clarify the problem, using when: false to declare variables was never supported. I know people used it, but that was considered an abuse of the feature since it landed. I explained the reason many times, but https://github.com/orgs/copier-org/discussions/1203#discussioncomment-6222124 shows a real world case where you had to go other route because you hit the problem that was preventing us from supporting this as a feature in the 1st place.

Now, regarding the solution you're proposing here @sisp, I can tell you that this is something that came to my mind in the past. However, this solution would be rather complex to implement, dealing with what is vs what is not lazy. At the end of the day, believe me: the next day you pushed this feature, some issue would appear because somebody hits https://github.com/orgs/copier-org/discussions/1203#discussioncomment-6221750.

At the end of the day, the main reason why we never supported when: false was the lack of predictability. As explained in the docs:

copier/docs/configuring.md

Lines 144 to 147 in c16b0b8

	If a question is skipped, its answer will be:
	- The default value, if you're generating the project for the 1st time.
	- The last answer recorded, if you're updating the project.

So, now that we don't record skipped answers, I think that the ultimate solution is what I said in #1206 (comment). This solution:

• Makes support of when: false official.
• Makes it predictable.
• Is a question in the middle of the questionary, so it solves the problem of when it is resolved. Just put the question in the proper place in the questionary, and it will work predictably.
• Avoids having to reinvent many things.

However, it doesn't solve the case explained here by @sisp: the fact of being able to pass in a python function. However, think about it. If we added support for that, we'd be introducing yet another unsafe path into a template. At the end of the day, if you need a version validator in context, that's better handled by a jinja extension, just like it happened with jsonschema not so long ago (and you created https://github.com/copier-org/jinja2-jsonschema which is great!). That is already an unsafe feature, but it's completely supported right now.

Also, remember (yes, I'm hammering with this, but this is really true!) that jinja supports a lot of features: macro, import, set, namespace... If you ever need a more complex type for your template, you have those features at hand.

At the end of the day, the only thing this feature adds is ergonomy. You can achieve anything today with just jinja.

Put in other terms: if we had this supported, would we need the more complex approach you propose here @sisp? IMHO probably not.

So, my proposal is that we just go forward with the MVP at #1206 (comment) and continue our happy lives.

[pawamoy]

Huge thanks to both of you @sisp and @yajo. Looking at PR #1220, the solution seems so obvious, and is so elegant and convenient. Thanks for the countless hours of thinking, design, writing you put in Copier, making it an absolutely amazing project ❤️