Scaffolding complex structures

[eperdeme]

Hello,

I'm trying to find a tool...any tool....to accomplish the following task - which seems fairly normal in my world.

I wish to have a template repo that I can use as a source to create destination with a post-templatised result based on inputs.

Input

    "env": {
        "dev": {
            "aws_account_id": "111111",
            "private_subnet_ids": {
                "subnets": [
                  "dev-x",
                  "dev-y",
                  "dev-z"
                ]
            }
        },
        "nonprod": {
            "aws_account_id": "222222",
            "private_subnet_ids": {
                "subnets": [
                  "nonprod-x",
                  "nonprod-y",
                  "nonprod-z"
                ]
            }
        },
        "prod": {
            "aws_account_id": "333333",
            "private_subnet_ids": {
                "subnets": [
                  "prod-x",
                  "prod-y",
                  "prod-z"
                ]
            }
        }
    },

Source:
my-src-repo/env/template/config.file
contents:

env=
subnets=
aws_account_id=

Destination:
my-src-repo/env/dev/config.file

env=dev
subnets="dev-x","dev-y", "dev-z"
aws_account_id=111111

my-src-repo/env/nonprod/config.file

env=nonprod
subnets="nonprod-x","nonprod-y", "nonprod-z"
aws_account_id=222222

my-src-repo/env/prod/config.file

env=prod
subnets="prod-x","prod-y", "prod-z"
aws_account_id=333333

I can make cookiecutter and copier make the directories but can't find anyway to then iterate through inputs to match them to the correct variable name.

[mshafer-NI]

handful of thoughts:
a) you could "preprocess" your data set into folders/file via python then call copier to copy the resulting "template"

b) What you're asking for sounds a lot like how ansible handles it's concept of templates and data (https://docs.ansible.com/ansible/latest/collections/ansible/builtin/template_module.html#examples), where vars are stored in a folder, then the template renders the vars into the specified file

c) if you want to do this in copier, you might try storing your source dictionary as default answers in the copier.yaml, then the templates can render out -> although, you may have to do the looping yourself.
something like

{% for key, value in answers.env.dev.items() %}
    {{key}}={{ value }}
{% endfor %}

[ssteinerx]

If I'm understanding what you're after, this almost looks like a direct mapping of a dictionary to a directory tree structure with a config file in each destination.

I'm not sure how the Copier/Cookiecutter part is supposed to make things easier; it's just adding another layer of indirection.

Put the config in your config of choice, YAML, TOML, JSON, and Just iterate through the top level dict, create the directory structure, and squirt out the config file indicated for that key.

And, as @mshafer-NI said, this looks very much like Ansible.

[eperdeme]

We managed to do this in cookiecutter+cruft using the pre_gen_project.py support and a custom python script which create the required directory stucture prior to cookiecutter running against the files.

I can't find any functions in copier to run pre-gen scripts, tasks is post generation and migrations isn't the right place as its on every run.

[yajo]

The design for this feature request already landed in #1271, but still nobody had time to implement it.