meta: Managing generated and derived files

[cdecker]

The Core Lightning project relies on a whole range of generated and derived files:

• Wire message files, generated from the protocol specification, for both C and Python source code
• Documentation files, and schema.json aggregating the docs, and metadata files from which they are generated
• msggen.json and similar to track the evolution and historic values over time
• Copies of files that are needed for python packages. These are built in isolation by the buidlers, and so they can't rely on files that aren't in their respective subdirectory.

So far we have defaulted mostly to just committing these files. However, they cause a lot of conflicts, since they aggregate things from a variety of places, causing issues even for changes that are not otherwise in conflict. They have a bit of a submarine effect, i.e., a change in one PR, that causes a later PR to fail, because the derived file was not updated, and not checked, forcing the latter PR's author to fix things they did not break. And finally of course, new contributors often stumble over this convention, making onboarding of new contributors difficult.

I'd like to propose that we stop committing the derived files wherever possible.

The open question I have is whether it is acceptable to ask contributors to call a setup, say make sync to setup all generated files and derived files, before being able to for example bundle msggen or similar packages?