Auto-generating the Plasma Python stubs

[dgelessus]

I'm talking about the .py files under Scripts/Python/plasma (Plasma.py, PlasmaConstants.py, etc.) that show which modules and APIs are defined by the engine in C++. These stubs look like they were automatically generated based on the C++ definitions at some point in the past, but now we seem to be updating them manually whenever the C++-defined APIs change.

Does anyone know if the code to automatically generate these stubs exists somewhere? Or was that lost to time and/or never open-sourced?

If we don't have anything like that yet/anymore, I might try to build a generator for the stubs. It seems like all the necessary information is available at runtime in the in-game Python environment, so it should be possible to write the generator as a script/function that can be run from an internal client's Python console.

Right now, the stubs have a few random mistakes/inconsistencies that I occasionally stumble across (whenever my IDE complains that the Python scripts are calling functions "incorrectly"). Auto-generating the stubs would hopefully help with that.

[dpogue]

Does anyone know if the code to automatically generate these stubs exists somewhere? Or was that lost to time and/or never open-sourced?

I'm not aware of any code to do this, and even the random bits of PythonLib that were included in the original source release don't seem to have anything like that.

Getting these to be autogenerated would be cool. I know in some cases we've added type annotations manually as well, but I'm not sure if those can be known by a runtime generator? It would be very cool if we had proper type annotations for everything.

[dgelessus]

I think the original stub generator took the parameter lists directly from the C++-declared docstrings - note how most of the docstrings have a first line that says "Params: " followed by a parameter list that matches what you see in the stub. I would try to do the same thing, at least initially. In that case, it should be very easy to keep the type annotations, by putting them into the docstrings as well.

[dpogue]

Apparently there is a special docstring format that Python itself uses internally for help() documentation, but it looks like it would also bloat out all our docstrings and be a pain to author for all the existing methods: https://stackoverflow.com/questions/1104823/python-c-extension-method-signatures-for-documentation/41245451#41245451

[dgelessus]

Yeah, I also looked at that, because it seemed worth considering for the future. Unfortunately, the Python core devs seem to consider this docstring format and the related __text_signature__ attribute an implementation detail and don't want third-party code to use it (at least not yet) - see this discussion. So for the moment, I don't think that would have any advantage over the existing custom format.

[dpogue]

I'm not entirely clear on the tradeoffs, but another thing to maybe look at is whether a .pyi file would be any better than the current stub file (with every method implementation calling pass).

I don't think that makes much meaningful distinction in terms of the generator though, so we probably don't need to make a decision immediately

[dgelessus]

Well, our current .py "stubs" already look very much like an equivalent .pyi stub would. If we replace every pass with ... and change the file extension, I think we would have a valid .pyi stub. (Some tools might be unhappy with our very sparse type annotations though - not sure...) I definitely want to look into this after I have the stub generator finished and merged, because then it will be much easier to make large-scale changes to the stubs like this.

[dgelessus]

Update: I had a go at writing a stub generator and it went quite well - I can now run a single call in the Python console to generate a Plasma.py that's pretty close to what we have currently in the repo.

As expected, there are a number of missing/outdated declarations and lots of small inconsistencies that have accumulated over time. I'll make a PR to address those soon.

I haven't yet looked into how we can handle type annotations. My current implementation loses them all, but that's because they're missing from the C++-defined docstrings. Once I fix that, type annotations should work too.

[dgelessus]

Work in progress in this PR: #1661