Shader Bus | Lanes between Scripts and Shaders - #665
Conversation
|
I recommend you to translate Claudish descriptions into human readable English, this is hard to read and understand |
Which parts are you having difficulty understanding with? If I used Claude here it likely would've have been easier to actually understand probably lol. Also just for the record, I have used Claude in the past and when I do, I don't shy away from noting it as such, the reason I formatted it this way and used this language is because this is a semi-large PR so I wanted it "clear" to follow through. |
From the How section and so on. |
No worries, appreciate the feedback. I'll re-write it in a bit. As for the retargetting, do you want me to make a separate PR to MT branch if there's a conflict? Or I guess this could be the MT PR and the other one I'd open with be base. Going through the rebasing now, looks like there are minimal changes, if any. So I'll just make this target the main branch and it should merge cleanly into MT. The semantic difference is that |
shader_bus keeps one float4 lane per registered id, owned by the registrant and readable by anyone. Lanes are created on demand and never removed. Writes land in a pending value that the frame latch copies to the bound value, so a write never splits across a frame. The registry is dumped to the log after level load.
SetMapping walks the compiled constant table after the generic setup list and attaches a binder to every constant whose name starts with bus_, so a shader reaches a lane by declaring it. The lane resolves at attach time and one binder serves every shader that declares it.
Module shader_bus registers a lane and hands back a token, and only that token can write the lane. Reads, listing and describe are open to any script. register, try_register and list take the calling lua_State through luabind's raw policy, so the registering script is read from the thread that made the call and a duplicate id from a second mod names both scripts in the fatal.
bus_list prints every registered lane with its owner and value, the lanes a shader declared but nobody claimed, and the twelve shared console lanes. bus_get prints one lane. The eight shader_param and four s3ds_param commands log once per session on their first write pointing at the bus.
Each lane counts writes, the set calls made through its token, and changes, the value changes the latch sees, so a lane rewritten every frame with the same value shows many writes and no changes. The binder stamps bound_frame with the last frame a shader read the lane. The bus version is 2. list() rows carry state and source, list(true) adds the lanes a shader declared that nobody registered, stats(id) returns the counters and frames, and get_pending(id) reads the value the owner last wrote. describe and owner_of still return nil for an unknown id. bus_force <id> x y z w holds a lane at finite values, the latch feeds the held value to shaders while the owner keeps writing pending, and bus_release hands the lane back on the next latch. This is a console debug path, the lua write path still needs the token. bus_list prints state, counters and the last bound frame.
The lua console export records the calling script and line for the duration of the execute in a thread local console slot, walking past C frames and the _g.script wrapper so the mod that set the lane is named, and a nested execute keeps the outer caller. A valid write to one of the twelve legacy vector commands records that writer, the config file name before the device is up and console when no script is on the stack. The deprecation nudge fires once per command and writer, and the legacy rows of shader_bus.list carry the last writer.
The level load count line and the per lane value lines now need the -dbg key. A lane declared by a shader and registered by nobody still warns in every log.
SHADER_BUS.md at the repo root next to DXML.md, a patch list entry in the README, and the lua module and console commands in lua_help_ex.
The shared registration body moved to a file static helper so register_lane can refuse a taken lane quietly and let its fatal do the reporting.
63c9ab3 to
73f2b38
Compare
|
Re-based and force pushed. Just tested and the same stress test mod I wrote works perfectly fine on non-mt exes. Let me know if you have any issues. Additionally, I've uploaded the stress test mod itself. Just load it last in MO2 and make sure it wins combine_2_naa for the visual confirmation (rainbow effect), the actual test results will be in the log. |
2 participants
Quick Disclaimer
This is NOT a visual mod. It does nothing to the game visuals, and merely exists to augment/"replace" an archaic system that is being used and abused way beyond what the intended scope for it initially was (quick debugging for developers). There has been a lot of contention around which mods "own" the shader_param(s) and which should not. This PR aims to fix this once and for all.
To re-iterate again, this existing system STILL WORKS. The hope is to slowly move away from using the
shader_param_Nands3ds_param_Nso that mod incompatibility with clashing slots will no longer be an issue.Why
Right now, mods pass values to shaders through eight shared console variables (
shader_param_1toshader_param_8, plus the fours3ds_paramones). These slots have no names and no owners. Any script can overwrite any slot at any time, and when two mods pick the same slot nothing warns anyone. On GAMMA all eight slots are already in use, and two enabled mods currently write toshader_param_5at the same time. A new shader mod has to either take a slot that may already be in use, or modify the engine.How it works
shader_bus.register(id, owner, description)and gets back a token. Only the script holding that token can write the lane. Any script can read any lane.try_registerdoes the same check but returns nil instead of crashing the game.For debugging there are
shader_bus.list(which can also show lanes that a shader declares but no script has registered),shader_bus.stats(how many times a lane was written, how many times its value actually changed, the frame of the last change, and the last frame a shader read it),shader_bus.get_pending, and four console commands:bus_list,bus_get,bus_forceandbus_release.The twelve old console commands keep working exactly as before. The only addition is a log line, printed once per console execution and per writing script, that attempts to name the script file and line number that still writes to that command. This makes it easy to find what is left to migrate on a big load order (i.e. GAMMA).
Documentation:
SHADER_BUS.mdin the repository root, the module and the console commands inlua_help_ex.script, and an entry in the README.Performance
Once per frame the engine loops over the lane list, which holds a few dozen lanes on a typical load order, and copies one float4 per lane. Binding a lane to a shader constant is one integer store per bound constant. Shaders pay nothing extra, because the value arrives as an ordinary constant. Code that does not use the bus is not affected, and the old commands do exactly what they always did. The full list of lanes is printed at level load only when the game runs with
-dbg.Testing
I wrote a stress test mod with 53 checks and ran it on a full GAMMA profile. It covers the id, owner and description length limits, every which way a registration can be refused, the one frame delay between a write and the shader seeing it, the change and write counters, the "last bound frame" value, the console hold and release commands, NaN and infinity values, the writer attribution for the old commands (including writes that go through the
exec_console_cmdwrapper in _g.script), and 64 lanes written every frame for 300 frames with every read checked. All 53 checks pass. The duplicate owner error was verified with a second mod that deliberately registers an id the first mod already owns.I made a patch for 3DSS 5.01.2 that moves its scripts and shaders onto 13 lanes for example. Regular play and debugging showed no visible issues. The patch is attached as a proof of concept. 3DSS Shader Bus Patch.zip
An optional Shader Bus Monitor mod is in development. It shows the registered lanes and their values on screen while playing. If you would like it included in this PR for completeness, let me know.