Some fixes and features to make it easier to access data

[kamilkrzyskow]

I fat fingered the enter key on title and it created an empty PR 🙄

Anyway, I always wanted to be able to access script data from Python, there is PyDecDat but it doesn't support Ikarus/Lego extended scripts, and iirc was rather slow.
So for now I typically used DecDat and parsed the .d file with Python to extract data.

When I saw this project I was happy that I don't have to parse the .d myself 😄...or so I thought.
Turned out there are some kinks to be straightened out, so I will fix them along the way.
Edits for maintainers are enabled, so if you want to adjust some things, then go ahead ✌️

My goal is not to create another DecDat, or so I think for now 🤔
My goal is to easily extract data, and cross-connected it afterwards in my external scripts.

Example of previously extracted data about dialogues of an NPC together with their possible routines:

"NONE_2246_BRADLOCK": {
  "dialogues": {
    "DIA_BRADLOCK_AFTERMINE": "",
    "DIA_BRADLOCK_AMBIENT": "How are you doing?",
    "DIA_BRADLOCK_CANTPASS": "",
    "Some": "Other"
  },
  "id": 2246,
  "instance_name": "NONE_2246_BRADLOCK",
  "name": "Bradlock",
  "routines": [
    "START",
    "Q308",
    "GUARD",
    "VOLKERTRIALOG",
    "TOT"
  ]
},

[lmichaelis]

Hi! Thank you very much for submitting a PR! I like the changes you're proposing, however I can see an issue with setting the string encoding. The way it is now, whenever zenkit._native.load is called with an encoding, it will change the encoding for all strings loaded by the library. This is unexpected behaviour, I'd think.

I propose instead, to add a function for setting the encoding globally and then applying ot to all string encoding and decoding performed by the library (e.g. a zenkit.set_string_encoding function). This should then also replace the hardcoded "windows-1252" in all str.encode calls (you don't need to do that right now, but that'd be the plan).

[kamilkrzyskow]

What's the "real" functional difference between DaedalusScript and DaedalusVm? The latter is a subclass, and the load function has a different value. I began working with DaedalusScript, but then trying to load DaedalusInstance.from_native resulted in INVALID type. Fortunately, the from_native is used in the code, so I could find that the symbol first needs to be initialized with DaedalusVm.init_instance. Would it be possible to move the init_instance to DaedalusScript?

Would it be possible to detect if a symbol is a function parameter inside the DLL, other than the .PAR suffix?
I was thinking about adding a is_param which would check the name, but perhaps there is something more performant 🤔

The DaedalusScript.symbols property generates a list of symbols every time it's accessed. This is a huge performance hog.
I'm not sure whether this should be cached (either with LRU / directly as self._symbols), or replaced with a generator to force the user to make their own list(symbols).

Same goes to other aspects, like the get_parent, I'm not sure whether or not to cache parent indexes for faster lookup in subsequent symbols 🤔

Lastly, it seems that DaedalusVm requires registering externals, even the defaults from Gothic. Why can't ZenKit provide them with the DLL, is there some licensing issue?

[lmichaelis]

The difference between DaedalusVm and DaedalusScript is, that the script cannot run any Daedalus code. Its purpose is to facilitate access to all symbols in the binary as well as the bytecode instructions associated with them. The VM is capable of actually running code.

Daedalus instances are runtime-initialized objects, meaning they have to be explicitly initialized by the engine by running some Daedalus code. That's the job of the VM. So no, init_instance can't be moved into DaedalusScript. You can, however, create a VM the same way as a script, and as you already noticed, you will retain access to all functionality of DaedalusScript.

---

Getting the parameters of a Daedalus function isn't as simple as checking for a .PARn suffix. ZenKit has a function for getting the parameters of a given function, though I am not sure if it's exposed to Python right now. I am unsure if an is_param on symbols is feasible with the current DAT parser.

---

Caching is something I haven't considered yet, because there are many cases in ZenKit where caching is the wrong answer. In those examples specifically though, I think caving would be a valid strategy, because the script itself is immutable. A generator could be great too.

---

Externals are not registered by ZenKit, because most of them have something to do with game engine stuff, which ZenKit is not concerned with. Of course, things like CONCATSTRINGS could be implmented without issue, but WLD_InsertNpc is another story.

[kamilkrzyskow]

Hello again, I wish you a belated happy new year ^^
as for the progress on the PR since last time, the PR became a victim of me attaching my personal project to the PR, so other projects got in the way, but I'm back at it, and even sort of got it working ✌️

However, there is some bug, which is possibly there in 1.3.0 or I'm missing something about memory management and DAT file access.

"$INSTANCE_HELP": {
  "crashed": "Init Failed - NPC is None"
},
"BDT_10030_ADDON_BUDDLER": {
  "crashed": "exception: access violation reading 0x000000474E49525C"
},
"BDT_10031_ADDON_WACHE": {
  "dialogues": {
    "DIA_ADDON_BDT_10031_WACHE_HI": "Alles klar?"
  },
  "id": 10031,
  "instance_name": "BDT_10031_ADDON_WACHE",
  "name": "Wache",
  "routines": [
    "START"
  ]
},

As you can see there are NPC Instances, which crash during initialization (OSError), I think this happens with Info Instances as well, but if that happens I skip them without any logging. The worst part is that, the output of my script changes every time, I mean the amount of crashed values found in the final file change every time the script runs python script.py, and there is also some rule to it, as BAU_4300_ADDON_CAVALORN, PC_HERO, crash more often than the other instances, but this could be a false positive.

Sometimes, but not always, the script also shows this Traceback:

$ python script.py 
Exception ignored in: <function DaedalusScript.__del__ at 0x000002203C8216C0>
Traceback (most recent call last):
  File "...\DecDat-Zenkitpy_project\venv\Lib\site-packages\zenkit\daedalus_script.py", line 277, in __del__
    self._deleter()
  File "...\DecDat-Zenkitpy_project\venv\Lib\site-packages\zenkit\daedalus_vm.py", line 203, in _deleter
    DLL.ZkDaedalusVm_del(self._handle)
OSError: exception: access violation reading 0xFFFFFFFFFFFFFFFF

Sometimes, but not always, when I enable the Debug LogLevel, I get different errors;

2025-01-20 23:33:46 [ZenKit] (ERROR  ) ÔÇ║ DaedalusVm: Internal Exception: illegal access of type 2 on DaedalusSymbol PRINT_KOSTEN which is another type (3)
2025-01-20 23:33:46 [ZenKit] (ERROR  ) ÔÇ║ DaedalusVm: Internal Exception: illegal access of type 2 on DaedalusSymbol PRINT_LP which is another type (3)

there can be also no errors in the Terminal, and crashed values would still be there in the output 😞

Here is the script.zip, it expects an exported _COMPILED/BASE_GOTHIC.DAT next to the script.py. I started off with using Archolos binaries, but the crashes appear also in base Gothic 2 NotR. The script has additions from the PR to make it work in 1.3.0

Any advice how could I debug it? I was thinking of setting externals to lessen the burden of the default skip behaviour in case of missing externals, as this could perhaps improve the state of the stack, but I don't know which external would be most problematic and I don't know if the randomness is even related to the externals or the stack.

[kamilkrzyskow]

Next day of investigation about the random crashes. I've added some externals and started getting this error

RecursionError: maximum recursion depth exceeded

I don't really understand where the recursion is, but looking at the Traceback it's related to the DaedalusInstance being constantly converted into other proper NpcInstance ItemInstance for each external, so perhaps adding externals makes use of the cb() callback, and somehow there is a chain created 🤔

Exception ignored on calling ctypes callback function <function DaedalusVm._register_external.<locals>.<lambda> at 0x0000015599AF8540>:
Traceback (most recent call last):
  File "...\zenkit\daedalus_vm.py", line 198, in <lambda>
    fptr = DaedalusVmExternalCallback(lambda _, __: cb())
  File "...\zenkit\daedalus_vm.py", line 177, in _wrapper
    vals = [self.pop(arg) for arg in reversed(args)][::-1]
  File "...\zenkit\daedalus_vm.py", line 109, in pop
    return self.pop_instance()
  File "...\zenkit\daedalus_vm.py", line 133, in pop_instance
    return DaedalusInstance.from_native(handle)
  File "...\zenkit\daedalus\base.py", line 60, in from_native
    typ = DaedalusInstanceType(DLL.ZkDaedalusInstance_getType(handle))
RecursionError: maximum recursion depth exceeded
Traceback (most recent call last):
  File "...\script.py", line 178, in <module>
    main()
  File "...\script.py", line 62, in main
    if can_skip_symbol(vm_sym):
  File "...\script.py", line 50, in can_skip_symbol
    return sym.is_member or sym.type not in (DaedalusDataType.INSTANCE, DaedalusDataType.FUNCTION)
  File "...\zenkit\daedalus_script.py", line 180, in type
    return DaedalusDataType(DLL.ZkDaedalusSymbol_getType(self._handle))
RecursionError: maximum recursion depth exceeded

Worst part is that I still get random results, sometimes there is no RecursionError, but I get this corrupted external name Missing externals {'p┬že┬ę├╣\x7f'}, but the name could also contain non-ASCII characters, and my terminal has some issues with UTF-8 characters and Python, so this could be an issue with the encoding in the Terminal 🤔 Still, I doubt that the randomness is related to encoding.

script_with_externals.zip

[lmichaelis]

Hm I do see a crash with G2 NotR with your first script, so there's probably a bug here. I'll hopefully have time to investigate a bit on the weekend. In the mean time, the only good way to debug this is to add debug prints in the C/C++ source, then recompile the native library, since the crash happens in C-land, inside ZkDaedalusVm_initInstance.

[lmichaelis]

Okay @kamilkrzyskow, I did find at least one issue which, if fixed, seems to resolve your problem. Before calling init_instance, you should check that the symbol you're trying to initialize is_const, otherwise you'll be trying to initialize function parameters and variables too.

You could, for example, add that check to this:

if class_sym and vm_sym.is_const:
    ...

There is a bug with externals still though, which causes the program to crash when an incorrect parameter is taken. I've fixed that in GothicKit/ZenKitCAPI@2a1b554 and d9fedd0 :)

[kamilkrzyskow]

Thanks a lot for the fix 🫶 ! Also sorry for this blunder 😞 over the many iterations of random errors I've seen some NAME.SLF, but I didn't give this much thought, as I had the is_member check, but function variables aren't members... If I didn't take this long break, then perhaps I would figure it out on my own, but the errors don't really help 😅

As for other observations I've had from my shenanigans:

• vm_sym.value for INSTANCEs, will always instantiate them anew, so if somebody isn't careful they might fill up their memory with constant recreation of instances. This kind of happened in this case, as to check if isinstance(info, InfoInstance) I needed the info = vm_sym.value, and later I instantiate the NPC from info.npc, so it's created twice. The game also kind of allows to insert the same NPC twice into the world, so multiple memory allocations is expected, but I still feel this should be somehow addressed to prevent bad coding🤔
• I tried to add caching to get_parent_as_symbol to limit the amount of indirection and looking for the same index of C_INFO, but I saw no significant improvement. I profiled the main() using time.perf_counter, and in best case scenario it went down from ~2.4s to ~2.3s, but the results varied so much that I consider this as a false positive.
• Ctrl+C aka KeyboardInterrupt doesn't interrupt the script, and combined with the default logger, which doesn't cache the messages it can happen that a bunch of error messages will fill up the terminal, and there will be no other choice than to wait for the program to finish writing out all of these messages.

So with this, the dialogue and routine (script above needed a small fix) extraction solution is superior to my previous DecDat -> Regex/Text extraction method, and less buggy 🥳

I was thinking about simplifying vm.get_symbol_by_index(info.npc) into info.get_instance_of(attribute: str), however, given the point above about instantiation and memory allocation, then I guess this abstraction could get quite costly 🤔

So I'm getting closer to being happy with this PR, thanks again ✌️

[lmichaelis]

vm_sym.value for INSTANCEs, will always instantiate them anew, so if somebody isn't careful they might fill up their memory with constant recreation of instances. This kind of happened in this case, as to check if isinstance(info, InfoInstance) I needed the info = vm_sym.value, and later I instantiate the NPC from info.npc, so it's created twice. The game also kind of allows to insert the same NPC twice into the world, so multiple memory allocations is expected, but I still feel this should be somehow addressed to prevent bad coding🤔

I was at first confused but I think I understand now. The way you implemented .value for instances is not actually the way it should be done. I have a patch locally which adds that functionality in the native library, which I can make available. Essentially, you'll get an get_instance() function, analogous to get_{string, int, float}() which will return an already initialized instance bound to that symbol. This would mean that you initialize the instance once and then you can access it directly from the underlying symbol its attached to.

Generally, you should not use the _keepalive member, because in this case especially, it could also be pointing to a DaedalusScript instance, not necessarily a DaedalusVm. Instance instantiation should only be done externally in client code, i.e. not in the library itself.

Ctrl+C aka KeyboardInterrupt doesn't interrupt the script, and combined with the default logger, which doesn't cache the messages it can happen that a bunch of error messages will fill up the terminal, and there will be no other choice than to wait for the program to finish writing out all of these messages.

You can write your own logger if you like. Just use set_logger and give it a level and a callback function like (LogLevel, str, str) -> None.

[kamilkrzyskow]

which I can make available. Essentially, you'll get an get_instance() function

I like the sound of that, instantiation-aware instances 🤩 I like how you have solutions already in the making to the issues I face 😆

_keepalive member, because in this case especially, it could also be pointing to a DaedalusScript instance, not necessarily a DaedalusVm

That's why there is the protection with hasattr in the PR, I omitted those in the local scripts, as it was only ported over for testing the Vm. Once your patch is available, then DaedalusInstance.get_instance_of(attribute: str), would be feasible.

You can write your own logger if you like

Thanks, I've written one for debugging the vm_sym during an ERROR log, however due to the randomness before it didn't get much use. It lacks the coloring of sections, but I didn't need it.

class Container:
    logger_messages = dict()

def main():

    # Setup logger
    vm_sym_name = "*"
    
    def logger_callback(lvl: LogLevel, name: str, message: str):
        count = Container.logger_messages.get(message, 0)

        if count == 0:
            nonlocal vm_sym_name
            print(lvl.name, name, vm_sym_name, message, sep=" - ")

        Container.logger_messages[message] = count + 1

    set_logger(LogLevel.DEBUG, logger_callback)

    # ...

    for vm_sym in vm.symbols:

        vm_sym_name = vm_sym.name
        
        # ...

[kamilkrzyskow]

There is a bug with externals still though, which causes the program to crash when an incorrect parameter is taken.

Small progress with the externals, as I use 1 in my next extraction project. My bug came from reading the parameters from the DecDat extraction header comment.

// func void CreateInvItem(var __class par0, var __class par1);
// func void CreateInvItems(var __class par0, var __class par1, var int par2);

Where it says __class I assumed it's the DaedalusInstance, however I noticed on the example page that the itemInstance was of type int. So after checking the source, I've learned that in those functions above the item instance should be also an int, just like the example.

[lmichaelis]

I've add DaedalusSymbol.get_instance in 85484af. You need to call DaedalusVm.init_instance on the symbol in order to get a value return from get_instance.

[kamilkrzyskow]

Hello again,
my next extraction project, which I mentioned before, is trade item extraction per chapter. To analyze the requirements I went with the bottom up approach (which turned out to be a mistake...). Archolos adds items to NPCs in functions that look like this:

func void b_givetradeinv_veit(var c_npc slf) {
    if ((kapitel >= 1) && (veit_itemsgiven_chapter_1 == false)) {
        CreateInvItems(slf, itse_arrowpacket_10, 5);
        CreateInvItems(slf, itse_boltpacket_10, 5);
        // ...
        veit_itemsgiven_chapter_1 = true;
    };
    if ((sq227_veitmarket == 2) && (veit_itemsgiven_city == false)) {
        CreateInvItems(slf, itse_arrowpacket_10, 2);
        CreateInvItems(slf, itse_boltpacket_10, 2);
        // ...

So in my previous text-base and regex-based approach I scanned for such lines. However, here the Vm only executes the code and doesn't show what variables are being accessed / assigned / compared. So I was almost at the point of return to go back to my old ways. However, my previous script didn't cover all cases, like typos in variables, infinite re-adding of items, etc., so I really wanted to be able to find all permutations and compare the results with it. Hence, from the looks of it I had to implement a DecDat-like function, which goes over the instructions in OP code... The docs really helped https://zk.gothickit.dev/engine/formats/bytecode/#opcode 👍, but the naming could be improved on the Python side to make it more readable like BL -> CALL etc., but I don't know assembly to know if Python has a "style guide" to naming such variables here.

This is what I came up with:

def get_variables_used_in_function(vm: DaedalusVm, func_sym: DaedalusSymbol):
    """Get all used variables in the function to later try out different permutations of values"""

    @dataclass
    class Flags:
        found_return: bool = False
        found_next_symbol: bool = False

        def __iter__(self):
            return iter(asdict(self).values())
    
    Flags = Flags()
    variables = set()
    current_address = func_sym.address
    
    while not all(Flags):
        instruction = vm.get_instruction(current_address)
        addr_symbol = vm.get_symbol_by_address(current_address)

        if instruction.op == DaedalusOpcode.RSR:
            Flags.found_return = True
        elif Flags.found_return and addr_symbol:
            Flags.found_next_symbol = True
            continue  # End of loop, no need to get size
        elif instruction.op == DaedalusOpcode.PUSHV:
            index_symbol = vm.get_symbol_by_index(instruction.symbol)
            variables.add(index_symbol.name)

        current_address += instruction.size

    return list(variables)
    
# ['VEIT_ITEMSGIVEN_CITY', 'KAPITEL', 'VEIT_ITEMSGIVEN_CHAPTER_1', 'FALSE', 'TRUE', 'SQ227_VEITMARKET']

Afterwards I did the usual shtick and looped over the vm.symbols and the change to require to init the instances from the client side became, somewhat cumbersome 🤔 Even after adding the automatic guessing of types in vm.init_instance.

for vm_sym in all_symbols[1:]:
    if can_skip_symbol(vm_sym):
        continue
    
    if _CLASS_TYPES.get(vm.get_parent_symbol(vm_sym, find_root=True).name) != DaedalusInstanceType.INFO:
        continue

    info_instance: InfoInstance = vm.init_instance(vm_sym)
    
    if not info_instance.trade:
        continue

    # Initialize the NPC here so that every instance is in memory?
    trade_npc_sym = vm.get_symbol_by_index(info_instance.npc)
    npc_instance: NpcInstance = trade_npc_sym.value
    if npc_instance is None:
        npc_instance = vm.init_instance(trade_npc_sym)
        
# ...

for trade_info_sym in trade_info_symbols:
    info_instance: InfoInstance = trade_info_sym.value
    if info_instance is None:
        info_instance = vm.init_instance(trade_info_sym)
    
    trade_npc_sym = vm.get_symbol_by_index(info_instance.npc)
    npc_instance: NpcInstance = trade_npc_sym.value
    if npc_instance is None:
        npc_instance = vm.init_instance(trade_npc_sym)

    if trade_npc_sym.name not in chapter:
        npc_handle = chapter[trade_npc_sym.name] = {}
    
    trade_npc = npc_instance
    trade_func = vm.get_symbol_by_index(info_instance.information)

    vm.global_self = trade_npc
    vm.global_other = trade_npc
    vm.call(trade_func, trade_npc)

idk, it just feels a bit weird or "clunky" to use for a Python enjoyer like myself ✌️

Later it turned out that trade_instance.information first calls a "manager" function B_GIVETRADEINV (my mistake for not going top to bottom), which later runs the appropriate function for the NPC like b_givetradeinv_veit, and again I faced the issue where I lack exposed information. The vm.call executed the function, but I don't see how I can check what other functions are being called in order, so I can't call my function to extract the variables🤔

So I felt like another DecDat-like function is needed:

def get_trader_functions_calls(vm: DaedalusVm):
    """
    Archolos trade manager function is B_GIVETRADEINV(C_NPC slf), 
    so data extraction is required to get the proper functions.
    """

    @dataclass
    class Flags:
        found_return: bool = False
        found_next_symbol: bool = False

        def __iter__(self):
            return iter(asdict(self).values())
    
    Flags = Flags()    
    current_address = vm.get_symbol_by_name("B_GIVETRADEINV").address
    pushed_instances: list[DaedalusSymbol] = []
    pushed_ints: list[int] = []
    global_npcs: list[DaedalusSymbol] = []
    local_sym_to_global_map: dict[str, str] = {}
    func_sym_to_local_map: dict[str, str] = {}

    def _recent_local_trd() -> str:
        nonlocal pushed_instances
        for sym in reversed(pushed_instances):
            if ".TRD_" in sym.name:
                return sym.name
    
    while not all(Flags):
        instruction = vm.get_instruction(current_address)
        addr_symbol = vm.get_symbol_by_address(current_address)
        op = instruction.op

        if op == DaedalusOpcode.RSR:
            Flags.found_return = True
        elif Flags.found_return and addr_symbol:
            Flags.found_next_symbol = True
        elif op == DaedalusOpcode.PUSHI:
            pushed_ints.append(instruction.symbol)
        elif op == DaedalusOpcode.PUSHVI:
            pushed_instances.append(_get_by_index_or_address(vm, instruction))
            pushed_name = pushed_instances[-1].name
            if global_npcs and pushed_name not in local_sym_to_global_map:
                local_sym_to_global_map[pushed_name] = global_npcs[-1].name
        elif op == DaedalusOpcode.BL:
            maybe_sym = _get_by_index_or_address(vm, instruction)
            if maybe_sym.name.startswith("B_GIVE") and maybe_sym.name not in func_sym_to_local_map:
                func_sym_to_local_map[maybe_sym.name] = _recent_local_trd()
        elif op == DaedalusOpcode.BE:
            maybe_sym = _get_by_index_or_address(vm, instruction)
            if maybe_sym.name and maybe_sym.name == "HLP_GETNPC":
                global_npcs.append(vm.get_symbol_by_index(pushed_ints[-1]))

        current_address += instruction.size

    merged_map: dict[str, str] = {}
    for key, value in func_sym_to_local_map.items():
        npc = local_sym_to_global_map[value]
        merged_map[npc] = key

    return merged_map
    
# len(merged_map) == 97
# {'BAU_11041_SHEPHERD': 'B_GIVETRADEINV_SHEPHERD',
#  'BAU_2243_GUMBERT': 'B_GIVETRADEINV_GUMBERT',
# ...

So now I have both the NPCs and their functions, I don't even need to go over the vm.symbols, but I fear more and more DecDat-like functions would need to be implemented to fetch what data is being processed at a given time.

Like mentioned in the past:

My goal is not to create another DecDat, or so I think for now 🤔
My goal is to easily extract data, and cross-connected it afterwards in my external scripts.

Would it be possible for ZenKit4Py to somehow make things easier here?

Generally, you should not use the _keepalive member,

I added the get_parent_symbol to the DaedalusScript itself, so that the Symbol doesn't need to use _keepalive (haven't removed the other commit yet), but now it feels a bit weird 🤔
Off-topic: The typical conundrum of game dev. Should the torch be aware that it's equipped if torch.isEquipped() -> torch.lightUp() or should the character have a container for the equipped items and handle the activation if leftHandSlot.hasTorch() -> torch.lightUp(). Such cases of interlinking dependencies always made my blood boil as there is no "good answer" 😞I get the same feeling here with using _keepalive

[lmichaelis]

Heyo, providing a better API for script decomp is currently not really in scope for ZenKit itself I think.

Have you tried calling these functions using the VM and either registering externals for the CreateInvItem calls? I see I've also not implemented the override_function method on the VM in the Python wrapper. That method allows you to override any Daedalus function with one in Python, like you can with externals. Would that be helpful?

Maybe interesting as well: https://github.com/GothicKit/mdd (see Releases as well)

[kamilkrzyskow]

Thanks for the quick reply, and your time.

Heyo, providing a better API for script decomp is currently not really in scope for ZenKit itself I think.

Totally understandable, since other tools are targeted for this purpose.

Maybe interesting as well: https://github.com/GothicKit/mdd (see Releases as well)

I did try mdd after starting this PR:

I did have some issues with how it works, which could probably be resolved, but I would ultimately end up with text based processing of the .d code (as I don't plan to work with Java), so I didn't go with that route. If you want I could open up an Issue over at the other repository to discuss them ✌️

Have you tried calling these functions using the VM and either registering externals for the CreateInvItem calls?

I'm not sure what you mean with "calling these functions using the VM", but I guess you mean calling b_givetradeinv_veit directly without going the trade_info.information route? That way I still don't have the cross connection for NPC -> CreateInvItems, because it requires an NPC handle, and without going through trade_info.information or my get_trader_functions_calls function I don't have the data to input into CreateInvItems.

I guess gathering the trade items for the functions and not the NPCs would be easier, and then doing some post-processing to replace the function names with their NPCs would maybe be even more performant than constantly resolving vm.get_symbol_by_index(npc_instance.index), but I don't want to go this route for now. Oh and also TBH every Instance should have a reference to the symbol. Hence, I added it for direct init_instance, but it's not there when popped from the stack 😞

I mean the issue is resolved, I just mentioned my process, to hint at me maybe going circling around the proper solution.
I did use the CreateInvItems external to keep track of the added items (for now only printed them):

def createinvitems(npc: DaedalusInstance, item: int, amount: int) -> None:
    nonlocal chapter  # Handle for chapter dict, chapter[npc_sym.name]
    npc_sym: DaedalusSymbol = vm.get_symbol_by_index(npc.index)
    item_sym: DaedalusSymbol = vm.get_symbol_by_index(item)
    print(f"<local>{npc_sym}, {item_sym}, {amount}", flush=True)
    return None
    
vm.register_external(*_get_registration_data(createinvitems))

But after you mentioned it now, I had the 💡 OMG 🤦 realization that I have not tried to run the vm.print_stack_trace() inside of it... I've printed the stack only after vm.call(trade_func, trade_npc) or maybe before and the CALL STACK was always empty, well duh... I also tried to use LogLevel.TRACE, but didn't see anything helpful in the output.

nonlocal printed_trace
if not printed_trace:
    vm.print_stack_trace()
    printed_trace = True

# ERROR - DaedalusVm - DIA_GUMBERT_TRADE - ------- CALL STACK (MOST RECENT CALL FIRST) -------
# ERROR - DaedalusVm - DIA_GUMBERT_TRADE - in B_GIVETRADEINV_GUMBERT at 2fff98
# ERROR - DaedalusVm - DIA_GUMBERT_TRADE - in B_GIVETRADEINV at 312772
# ERROR - DaedalusVm - DIA_GUMBERT_TRADE - in DIA_GUMBERT_TRADE_INFO at 399dbf

Now a train of thought from the top of my head. The stack is printed to the stdout, so I would have to redirect it to some file and read it later again or redirect to some string IO object (I'm not sure if there is one in Python, but should be doable with a custom class that proxies the io.write function into appending to a string). The information comes also late, because it's coming from inside the CreateInvItems, which is already inside of a (kapitel >= 1) && (veit_itemsgiven_chapter_1 == false) conditional, so I would probably need to do a double execution to first get the call stack, then get all variables inside the function read from the call stack, then execute the function again with controlled permutations of the variables.

So maybe it would be cool to get a vm.get_call_stack_items 🥹

I see I've also not implemented the override_function method on the VM in the Python wrapper. That method allows you to override any Daedalus function with one in Python, like you can with externals. Would that be helpful?

If it would work in similar fashion to Union hooking, where the old function is also somehow available then it would be helpful I think.

# vm is global, or the function would be a member of the DaedalusVm object
def decorator(func_sym: DaedalusSymbol):
    def wrapper(some: int, args: str, with_: float, type_hints: DaedalusInstance) -> None:
        print("wrapper did execute")
        return vm.call(func_sym.name + "_old", some, args, with_, type_hints, rtype=":hmm: reading it from the wrapper would be best, but don't know if it's possible")
    return wrapper

vm.override_function(func_sym, decorator)

In general I find it funny that Daedalus sort of forces the usage of globals, or nonlocal accessing of variables 😆

[lmichaelis]

Hm it's likely that I don't understand your exact use-case, but adding an API for retrieving the call stack would be possible. I will come up with something and get back to you :)

[kamilkrzyskow]

it's likely that I don't understand your exact use-case

Oh don't worry, perhaps I do not understand it myself 😅 😆

So this approach works in Archolos:

def createinvitems(npc: DaedalusInstance, item: int, amount: int) -> None:
    # SG is SharedGlobals
    if SG.get_func_from_stack:
        SG.is_printing_trace = True
        with io.StringIO() as stack_trace:
            with contextlib.redirect_stdout(stack_trace):
                SG.vm.print_stack_trace()
            SG.last_npc_give_func = get_last_call_func_name(stack_trace.getvalue())
        SG.is_printing_trace = False
        # No need to refech last_npc_give_func, so set False early
        # However, it should be also set to False after the information func call
        SG.get_func_from_stack = False
        return None
     # ...

However, I've noticed there's a buggy case with this approach, as the external createinvitems needs to run inside of the called function to expose the call stack with the function name, and I need the function name to extract the variables used inside of that function using the other OP code processing.

So if there is a function with a guard statement if something_happened == TRUE -> createinvitems(...) and that something_happened is initially FALSE, then the external won't be called, and I won't be able to find out via the call stack what function got called, and I won't be able to find that it needs something_happened to work.

but adding an API for retrieving the call stack would be possible

So instead of vm.get_call_stack_items, which would return symbols of functions in the current call stack, I would likely need something more akin to stack_trace = vm.call(func_sym, return_stack_trace_dict=True), which would amass a "trace" of called functions / variables during that vm.call

[EDIT: So in other words, in OP code language, I would like to be able to hook the OP.CALL and OP.CALL_EXTERNAL to know when it executes, perhaps no need to process the stack trace, "just" expose a hooking mechanism for these actions. I understand this might not be "exposable", but worth asking 😸]

I had like 3-5 rewrites of my approach for this traders items case (still not finished), and only now I see that working with ZenKit is more akin to being inside of the "daedalus maze", and I need to find all of the correct keys (variables set to certain values) and I don't know what those keys are, so I need to break the lock to see inside (OP code processing) to know what keys could fit to that lock and later after opening the door I get all of the goodies like already processed DaedalusSymbols and Instances with parsed data inside of the code to extract.

And processing DecDat exported scripts is more akin to seeing the map of the maze, with all of the information about possible doors and combinations, however the map is only on paper I need to load it into the code again deserialising it again into some kind of structure.

Initially, I thought it would be easier to get all of the data with ZenKit, since I'm already "inside of the game", as I didn't want to write a proper parser for the text if-statements to extract all possible variables and to do things properly, however I see I would still have to do it here in ZenKit to be 100% correct with my approach😞

Still, for now I'm slowly progressing with my brute force approach, where I call the same function with multiple permutations of variables multiple times😃

Side note: ZenKit supports reading directly from VDFs, so it should save time on doing the manual steps like exporting the GOTHIC.DAT and later exporting the decompiled text from a decompiler. So I still believe in the change to ZenKit, so I'm continuing ✌️

[lmichaelis]

So if there is a function with a guard statement if something_happened == TRUE -> createinvitems(...) and that something_happened is initially FALSE, then the external won't be called, and I won't be able to find out via the call stack what function got called, and I won't be able to find that it needs something_happened to work.

This really does sound like an issue where an AST would be better suited. MDD does actually generate one from the bytecode, so maybe that'd be a good place to start. You could port that implementation to Python and then you'd have a fully capable bytecode -> AST generator with which you could do exactly this.

Just scanning the bytecode for specific CALLs to the external wouldn't be enough because after finding one, you'd need to backtrack to the last BRANCH instruction to figure out the branch it was in.

Maybe another option would be to make MDD capable of outputting the AST as JSON or some such so it could be imported again in Python? Maybe something like this Gothic2_GOTHICDAT.json.zip?

[kamilkrzyskow]

Hmm, I never worked with AST data, but I can imagine that a JSON file generated for Archolos would be quite big 😅, so maybe another data type like YAML or somehow SQLite would be better.
Also I guess that a tree like structure with branching paths would have a similar difficulty as reading the OP code, though a bit easier since the interconnection of function calls to specific blocks would be already nested, still deeply nested trees might require using recursion and would hit the max recursion depth...

Here is an old output file from 2022:
traders_en.json

The solution sort of worked, as I separated the "blocks" with { and DecDat exported each if-statement on a single line.
However, I had a separate scripts to 1st clean up the DecDat output (+encoding), 2nd extract based on type, 3rd extract INFO instances, 4th extract NPC / ITEM instances, 5th extract traders, 6-7th merged traders with item / npc data. So the workflow after each Archolos update was to VDFs extract Gothic.DAT, export DecDat data, run each script in order. Lastly for the traders I had to update Google Sheets. 😅 So the next iteration most likely won't be a Sheets, but some plain site with DataTables loading in parsed JSON.

So yeah after a longer break between updates of Archolos my setup became too cumbersome to manage, so I'm looking for alternatives before Archolos 2.0 ✌️

While writing the message I saw your edit above, so I'll take a look at it 🕙 ...
... 🕐 So yeah, the minified file is 30MB and after downloading Prettier to format it became 60MB, just for Gothic 2

I do see value in seeing the variables inside separated into blocks 👍 Would be nice to be able to either target specific functions or only limit the output to FunctionDecl kind, as I can extract the rest of the data about instances from ZenKit... I think so at least, but I haven't tried to extract most of the things yet.

Going back to the example before with the call stack:

------- CALL STACK (MOST RECENT CALL FIRST) -------
in B_GIVETRADEINV_GUMBERT at 2fff98
in B_GIVETRADEINV at 312772
in DIA_GUMBERT_TRADE_INFO at 399dbf

The function B_GIVETRADEINV_GUMBERT doesn't have a 1:1 connection with the NPC BAU_2243_GUMBERT.
When parsing the DecDat text output I probably used the suffix (which doesn't always match) to know if it's the included in the NPC symbol name.
So ZenKit makes this easier, as I can set vm.global_other to the NPC instance and run the BAU_2243_GUMBERT.information function, which would resolve the proper path. Assuming that externals are set... oh how many bugs I had because of missing externals set to pass on data 😓

func void b_givetradeinv(var c_npc slf) {
    var c_npc trd_veit;
    var c_npc trd_bastian;
    var c_npc trd_keth;
    var c_npc trd_gumbert;
    // ...
    trd_gumbert = Hlp_GetNpc(bau_2243_gumbert);
    // ...
    if (Hlp_GetInstanceID(slf) == Hlp_GetInstanceID(trd_gumbert)) {
        b_clearfakeitems(slf);
        b_clearjunktradeinv(slf);
        b_givetradeinv_gumbert(slf);
    };

Therefore, I don't think that the AST is the singular best solution which would solve all of the issues, as I would have to traverse DIA_GUMBERT_TRADE_INFO call together with the conditional tree of the B_GIVETRADEINV function to extract the B_GIVETRADEINV_GUMBERT call. Looks rather similar to processing OP code to me 😄 It's definitely easier, so perhaps it would open up a path for parsing C_INFO.conditional, which is also a part of the issue, as some dialogues are blocked via some condition, and this condition isn't present inside of the B_GIVETRADE function, because it's not needed there, as the player wouldn't access it without the dialogue. I planned to brute force it as well, but didn't get to this part of the problem yet...

Currently in my imagination this solution seems "best" -> Let's say that I could detect the function calls that happen in ZenKit during vm.call and then use that information to directly access a JSON file generated with MDD containing only the specified called function via some flag

javaw.exe -jar mdd.jar --path-to-dat ../gothic.dat --export-ast-to-json ./ast.json --only-symbol B_GIVETRADEINV_GUMBERT

executed via subprocess, then I would load the JSON and use the data provided to fill out the blanks of ZenKit. However, this would likely load the Gothic.DAT anew each time the command is run, so it would be hellishly slow.

For now I want to finish the brute force approach, to see how close I can get to a 100% solution ✌️

[lmichaelis]

So is there anything specific you need from me right now?

[kamilkrzyskow]

Thanks for the continued support 🫶

The vm.override_function(func_sym, decorator) mentioned in #5 (comment) doesn't seem so helpful anymore, as this would require knowing what func_sym to override.
I initially wanted to go over func_sym.name.startswith("B_GIVEITEM"), symbols, but there is no guarantee for my functions to start this way. It could be that only Archolos supports this naming scheme, so this wouldn't make things easier for me anymore 🤔

So the idea from #5 (comment) seems the most useful at the moment:

stack_trace = vm.call(func_sym, return_stack_trace_dict=True)
# or 
vm.call(func_sym)
vm.print_stack_of_previous_call_not_the_current_one()
# or
logger.setlevel(LogLevel.TRACE_DAEDALUS) # outputs Called internal function: name or Called external: name
vm.call(func_sym)  # use the logger messages to know what was called inside

When you said that ZenKit shouldn't initialize instances with get_instance, but instead the user should explicitly call vm.init_instance from client side, then I thought that perhaps we have a different perspective. I think that the library should do a lot of implicit stuff to make things easier for the end user and allow them to "explore" what happens, and you maybe consider that the user should know what he wants to achieve and "configure the proper state of the in-game values", like for making unit tests for Daedalus code 😅

So is there anything specific you need from me right now?

Basically somehow, doesn't matter how TBH, expose information from inside the vm.call execution, so that a user that doesn't know what happens during vm.call(specific_function), somehow gets to know this information. It doesn't have to be everything of course, but I think OP.CALL / OP.CALL_EXTERNAL is feasible? 🤔