Serializing modules

[girvel]

• ldump.mark
• ldump.mark_module
• ldump.mark_itself

[girvel]

Actually, it is often the case that modules have mutable data inside. Maybe somehow mark modules as containing mutable data inside to prevent the error?

[girvel]

Marking modules as containing mutable data probably shouldn't be done through an argument to ldump.require, but something that can be done only once.

[girvel]

Actually, the problem with modules with mutable data inside is more complex than I thought. Even if we mark some modules as containing mutable data, we don't prevent mutable data from becoming static -- we just prevent erroring. Then, some function may capture the current state of the mutable value and be serialized; then the value changes; then the function is deserialized, and now it has the wrong value, as the mutable data was wrongly marked as static.

My best idea is to go back to manually marking data as static. There is a silver lining -- the new logic would still have use. ldump.require can be redone as ldump.static; it would receive the value and the module's vararg parameter, containing the module path. Also, there could be ldump.static_module_at, that receives just the path to the module -- this could be called outside the module to handle external libraries. In partially mutable modules, ldump.static would be uncomfortable to use -- the user would need to provide the path over and over again. So, we introduce the third function -- ldump.get_static(...) -- it would just curry the ldump.static with the path. Maybe we can even unify get_static and static -- do a ldump.static(...)(value). Also, it should be possible to mark internally mutable values as static -- we would need non-recursive static mark.

This becomes too complicated for my tired brain, I am going to pick this up later.

[girvel]

So, the sketch of the new API:

ldump.const(path)(value)  -- mark as recursively static
ldump.static(path)(value)  -- mark as shallow static
ldump.const_module(...paths)  -- mark modules as recursively static
ldump.static_module(...paths)  -- mark modules as shallow static

The only issue is -- we can't use ldump.static anywhere except for the root of the module, as it does not receive the key path to itself. It is kind of important. Ideas?

[girvel]

So, const shouldn't be a double function, the same with static. Static may receive a schema, that indicates which of the fields are static.

ldump.const(value: T, path: string) -> T  -- (internal use) mark as recursively static
ldump.static(value: T, path: string, schema?: table) -> T  -- (internal use) mark as shallow static
ldump.const_module(...paths: string)  -- (external use) mark modules as recursively static
ldump.static_module(...paths: string | [string, table])  -- (external use) mark modules as shallow static

Schemas example:

ldump.static(module, select(1, ...), {
  _type_info = {
    name = true,
    fields = true,
    posix_path = true,
  },
  _statistics = true,
})

If the child is static, that should always mean that the parent is static too.

Immutable values with a static parent are not considered static by default, as they can be reassigned. Immutable values with immutable static parent should be considered static. Should the mutable values with immutable static parent be considered static? I think yes.

It would probably be helpful to mix static/const values in the tree. Like it is possible to have some_type module, which is static with mutable some_type._statistics table, that has some_type._type_info field, which is const. That means, that there shouldn't be separation between const-functions and static-functions in the API. Instead, there should be something like ldump.mark.

[girvel]

So, it would work like this:

ldump.mark(module, {
  _type_info = "const",
  _statistics = "static",
}, ...)

Or:

ldump.mark(module, "const", ...)

Or:

ldump.mark_module("lib.some_type", {
  type_info = "const",
  statistics = "static",
})

[girvel]

Actually, using const/static neologisms makes less sense here. We can mark statics with no static children with empty table, and for consts we can still say "const", this literal still seems good:

ldump.mark(module, {
  _type_info = "const",
  _statistics = {},
}, ...)

ldump.mark(module, "const", ...)

ldump.mark_module("lib.some_type", {
  type_info = "const",
  statistics = {},
})

[girvel]

2 issues:

1. There could be a fully constant library, such as tech.sound in fallen, with a bit of mutability inside (sound.cached and sound.multiple_cached). Of course, you can rewrite memoization to store values externally, but maybe ldump should not presume stuff about architecture? Two solutions: don't care, keep strict const, force to rewrite or redefine stuff as static; do an almost_const thing with defining non-static children instead of defining static ones.

2. Schemas are boilerplate. Maybe use them only for mark_module and revert to using fallen-style static/const keywords? Can we do that without overloading __newindex? Probably not, right?

[girvel]

1. Maybe later? Feels complicated.

2. The __newindex way is too complicated, it adds unnecessary __newindex logic and metatables, which would clash with user-defined metatables. Boilerplate is okay.

[girvel]

The resulting API:

ldump.mark(module: T, schema: table | "const", modname: string) -> T
ldump.mark_module(modname: string, schema: table | "const")

[girvel]

Maybe also an additional call to mark itself:

ldump.mark_itself()