Repository navigation
Conversation
On exit(), uc_vm_execute_chunk() released the call frames of all active invocations, also those of the C functions that called back into ucode code and still run. A C function that pushed values before it called uc_vm_call(), such as include() or call() with a scope, then popped from an empty stack and wrote below it, which crashed the VM later. Release the call frames only down to the depth of the invocation, as an uncaught error already does. Each caller then sees the exit exception and unwinds its own frames. uc_require_ucode() no longer needs to skip its stack cleanup after an exit. Signed-off-by: John Crispin <john@phrozen.org>
uc_vm_call() decided by the type of the called value whether the interpreter runs after the call. For an object whose __call__ metamethod is a C function, the C function already ran in uc_vm_call_function(), and the interpreter then continued the bytecode frame on top of the stack, which is the frame of the caller, and corrupted the memory. A signal handler of this kind made the interpreter abort. The interpreter now runs only if the call left a new frame behind, which only a closure does. Signed-off-by: John Crispin <john@phrozen.org>
uc_vm_call_function() moved the arguments of a call with spread arguments into a temporary array that the garbage collector did not see. A tostring method that called gc() while the VM formatted the error for a spread value that is not an array freed the array. The error path also leaked a reference to the spread value. The arguments now stay on the stack while the VM pushes their expansion above them, and the expanded arguments then replace them. Signed-off-by: John Crispin <john@phrozen.org>
Move the parsing of the global definition, search path and library preload options from main.c into libucode, so that other programs than the ucode CLI can accept them with the same behaviour. No behaviour change. Signed-off-by: John Crispin <john@phrozen.org>
Objects were json-c hash tables: 40 bytes per entry, 16 entries allocated up front, a copy of each key, calls through function pointers, and keys that ended at a NUL byte, so that "a\0b" and "a" were the same key. - Objects keep their entries in insertion order, each a key string value and a value, so that short keys live in the pointer and keys from constants are shared. Objects with more than eight keys add a hash index; the hash is SipHash-1-3 with a seed per process. Reads and writes of own keys skip the generic key path. - The string buffer is uc_stringbuf_t in ucode/util.h. - json.c parses JSON into ucode values incrementally, with the grammar, errors and stopping positions of the json-c 0.18 tokener. 90000 generated inputs, whole and in chunks, give the same results as before, except that keys keep their NUL bytes. Its API is in ucode/json.h. - uniq() and the debug module collect values in uc_set_t. - lc() and uc() convert a string argument without a string buffer. ucv_from_json() and ucv_to_json() are removed, ucv_object_foreach() iterates with ucv_object_next(), and ucode/types.h no longer includes json-c. Bindings must be rebuilt, and those that use json-c must include it themselves. 100k objects, peak RSS: empty 78.4 -> 11.0 MB, three keys 45.2 -> 17.9 MB. Parsing 8 MB of JSON: 1.41 -> 0.40 s, peak RSS 216 -> 53 MB. Signed-off-by: John Crispin <john@phrozen.org>
…ount uc_vector_extend() computed the capacity from the count with a loop at each push. After a pop the derived capacity was smaller than the allocation, so each push across a capacity boundary reallocated and cleared the spare entries again; the call frames of a recursion that goes up and down at such a depth did this at every call. Vectors now have a capacity field. Entries set to NULL by code outside the macros are allocated anew, and a capacity of 0 next to allocated entries causes a reallocation, so code that sets the entries itself keeps working. The hand written vector structs get the field as well. This changes the layout of uc_vm_t, uc_array_t and other public structs. Cycles: fib -3 %, strings -5 %. 100k empty and 100k three element arrays: peak RSS 21208 -> 21428 kB. Signed-off-by: John Crispin <john@phrozen.org>
A runtime for native programs must share the VM state handling with the interpreter without containing the interpreter, the compiler or the bytecode format. Move the code that reads no bytecode from vm.c to vmcore.c: VM setup and teardown, globals, signals, the stack, registry, exceptions and stack traces, value, iterator, property and container operations, global variable access, closure and module operations, and uc_vm_call(). The parts that depend on a bytecode interpreter or a compiler are reached through hooks that the respective code registers when it is linked: - uc_vm_call() enters bytecode closures through an interpreter hook, which vm.c installs from a constructor, and native closures through a hook of the native runtime. - The loaders compile source through a function that the compiler registers; without it, loading source fails as with COMPILE_SUPPORT off. A runtime can register scripts that it contains, which the loaders take before the file system, and their compile errors. - A runtime can register a function that marks the values it holds outside the VM during a collection. The bytecode reader and writer move to bytecode.c, so that a runtime can leave them out. A static program that uses libucode.a must reference a function of vm.o, for example uc_vm_execute(), to run bytecode closures through uc_vm_call(). Signed-off-by: John Crispin <john@phrozen.org>
With the LLVM_SUPPORT build option, `ucode -c,llvm` writes a program and
the scripts it loads as LLVM IR. The build also produces the runtime as
LLVM bitcode, libucode-native.bc, and the list of its API symbols,
libucode-native.api:
ucode -c,llvm -o prog.ll prog.uc
llvm-link prog.ll libucode-native.bc -o all.bc
opt -passes='internalize,default<Os>' \
-internalize-public-api-file=libucode-native.api all.bc -o prog.bc
llc -relocation-model=pic -filetype=obj prog.bc -o prog.o
cc prog.o -lm -ldl -rdynamic -Wl,-rpath,/usr/lib/ucode/native -o prog
Every function becomes a native function. A depth analysis of its
bytecode gives each value of the operand stack a slot of the native
frame, and each instruction becomes a call of a runtime helper, which
runs the vmcore.c code of the instruction; frequent sequences of
instructions run in one helper call. Branches and exception handlers
become native control flow. The output holds no bytecode, only tables of
the constants, functions, debug information and sources, which contain
no pointers.
The runtime pushes a call frame for every native call, so stack traces,
the `this` of callbacks and the recursion limit work as in the
interpreter, and runs tail calls in place. Native closures call each other
without the VM stack. Captured locals move into closed upvalues, and the
native frames are GC roots. Constants are created once per program, and
string constants keep their hash as object keys, so that a property read
with a constant key does not hash it again. A statically or dynamically
imported module has one namespace object.
Signal handlers run after calls and at backward jumps, where the
interpreter runs them after every instruction.
Scripts that the program loads by literal names, and those named with the
embed= and embed-template= flags, are compiled into it; bundle_files(),
a global of native programs only, returns their paths. A program built
without the `debug` flag has no source and no positions; its stack traces
name function ids and offsets, which `ucode-trace` resolves with the file
that the map= flag writes.
A library with the SONAME of libucode but without code, installed in
lib/ucode/native, lets bindings that are linked against libucode load
without the interpreter library. The program finds it through its
RUNPATH, and warns if it does not. A program cannot load it by an
absolute path, as musl reuses a loaded library only if a search by name
found it.
The runtime is for Linux. The custom tests run against native programs
in the custom-native ctest, under valgrind, and the cram tests cover the
command line of native programs. native-ucode.sh builds the programs,
also for another target, with the library of the build tree.
Signed-off-by: John Crispin <john@phrozen.org>
- Global variables are looked up with the cached hash of their constant name, in the interpreter as well, and a new global shares the name as its key. Native helpers read constants without a reference of their own. - A call of a native closure moves the arguments from the slots of the caller into those of the callee, and calls through the VM stack move them there, instead of copying references. - Two loads of locals, upvalues or constants run in one helper call, as do a load of a local and the call after it, and a copy, the conditional jump after it and the pop on the path that does not jump. A return releases the frame in its own helper call, and only the slots below the returned value. - Functions in which no closure or export can capture a slot share one constant array of empty cells instead of a zeroed array per call. - The runtime bitcode keeps no target CPU and feature attributes, which kept LLVM from inlining any helper into the generated code. Helpers whose inlining costs more space than it saves time stay out of line. - NATIVE_TARGET_FLAGS sets the CPU and the float ABI of the runtime in clang syntax, for example "-march=mips32r2 -msoft-float" for OpenWrt mips_24kc. Without target attributes, the program bitcode must then be compiled for the same CPU and float ABI, for example by clang with these flags; llc without matching options gives code for the default ABI of the target without a warning. native-ucode.sh uses clang when NATIVE_TARGET_FLAGS is set. Cycles at -O2, measured per change: global lookups props -9 %; argument moves fib -9 %; inlining loop -10 %, props -7 %, fib -5 %; shared cells fib -14 %, closures -4 %. MIPS32 big endian under QEMU: native suite 249/250, the failure is a musl regex case. Signed-off-by: John Crispin <john@phrozen.org>
A native program could not load a script that it does not contain, such as a plugin that it finds by a computed name, or code that loadstring() compiles. Such a script is now compiled by the ucode CLI, which the program runs as a child process with the settings of the load, and the program loads the bytecode that it writes. The runtime contains the interpreter and the bytecode loader for it, which add about 48 kB of code. The CLI is `ucode` from PATH, or the one that --ucode-compiler names. Native and interpreted code share the VM and call each other: a native call of a bytecode closure enters the interpreter, and the interpreter runs a native closure through the native runtime instead of pushing a bytecode frame for it. The custom tests of loadstring() and loadfile() now pass with native programs; only the cases that run the program as the ucode CLI or expect the instruction trace of the interpreter remain skipped. Signed-off-by: John Crispin <john@phrozen.org>
The 180 cases in tests/custom/19_native cover values, iteration, spread arguments, regular expressions, require(), include(), render(), stack traces and the command line. The interpreter output is the expectation, so the custom test runs them in the interpreter and the custom-native test runs them as native programs. The interpreter runs no source with the dynlink flag, so the 15 cases of deferred imports are in native_build.t. They compare the output of a native program with that of the interpreter, which runs bytecode compiled with the same flag, and run the native program without its source files. Signed-off-by: John Crispin <john@phrozen.org>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work in progress. Do not merge yet. The series is open for an early
review of the design and of the VM changes.
The series has three parts. The fixes and the enhancements do not
depend on the native compiler, and can be merged on their own.
Fixes:
Each fix adds a test in tests/custom/99_bugs that fails without it.
Enhancements:
parser. ucode no longer depends on json-c. Bindings must be rebuilt,
and bindings that use json-c must include it themselves.
count
Native compiler (LLVM_SUPPORT build option):
interpreter
ucode -c,llvmwrites a program and the scripts it loads as LLVM IR.The build links the IR with the runtime bitcode (libucode-native.bc)
into an executable that contains no bytecode and no interpreter. If a
program loads a script that it does not contain, the interpreter runs
that script. Bindings that are linked against libucode load a stub
library from lib/ucode/native, which the program finds through its
RUNPATH.
@nbd168 @aparcar @cmonroe @dangowrt @jow-