Upcoming release (v 0.10) has a significant breaking change - in order to bring ... Pure Go JS script support 🎉

[stroiman]

In the near future, a new release will bring some major changes to the script engine, including pure Go support using grafana/sobek

Sobek support 🍾

The first version of Gost-DOM had an integrated V8. While I think this was quite the achievement, V8 does come with a huge overhead; being written in C++ it depends on CGo, and has a hefty compilation time. Also, while JS itself is garbage collected, the JS GC is not connected to the Go GC. Currently, Gost-DOM will leak memory in the scope of a web page.

I'm proud to announce that a pure Go alternative is now available sobek as an alternate JS engine. Sobek is a fork of goja adding ESM support.

Sobek still needs to be battle tested, but it will at least fix the GC problem of V8.

Note: V8 support will stay, as I can be virtually certain that V8 will continue to be maintained, and match the capabilities of the browser environment that Gost-DOM simulates.

Breaking changes

• A script engine will no longer be configured by default
• A script engine abstraction layer
• v8 moved to new package
• Most deprecated functions removed

A script engine will no longer be configured by default

• Previous behaviour: When creating a browser using browser.New was to configure a script engine if not explicitly configured (falling back to V8)
• Upcoming behaviour: When creating a browser using browser.New, an explicit script engine must be supported (see below)

To create a browser with a script engine, call:

• Creating a browser with V8: browser.New(browser.WithEngine(v8engine.DefaultEngine()))
• Creating a browser with V8: browser.New(browser.WithEngine(sobekengine.DefaultEngine()))

A temporary helper exists in v8browser.New - behaves almose as the original browser.New. Exception is that an explicitly configured ScriptEngine is ignored; it will always use v8engine.DefaultEngine()

Note

A script engine is not required when creating a browser. <script> tags will just not do anything.

ScriptEngine abstraction layer

A script engine is the top layer representing a script engine. The script engine is configured with the web APIs to be accessible. Script engines are supposed to be reused. as they can cache expensive resources. E.g., the V8 engine caches V8 isolates, avoiding configuring root scope for each tests. (v8 has cached isolates for a long time, but in global scope, not scoped to a runtime value)

The script engine creates "hosts" and "contexts" as needed. This should be seen as implementation details from the caller, and may be changed.

V8 moved to new package

As the new top level abstraction is a "script engine", not "script host", the new package name for v8 is scripting/v8engine, not scripting/v8host`.

Most deprecated functions are removed

Most code marked with // deprecated: has been removed. An exception is Clock.RunAll() as it's not completely clear what is a sensible interface.

The grand plan

The changes here support to points in the greater scheme

• Move script engines out of the browser repository
• Allow client code to add new Web APIs

Move script engines

Instead of having a script engine in, e.g. gost-dom/browser/scripting/v8host you would have gost-dom/v8engine as a separate module.

This reduces the module dependencies of client code. You only get module dependencies to v8go if you really want the v8 engine.

Allow client code to add new Web APIs

The ScriptEngine adds a layer of abstraction currently only accessible through internal APIs, but all Web API implementation is currently independent of the actual script engine.

It is the overall goal that Gost-DOM provides core browser functionality, e.g., the DOM and HTML DOM and Network functionality. But 3rd party libraries can provide support for other Web APIs, e.g. Geolocation, Passkey, Local storage (yes, no local storage yet).

• Have a simple version that resolves a static location
• Have a complex version replaying a pre-recorded GPX track, verifying geofencing behaviour implemented in the browser.

The API for configuring this is currently internal, but I believe it will soon be ready for experimental release.

Kudos

Whenever I mention V8, I have must give a shoutout to

• rogchap/v8go for creating the original v8go project
• tommie/v8go for keeping the project up-to-date
  • Adding essential features, e.g., Symbol support
  • Keeping it up-to-date with original sources from Chromium automatically in a pipeline

[stroiman]

Initial tests seems that runtime performance is slightly slower with Sobek than V8. One significant difference is that V8 supports "templates", so I don't need to create all objects in global scope for every single script context.

Sobek/Goja has "templates" internally, but the functionality is unexported. Having templates available would be able to reduce significant browser configuration overhead.

[nickchomey]

Did you ever check out https://github.com/fastschema/qjs, which I had suggested in the sobek issue? Now that you have things abstracted, it might not be too difficult to try out and see how the perf and functionality compares. It seems like it might be a nice middle ground between v8 and sobek, both features and perf-wise

[stroiman]

I didn't, but, looking at the documentation, it appears that some essential features aren't available in regards to what can be configured from the Go-side, e.g., installing iterators as an attribute with key Symbol.iterator, as well as creating what V8 calls named/indexed property handlers, and Goja/Sobek calls dynamic objects/dynamic arrays - the ability to have a callback into Go code whenever JS code tries to access a named or indexed attribute that isn't explicitly configured.

The iterator could have a clunky workaround, calling into JS, evaluating Symbol.iterator and use that to create an attribute. But without indexed handlers, it would be useless, and without named handlers, Datastar wouldn't work.

On top of that ... it's not a priority to support a new script engine. Sobek was reasonably easy, as the Goja engine was almost complete - and I'm glad to have a pure Go alternative. Some of the overhead of Sobek can be solved by forking, if maintainers aren't sympathetic to exposing the internal template mechanism.

Now I want to focus on improving support for the intended use case, improving the supported browser APIs

[nickchomey]

I have started trying to use gost-dom in an... unorthodox... way and am facing some perf issues.

I'm trying to make my server-side html templates with JS template literals, rather than using, say, Go's html/template package, templ, or something else. Its surely a novel idea, though I don't think it is crazy - its just a templating engine (could even be jinja, handlebars, or anything else), but hopefully comes with the benefit that the same isomorphic templates could be used sparingly within the browser (eg in a Service Worker or Shared Worker context, when offline).

I set up some basic templates and tried rendering with gost-dom's sobekengine, as well as with sobek directly, and notice that the latter is 10-20x faster. Some profiling suggests that the overhead is largely related to the polyfill and other dom api instantiation every time you use browser.Open. I suppose this affects any usage of gost-dom, not just my weird approach.

Perhaps it would be best to just use Sobek directly, but I dont think it would have support for the event loop and other things that gost-dom has added on top of it.

I would then use full gost-dom for doing actual testing of what happens when the ssr html + associated js gets executed in the browser. Tests would be run far less frequently, and wouldnt be quite as time-sensitive as rendering responses for actual requests

You mentioned above

Initial tests seems that runtime performance is slightly slower with Sobek than V8. One significant difference is that V8 supports "templates", so I don't need to create all objects in global scope for every single script context.

Sobek/Goja has "templates" internally, but the functionality is unexported. Having templates available would be able to reduce significant browser configuration overhead.

I tried futzing with gost-dom and sobek for a bit to try to get this to work, but don't have a sufficient understanding of either to make it happen.

Is this something that you might be able to work on implementing, such that those polyfills etc are only instantiated once and then we can reuse them all (or even the entire browser instance) across various renders?

Alternatively (or perhaps in addition) to that, I wonder if there could be a way to specify which polyfills, dom apis etc you want to use - such that in this sort of SSR context, i'd probably use none of them as I just need JS features. This all seems aligned with what you said above

A script engine is the top layer representing a script engine. The script engine is configured with the web APIs to be accessible. Script engines are supposed to be reused. as they can cache expensive resources. E.g., the V8 engine caches V8 isolates, avoiding configuring root scope for each tests. (v8 has cached isolates for a long time, but in global scope, not scoped to a runtime value)

The script engine creates "hosts" and "contexts" as needed. This should be seen as implementation details from the caller, and may be changed.

And somewhat the reverse of the Allow client code to add new Web APIs plans (be able to remove Web APIs as well)

Thanks!

[stroiman]

In my own use case, it's also a matter of starting many short-lived browsers, so optimizing the initialization of global scope is definitely a goal. But so far, I've been prioritizing discovering architectural shortcomings. That's also the reason for my liberal use of polyfills - to more quickly get to a state where I could actually use it on real apps.

I have since the release looked into the internals of goja/sobek - and internally, it does use a "template" system similar to V8 for building its own native objects only once; things like Object, Array, Proxy, etc.. Unfortunately, it's not part of the public API.

It could definitely be a route to fork it for the purpose of tapping into that capability. But ... you mentioned the primary issue, time! ;)

At any rate, seeing how it's used in the wild will be tremendously helpful in discovering shortcomings.

[nickchomey]

Ok, i sort of carved out an example repo that you can try running. https://github.com/nickchomey/gost-ssr-simple

i use mise for tools, running tasks etc... But you can just run go test -v . to see the output, something like this:

=== RUN   TestSobek
=== RUN   TestSobek/simple
    common_test.go:233: simple       cold:   1.71ms 
=== RUN   TestSobek/list
    common_test.go:233: list         cold:   1.64ms 
=== RUN   TestSobek/conditional
    common_test.go:233: conditional  cold:   1.75ms 
=== RUN   TestSobek/nested
    common_test.go:233: nested       cold:   1.80ms 
=== NAME  TestSobek
    sobek_test.go:154: 
        Sobek TOTALS: cold: 6.91ms
--- PASS: TestSobek (0.01s)
    --- PASS: TestSobek/simple (0.00s)
    --- PASS: TestSobek/list (0.00s)
    --- PASS: TestSobek/conditional (0.00s)
    --- PASS: TestSobek/nested (0.00s)
=== RUN   TestBrowser
=== RUN   TestBrowser/simple
    common_test.go:233: simple       cold:  11.82ms   [setup:207.4µs browser:4.4µs open:11.59ms exec:12.3µs read:2.9µs]
=== RUN   TestBrowser/list
    common_test.go:233: list         cold:   8.78ms   [setup:131.7µs browser:2.4µs open:8.64ms exec:8.9µs read:2.4µs]
=== RUN   TestBrowser/conditional
    common_test.go:233: conditional  cold:   8.94ms   [setup:199.0µs browser:2.3µs open:8.73ms exec:8.5µs read:2.4µs]
=== RUN   TestBrowser/nested
    common_test.go:233: nested       cold:   9.68ms   [read:2.4µs setup:149.0µs browser:2.9µs open:9.52ms exec:8.6µs]
=== NAME  TestBrowser
    template_test.go:165: 
        Browser TOTALS: cold: 39.22ms
--- PASS: TestBrowser (0.04s)
    --- PASS: TestBrowser/simple (0.01s)
    --- PASS: TestBrowser/list (0.01s)
    --- PASS: TestBrowser/conditional (0.01s)
    --- PASS: TestBrowser/nested (0.01s)
PASS
ok      gost-ssr-simple (cached)

I commented-out some of the templates in common_test.go, because they use some fancier templating mechanisms that I made (Include, Extend, Super), because they rely on dynamic ESM imports. I had to change script_context.go->DownloadModule() to the following in a fork of gost-dom/browser in order to get that working (I'll submit a PR if you like)

func (c *scriptContext) DownloadModule(script string) (result html.Script, err error) {
	resolver := moduleResolver{
		c.host,
		c,
		make(map[sobek.ModuleRecord]string),
		make(map[string]sobek.ModuleRecord),
	}

	// Set up dynamic import support
	// This callback is invoked when JavaScript code uses dynamic import(), e.g., await import('./module.js')
	c.vm.SetImportModuleDynamically(func(referencingScriptOrModule interface{}, specifierValue sobek.Value, promiseCapability interface{}) {
		specifier := specifierValue.String()
		c.logger().Debug("Dynamic import requested", "specifier", specifier, "ref", referencingScriptOrModule)

		// Resolve the module synchronously (we're already in the event loop context)
		// In a more sophisticated implementation, this could be done asynchronously
		m, resolveErr := resolver.resolveModule(referencingScriptOrModule, specifier)
		if resolveErr == nil {
			resolveErr = m.Link()
		}

		// Complete the dynamic import by calling FinishLoadingImportModule
		// This resolves or rejects the Promise returned by import()
		c.vm.FinishLoadingImportModule(referencingScriptOrModule, specifierValue, promiseCapability, m, resolveErr)
	})

	rec, err := resolver.resolveModule(c.window.LocationHREF(), script)

	if err == nil {
		err = rec.Link()
	}
	if err == nil {
		result = sobekModule{c, rec}
	}
	return
}

The same is done in sobek_test.go, so you could un-comment all the tests in common_test.go and then run go test -v -run TestSobek . if you want to see those working.

The main point is that they're all running 10x slower, and its pretty much all due to browser.Open (and, eventually, the polyfills and other instantiation overhead). You can see some timing in the output above (repeated here: [setup:207.4µs browser:4.4µs open:11.59ms exec:12.3µs read:2.9µs])

Let me know if this works/is helpful!

[nickchomey]

I also wonder if there's anything at all useful in this comment from the sobek dev? Or did you already sort that stuff out

[stroiman]

@nickchomey I didn't get to look at your repo yet - I will.

But I just tried to compile the polyfills - as I said, it was a very low-hanging fruit. Unfortunately, it has no real impact on performance

❯ go test -count=1 ./scripting/sobekengine
ok      github.com/gost-dom/browser/scripting/sobekengine       0.486s

The time reported varies in the tens of milliseconds - saving 900+ compilation calls. So it's really all the initialization in Go code that takes up time.

However, you asked for the ability to not install all the APIs. I have been significantly refactoring the script engine initialization, although for a different reason. But the outcome is that the ability to create an empty script engine probably isn't far off. (But adding some API's isn't supported, as I still keep that in internal scope).

The reason for this is to support Workers - from the same prioritization as I've mentioned, addressing risk. So decoupling the script engine from a Window was pretty important.

---

Regarding the "event loop", resolving a module returns a promise. To the best of my knowledge, the promise will be resolved immediately, if the module doesn't contain an await in root scope and none of the dependent modules can resolve immediately.

import something from 'some-module'
const foo = await new Promise(r => { r(42) })
export default foo

I don't deal with async modules yet. I added ESM support to support Datastar, which appears to not have that (fortunately). So my code for evaluating ESM is as simple as this. (I see the error handling is also quite lacking)

	// Here m.record is a sobek.ModuleRecord
	p := m.record.Evaluate(m.ctx.vm)

	if p.State() != sobek.PromiseStateFulfilled {
		return p.Result().Export().(error)
	}

Note: AFAICT, Goja/sobek doesn't expose a next to allow Go code to install a callback to be called when a promise settles. I might PR that when it becomes a problem (i.e., I need to handle async code in ESM root scope)

[stroiman]

Ah, minor correction, the precompiled script does help quite a bit, it cuts the sobek suite execution time by about 20% - the sobek test suite now runs in ~390ms - which previously ran in ~470ms.

The relative speed increase will probably be less for real projects, as the initialization will contribute to a smaller part of the total execution time.

The wrong measurement was due to a stupid mistake, passing the "engine" owning the cache by value, not by reference.

a195163