Parser

[hildjj]

There's a great rust-based parser that can be turned into WASM at one-ini. The hope is that this could be one parser that most of the editconfig cores could use, so that there would be consistency.

There's also a parser I wrote in Peggy that does roughly the same thing, but handles comments in a few more places -- even though that's somewhat-discouraged by the spec. The extra comment handling could be removed, for a slight performance win.

I wrote a simple benchmark using Kelonio that parses each of the test input files, turns them into the expected results from parseString. I repeated this 10k times for each parser. Here are the results:

- - - - - - - - - - - - - - - - - Performance - - - - - - - - - - - - - - - - -
Peggy:
  0.64685 ms (+/- 0.00184 ms) from 10000 iterations
one-ini only, without remapping to expected types:
  1.09396 ms (+/- 0.00062 ms) from 10000 iterations
one-ini plus remap:
  1.10929 ms (+/- 0.00049 ms) from 10000 iterations
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

As you can see, one-ini is much more consistent (I assume because there's a lot less GC in that loop), but the Peggy parser is noticeably faster. I think some part of that is marshaling and parsing across the WASM boundary, and there is a small part that is the required remapping logic.

Sizes of all the files included in the distro:

• one-ini: 74kB
• peggy: 24kB

Both parsers parse all of the tests in the current test suite.

If other editorconfig cores start to use one-ini, the benefits of consistency might be worth a few more bytes, and a somewhat-slower parse.

Does anyone have an opinion?

[hildjj]

After spending a few days fighting to learn Rust, I've narrowed the perf difference slightly. The approach I took was to pass a Buffer/UInt8Array in to WASM rather than a string, to avoid a utf16->utf8 conversion getting to a Rust string. Then, instead of creating an AST in Rust (with a bunch of allocation, followed by a bunch of utf8->utf16 conversions for each string on the way out, causing more allocations), I create a Vec, which has TokenType, startByte, endByte triplets for each interesting parse token found. There's utf8->utf16 decodes that happen per field at that point. I also tried u16's instead of u32's, to see if that mattered. The benchmark results:

Peggy:
  0.64094 ms (+/- 0.00181 ms) from 10000 iterations
one-ini (original) only:
  1.48123 ms (+/- 0.00069 ms) from 10000 iterations
one-ini (original) plus remap:
  1.4904 ms (+/- 0.00056 ms) from 10000 iterations
one-ini Vec<u16>:
  1.16425 ms (+/- 0.00049 ms) from 10000 iterations
one-ini Vec<u16> plus remap:
  1.22825 ms (+/- 0.00082 ms) from 10000 iterations
one-ini Vec<u32>:
  1.1673 ms (+/- 0.00046 ms) from 10000 iterations
one-ini Vec<u32> plus remap:
  1.22973 ms (+/- 0.00091 ms) from 10000 iterations

Notes:

• The Vec versions are gamed slightly as I knew the max token length of all of the files I was processing, so I set the initial capacity of the Vec's to slightly higher than the that max length. That's probably a reasonable guess for average .editorconfig files though, so it doesn't feel like cheating too badly.
• I'm not sure why the original one-ini version is so much slower today than it was when I ran these numbers a few days ago. I don't think I changed any of that code path, but who knows what the Rust compiler did.
• The remap phase for the Vec versions is slower than that of the original one-ini code. That's the utf8 decode that happens in that phase, rather than in the WASM wrapper functions.
• If they ever fix Zero-copy pass ArrayBuffer from JS-land to WebAssembly-land WebAssembly/design#1162 we might save at least one copy into WASM memory.
• I should have gone about learning Rust systematically, instead of haphazardly, so if anyone wants to review my work to help squeeze some more perf out, you'll be able to do that in the one-ini PR that I'll file with my changes.

Conclusions:

• Vec is the best of the WASM solutions in my opinion, since the perf gain from 32->16 bits isn't enough to be worth the possibility someone writes a >64kB .editorconfig file.
• I think it's worth using the optimized one-ini version so that we can have the same front-end parser as everyone else one day.
• The perf is close enough that most people won't notice.

[hildjj]

See one-ini/core#21