Skip to content

optimize tokenizer #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
larshp merged 5 commits into from
Jul 19, 2026
Merged

optimize tokenizer #78

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
36bfa88
byte code
larshp Jul 18, 2026
45d63fa
Revert "byte code"
larshp Jul 18, 2026
ed013d5
update
larshp Jul 18, 2026
696e50a
cleanup
larshp Jul 19, 2026
d6b42a3
fix
larshp Jul 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 38 additions & 3 deletions src/zcl_mjs.clas.abap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,8 @@ CLASS zcl_mjs DEFINITION PUBLIC.
RAISING zcx_mjs_runtime.
CLASS-METHODS compile_function
IMPORTING ir_fn TYPE REF TO data.
CLASS-METHODS materialize_function
IMPORTING ir_fn TYPE REF TO data.
CLASS-METHODS collect_slots
IMPORTING ir_node TYPE REF TO data
CHANGING ct_map TYPE zif_mjs=>tt_slot_map
Expand All @@ -169,12 +171,13 @@ ENDCLASS.
CLASS zcl_mjs IMPLEMENTATION.

METHOD eval.
DATA lt_tokens TYPE zif_mjs=>tt_tokens.
lt_tokens = zcl_mjs_tokenizer=>tokenize( iv_source ).
DATA lr_tokens TYPE REF TO zif_mjs=>tt_tokens.
CREATE DATA lr_tokens.
lr_tokens->* = zcl_mjs_tokenizer=>tokenize( iv_source ).

DATA lo_parser TYPE REF TO zcl_mjs_parser.
CREATE OBJECT lo_parser
EXPORTING it_tokens = lt_tokens.
EXPORTING ir_tokens = lr_tokens.

DATA lt_stmts TYPE STANDARD TABLE OF REF TO data WITH DEFAULT KEY.
lt_stmts = lo_parser->parse_program( ).
Expand Down Expand Up @@ -242,6 +245,10 @@ CLASS zcl_mjs IMPLEMENTATION.

ASSIGN ir_fn->* TO <fn>.

IF <fn>-body_lazy = abap_true.
materialize_function( ir_fn ).
ENDIF.

" Lazy compile: assign variable slots on first call
IF <fn>-compiled = abap_false.
compile_function( ir_fn ).
Expand Down Expand Up @@ -353,6 +360,28 @@ CLASS zcl_mjs IMPLEMENTATION.
ENDIF.
ENDMETHOD.

METHOD materialize_function.
FIELD-SYMBOLS <fn> TYPE zif_mjs=>ty_function.
ASSIGN ir_fn->* TO <fn>.
IF sy-subrc <> 0 OR <fn>-body_lazy = abap_false.
RETURN.
ENDIF.

IF <fn>-body_tokens IS NOT BOUND.
<fn>-body_lazy = abap_false.
RETURN.
ENDIF.

DATA lo_parser TYPE REF TO zcl_mjs_parser.
CREATE OBJECT lo_parser
EXPORTING ir_tokens = <fn>-body_tokens.
lo_parser->pos = <fn>-body_pos.
<fn>-body = lo_parser->parse_block( ).
<fn>-body_lazy = abap_false.
CLEAR <fn>-body_pos.
CLEAR <fn>-body_tokens.
ENDMETHOD.

METHOD compile_function.
FIELD-SYMBOLS <fn> TYPE zif_mjs=>ty_function.
ASSIGN ir_fn->* TO <fn>.
Expand Down Expand Up @@ -1324,6 +1353,9 @@ CLASS zcl_mjs IMPLEMENTATION.
<fn_data>-params = <n>-params.
<fn_data>-default_params = <n>-default_params.
<fn_data>-body = <n>-body.
<fn_data>-body_lazy = <n>-body_lazy.
<fn_data>-body_pos = <n>-body_pos.
<fn_data>-body_tokens = <n>-body_tokens.
<fn_data>-closure = io_env.
DATA ls_fnval TYPE zif_mjs=>ty_value.
ls_fnval-type = zif_mjs=>c_type_function.
Expand Down Expand Up @@ -1666,6 +1698,9 @@ CLASS zcl_mjs IMPLEMENTATION.
<mfn>-name = ls_cm-name.
<mfn>-params = ls_cm-params.
<mfn>-body = ls_cm-body.
<mfn>-body_lazy = ls_cm-body_lazy.
<mfn>-body_pos = ls_cm-body_pos.
<mfn>-body_tokens = ls_cm-body_tokens.
<mfn>-closure = lo_cls_env.
DATA ls_mfnval TYPE zif_mjs=>ty_value.
ls_mfnval-type = zif_mjs=>c_type_function.
Expand Down
28 changes: 28 additions & 0 deletions src/zcl_mjs.clas.testclasses.abap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ CLASS ltcl_test DEFINITION FOR TESTING
METHODS test_replace_regex_anchor FOR TESTING RAISING zcx_mjs_runtime.
METHODS test_replace_empty_regex FOR TESTING RAISING zcx_mjs_runtime.
METHODS test_replace_empty_regex_g FOR TESTING RAISING zcx_mjs_runtime.
METHODS test_lazy_bodies FOR TESTING RAISING zcx_mjs_runtime.

METHODS test262 FOR TESTING RAISING zcx_mjs_runtime.

Expand Down Expand Up @@ -928,6 +929,33 @@ CLASS ltcl_test IMPLEMENTATION.
exp = |-a-b-c-| ).
ENDMETHOD.

METHOD test_lazy_bodies.
DATA(lv_nl) = cl_abap_char_utilities=>newline.
DATA(lv_js) =
`function unused() { let brace = "{"; console.log("BAD" + brace); }` && lv_nl &&
`function outer(x) { let brace = "}"; return (y) => { return x + y; }; }` && lv_nl &&
`class C { value() { return 40; } unused() { console.log("BAD"); } }` && lv_nl &&
`let add = outer(1);` && lv_nl &&
`let c = new C();` && lv_nl &&
`console.log(add(1));` && lv_nl &&
`console.log(add(2));` && lv_nl &&
`console.log(c.value());` && lv_nl &&
`console.log(c.value());`.

DATA(lt_tokens) = zcl_mjs_tokenizer=>tokenize( lv_js ).
DATA(lo_parser) = NEW zcl_mjs_parser( it_tokens = lt_tokens ).
DATA(lt_nodes) = lo_parser->parse_program( ).
READ TABLE lt_nodes INDEX 1 INTO DATA(lr_unused).
FIELD-SYMBOLS <unused> TYPE zif_mjs=>ty_node.
ASSIGN lr_unused->* TO <unused>.
cl_abap_unit_assert=>assert_equals( act = <unused>-body_lazy exp = abap_true ).
cl_abap_unit_assert=>assert_initial( <unused>-body ).

cl_abap_unit_assert=>assert_equals(
act = trim( zcl_mjs=>eval( lv_js ) )
exp = |2 3 40 40| ).
ENDMETHOD.

METHOD test262.
DATA(lv_nl) = cl_abap_char_utilities=>newline.
DATA(lv_js) =
Expand Down
76 changes: 66 additions & 10 deletions src/zcl_mjs_parser.clas.abap
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
CLASS zcl_mjs_parser DEFINITION PUBLIC.
PUBLIC SECTION.
DATA tokens TYPE zif_mjs=>tt_tokens.
DATA tokens TYPE REF TO zif_mjs=>tt_tokens.
DATA pos TYPE i.

METHODS constructor
IMPORTING it_tokens TYPE zif_mjs=>tt_tokens.
IMPORTING it_tokens TYPE zif_mjs=>tt_tokens OPTIONAL
ir_tokens TYPE REF TO zif_mjs=>tt_tokens OPTIONAL.
METHODS peek
IMPORTING iv_offset TYPE i DEFAULT 0
RETURNING VALUE(rs_tok) TYPE zif_mjs=>ty_token.
Expand Down Expand Up @@ -78,20 +79,27 @@ CLASS zcl_mjs_parser DEFINITION PUBLIC.
RETURNING VALUE(rr_node) TYPE REF TO data.
METHODS parse_member
RETURNING VALUE(rr_node) TYPE REF TO data.
METHODS defer_block
RETURNING VALUE(rv_pos) TYPE i.
ENDCLASS.

CLASS zcl_mjs_parser IMPLEMENTATION.
METHOD constructor.
tokens = it_tokens.
IF ir_tokens IS BOUND.
tokens = ir_tokens.
ELSE.
CREATE DATA tokens.
tokens->* = it_tokens.
ENDIF.
pos = 0.
ENDMETHOD.

METHOD peek.
IF pos + iv_offset >= lines( tokens ).
IF tokens IS NOT BOUND OR pos + iv_offset >= lines( tokens->* ).
rs_tok-kind = 5.
rs_tok-val = ``.
ELSE.
READ TABLE tokens INDEX pos + iv_offset + 1 INTO rs_tok.
READ TABLE tokens->* INDEX pos + iv_offset + 1 INTO rs_tok.
ENDIF.
ENDMETHOD.

Expand All @@ -105,6 +113,28 @@ CLASS zcl_mjs_parser IMPLEMENTATION.
next( ).
ENDMETHOD.

METHOD defer_block.
rv_pos = pos.
IF peek( )-val <> `{`.
RETURN.
ENDIF.

DATA lv_depth TYPE i.
WHILE peek( )-kind <> 5.
DATA(ls_token) = next( ).
" Only operator/punctuation tokens affect nesting. String and regex
" token values can themselves be "{" or "}".
IF ls_token-kind = 3 AND ls_token-val = `{`.
lv_depth = lv_depth + 1.
ELSEIF ls_token-kind = 3 AND ls_token-val = `}`.
lv_depth = lv_depth - 1.
IF lv_depth = 0.
RETURN.
ENDIF.
ENDIF.
ENDWHILE.
ENDMETHOD.

METHOD parse_program.
WHILE peek( )-kind <> 5.
DATA(lr_s) = parse_statement( ).
Expand Down Expand Up @@ -510,12 +540,15 @@ CLASS zcl_mjs_parser IMPLEMENTATION.
ENDIF.
ENDWHILE.
expect( `)` ).
DATA(lt_mbody) = parse_block( ).
DATA(lv_mbody_pos) = defer_block( ).
DATA ls_m TYPE zif_mjs=>ty_class_method.
CLEAR ls_m.
ls_m-name = lv_mname.
ls_m-params = lt_params.
ls_m-body = lt_mbody.
ls_m-is_ctor = boolc( lv_mname = `constructor` ).
ls_m-body_lazy = abap_true.
ls_m-body_pos = lv_mbody_pos.
ls_m-body_tokens = tokens.
APPEND ls_m TO lt_methods.
ENDWHILE.
expect( `}` ).
Expand Down Expand Up @@ -562,14 +595,25 @@ CLASS zcl_mjs_parser IMPLEMENTATION.
ENDIF.
ENDWHILE.
expect( `)` ).
DATA(lt_body) = parse_body( ).
DATA lt_body TYPE zif_mjs=>tt_nodes.
DATA lv_body_pos TYPE i.
DATA lv_body_lazy TYPE abap_bool.
IF peek( )-val = `{`.
lv_body_pos = defer_block( ).
lv_body_lazy = abap_true.
ELSE.
lt_body = parse_body( ).
ENDIF.
DATA lr_n TYPE REF TO zif_mjs=>ty_node.
CREATE DATA lr_n.
lr_n->kind = zif_mjs=>c_node_func_decl.
lr_n->str = lv_name.
lr_n->params = lt_params.
lr_n->default_params = lt_default_params.
lr_n->body = lt_body.
lr_n->body_lazy = lv_body_lazy.
lr_n->body_pos = lv_body_pos.
lr_n->body_tokens = tokens.
rr_node = lr_n.
ENDMETHOD.

Expand Down Expand Up @@ -662,8 +706,11 @@ CLASS zcl_mjs_parser IMPLEMENTATION.
IF <arrow_left>-kind = zif_mjs=>c_node_ident AND peek( )-val = `=>`.
next( ).
DATA lt_arrow_body TYPE zif_mjs=>tt_nodes.
DATA lv_arrow_pos TYPE i.
DATA lv_arrow_lazy TYPE abap_bool.
IF peek( )-val = `{`.
lt_arrow_body = parse_block( ).
lv_arrow_pos = defer_block( ).
lv_arrow_lazy = abap_true.
ELSE.
DATA(lr_arrow_expr) = parse_assign( ).
DATA lr_arrow_ret TYPE REF TO zif_mjs=>ty_node.
Expand All @@ -678,6 +725,9 @@ CLASS zcl_mjs_parser IMPLEMENTATION.
lr_arrow->str = ``.
lr_arrow->params = VALUE #( ( <arrow_left>-str ) ).
lr_arrow->body = lt_arrow_body.
lr_arrow->body_lazy = lv_arrow_lazy.
lr_arrow->body_pos = lv_arrow_pos.
lr_arrow->body_tokens = tokens.
rr_node = lr_arrow.
RETURN.
ENDIF.
Expand Down Expand Up @@ -1162,8 +1212,11 @@ CLASS zcl_mjs_parser IMPLEMENTATION.
IF peek( )-val = `=>`.
next( ).
DATA lt_ab TYPE zif_mjs=>tt_nodes.
DATA lv_ab_pos TYPE i.
DATA lv_ab_lazy TYPE abap_bool.
IF peek( )-val = `{`.
lt_ab = parse_block( ).
lv_ab_pos = defer_block( ).
lv_ab_lazy = abap_true.
ELSE.
DATA(lr_ae) = parse_assign( ).
DATA lr_ar TYPE REF TO zif_mjs=>ty_node.
Expand All @@ -1178,6 +1231,9 @@ CLASS zcl_mjs_parser IMPLEMENTATION.
lr_af->str = ``.
lr_af->params = lt_arrow_params.
lr_af->body = lt_ab.
lr_af->body_lazy = lv_ab_lazy.
lr_af->body_pos = lv_ab_pos.
lr_af->body_tokens = tokens.
rr_node = lr_af.
RETURN.
ENDIF.
Expand Down
Loading
Loading