Add tokenIndex to SymbolNode #2796

matthew-canestraro · 2022-09-28T17:02:38Z

This is a small POC to go with #2795

I kept this minimal, only adding tokenIndex to SymbolNodes specifically, but I could see the value in extending this to most/all nodes

Will mark as draft until the discussion resolves

josdejong

This is shaping up really nicely, and it looks well tested!

I made a few inline comments, can you have a look at those? Besides that, we should write a section in the documentation explaining source.

src/expression/parse.js

josdejong · 2023-03-17T15:29:02Z

test/unit-tests/expression/parse.test.js

+      // should have a source for brackets and item delimiters
+      const expected = [
+        { index: 0, text: '[' },
+        { index: 2, text: ',' },


If we parser a large matrix, I can imagine that listing all comma's impacts the performance. I think we should double check if this is an issue or not.

Can do! Are you worried about performance during parsing or evaluation or somewhere else?

Yes I was thinking about performance of parsing, there is a lot of extra objects created then. I think it cannot impact evaluation performance since the sources are not used during evaluation.

We can set up a small benchmark in the /test/benchmark folder to test if there is a performance impact or not, if you want I can do that.

If you're willing to do that, I'm glad for the help, thank you! Otherwise, I'll tackle it after finishing everything else

I'll do that 👍

I've run the benchmark test/benchmark/expression_parser.js and compared the develop branch with this PR (you can try it out yourself to see the differences).

The evaluate benchmark is just as fast in both cases, that is most important: about 1.7 million ops/sec

The parse benchmark is slower, goes from 16000 ops/second to 11000 ops/sec. I think that is acceptable, and simply a consequence of the parse step having to do more work.

Shall we leave it at that?

test/unit-tests/expression/parse.test.js

josdejong

Thanks for the updates @matthew-canestraro! I made a few comments, mostly about thinking through the exact API for meta regarding optional arguments/properties.

josdejong · 2023-05-02T08:05:00Z

docs/expressions/expression_trees.md

@@ -39,10 +39,11 @@ tree generated by `math.parse('sqrt(2 + x)')`.

 All nodes have the following methods:

-   `clone() : Node`
+-   `clone(options: MetaOptions) : Node`


The options are optional, can you describe that here and for all constructors in the docs?

josdejong · 2023-05-02T08:09:05Z

test/unit-tests/function/algebra/resolve.test.js

+
+function emptySourcesFromTree (tree) {
+  tree.traverse((node) => {
+    node.sources = []


How about using the immutable methods .clone({ sources: [] }) and .transform() in these helper functions to try out whether the immutable API is workable? (eat your own dog foo 😉 )

josdejong · 2023-05-02T08:19:43Z

src/expression/node/AccessorNode.js

     * @return {AccessorNode}
     */
-    clone () {
-      return new AccessorNode(this.object, this.index)
+    clone (meta = {}) {


The default value for meta, an empty object {}, does not correspond with the official TypeScript definition where sources is a require property.

I think it is best to have sources are required property. I would also like to get rid of mutating the input meta via meta.sources = ..., so I think we can rewrite the clone methods to something like:

/** * Create a clone of this node, a shallow copy * @param {MetaOptions} [meta] object with additional options for cloning this node * @return {AccessorNode} */ clone (meta) { const cloned = new AccessorNode(this.object, this.index, meta ?? { sources: this.sources }) return cloned }

Having to construct a default meta object via { sources: this.sources } is a bit odd. Would it make sense to just keep the meta object as it is instead of destructuring it in the Node constructor, and having to re-construct it in the methods clone before we can pass it to another constructor? I.e we could think about:

const defaultMetaOptions = { sources = [] } class Node { /** * @constructor Node * A generic node, the parent of other AST nodes * @param {MetaOptions} [meta] object with additional options for building this node */ constructor (meta = defaultMetaOptions) { this.meta = meta } // ... }

josdejong · 2023-05-02T08:22:10Z

test/unit-tests/expression/parse.test.js

+      // should have a source for brackets and item delimiters
+      const expected = [
+        { index: 0, text: '[' },
+        { index: 2, text: ',' },


I'll do that 👍

josdejong · 2023-05-02T08:34:16Z

src/expression/node/AccessorNode.js

@@ -47,9 +47,10 @@ export const createAccessorNode = /* #__PURE__ */ factory(name, dependencies, ({
     * @param {Node} object                 The object from which to retrieve
     *                                      a property or subset.
     * @param {IndexNode} index             IndexNode containing ranges
+     * @param {MetaOptions} object with additional options for building this node


Just for clarity: to make this valid JSDoc, the lines like:

@param {MetaOptions} object with additional options for building this node

Should be changed to:

@param {MetaOptions} meta object with additional options for building this node

And if meta is optional:

@param {MetaOptions} [meta] object with additional options for building this node

josdejong · 2023-09-13T15:32:27Z

@matthew-canestraro we never finished this PR. Are you still interested in getting it done?

matthew-canestraro · 2023-09-18T15:24:33Z

@matthew-canestraro we never finished this PR. Are you still interested in getting it done?

Hi @josdejong, sorry for dropping the ball on this, priorities shifted a lot over the past few months. That being said, I still believe this will be a good contribution to MathJS and it would be a shame to lose the work done. I'll do a final pass on the PR this week and ping you when I think it's ready

josdejong · 2023-09-20T06:45:28Z

Thanks @matthew-canestraro . Also if you can't find time for it just let me know, maybe someone else can finish your work then.

matthew-canestraro force-pushed the develop branch from 9c24329 to edd789f Compare February 7, 2023 21:47

Add source mappings for all nodes on parse()

742e662

matthew-canestraro force-pushed the develop branch from ad50348 to 742e662 Compare March 15, 2023 19:28

Merge branch 'develop' of github.com:josdejong/mathjs into develop

32793d7

matthew-canestraro marked this pull request as ready for review March 15, 2023 19:53

josdejong requested changes Mar 17, 2023

View reviewed changes

matthew-canestraro added 5 commits April 4, 2023 12:36

Merge branch 'develop' of github.com:josdejong/mathjs into develop

b722237

Use constructors instead of setSources()

173b20a

Add type for MetaOptions

e3706b8

Clean lint

fea1cbe

Update docs

c169f7d

josdejong reviewed May 2, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tokenIndex to SymbolNode #2796

Add tokenIndex to SymbolNode #2796

matthew-canestraro commented Sep 28, 2022

josdejong left a comment

josdejong Mar 17, 2023

matthew-canestraro Mar 23, 2023

josdejong Mar 27, 2023

matthew-canestraro Apr 4, 2023

josdejong May 2, 2023

josdejong May 2, 2023

josdejong left a comment

josdejong May 2, 2023

josdejong May 2, 2023

josdejong May 2, 2023

josdejong May 2, 2023

josdejong May 2, 2023

josdejong commented Sep 13, 2023

matthew-canestraro commented Sep 18, 2023

josdejong commented Sep 20, 2023

Add tokenIndex to SymbolNode #2796

Are you sure you want to change the base?

Add tokenIndex to SymbolNode #2796

Conversation

matthew-canestraro commented Sep 28, 2022

josdejong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josdejong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josdejong commented Sep 13, 2023

matthew-canestraro commented Sep 18, 2023

josdejong commented Sep 20, 2023