I think it's langium's bug

[negahama]

Hello! everyone
I am beginner of Langium.
I found something looks like a Langium's bug.

Let's get to the point

Here is my language grammar
This is base on langium-lox but I removed the unnecessary codes

grammar MyLanguage

entry Model: codes+=(FunctionDeclaration | Expression)*;

FunctionDeclaration infers Method:
  'def' name=ID '(' (params+=Variable (',' params+=Variable)*)? ')' (':' returnType=ID)?;
Variable: name=ID (':' type=ID)? ('=' value=Expression)?;

Expression: MemberCall | Literal;

MemberCall infers Expression:
  FeatureCall ({infer MemberCall.previous=current} 
    "." element=[Method:ID] (explicitCall?='(' Arguments? ')')?
  )*;

FeatureCall infers Expression: {infer MemberCall}
	element=[Method:ID] (explicitCall?='(' Arguments? ')')?;

Literal: value=NUMBER | value=ID;

fragment Arguments:
  args+=Expression (',' args+=Expression)*;

terminal ID: /[_a-zA-Z][\w_]*/;
terminal NUMBER: /[0-9]+(\.[0-9]+)?/;
hidden terminal WS: /\s+/;

Grammar above is for

def fn(param: number): number
fn()
fn(1)

As you see, FeatureCall and MemberCall is cross-reference
I append `my-language-scope.ts' file and this file looks below

export class MyLanguageScopeProvider extends DefaultScopeProvider {
  override getScope(context: ReferenceInfo): Scope {
    ...
    const prevTypeDesc = this.inferType(previous, new Map());
    ...
  }

  private inferType(node: AstNode | undefined, cache: Map<AstNode, TypeDescription>): TypeDescription {
    let type: TypeDescription | undefined;
    if (!node) {
      return createErrorType("Could not infer type for undefined", node);
    }

    const existing = cache.get(node);
    if (existing) {
      return existing;
    }

    // Prevent recursive inference errors
    cache.set(node, createErrorType("Recursive definition", node));

    if (isMemberCall(node)) {
      type = this.inferMemberCall(node, cache);
    }
    ...
  }

  private inferMemberCall(node: MemberCall, cache: Map<AstNode, TypeDescription>): TypeDescription {
    const element = node.element?.ref;
    if (element) {
      return this.inferType(element, cache);
    }
    ...
  }
}

Until here, There is nothing special or difficult..
Again, I removed a lot of codes, but my code is not changed a lot, it's very similar to langium-lox' codes

Example can be changed follow.. I know it cause the errors.. BUT error messages is very strange

def fn(param: number): number
fn(),c
fn(1).c

error message is

There are validation errors:
line 2: Could not resolve reference to Method named 'c'. [c]
line 3: An error occurred while resolving reference to 'c': Error: AST node has no document. [c]

Error of line 2 is normal result, but error of line 3 is not same as line 2, it is "AST node has no document"
I tested a lot... regardless of whether fn declaration has parameters, if argument exist in fn call, error become AST node has no document
And I found out it was caused by const element = node.element?.ref; in inferMemberCall() but I don't know why..

My point is Why "AST node has no document" cause and How can I fix it...

I append full source project for this.. you can get the same situation by runing npm run all
MyLanguage.zip

Thank you for listening my sufficient explanation and help me...
God bless you!

[msujew]

Hey @negahama,

Thanks for providing a reproducible project. I've been trying to hunt down this issue forever, but failed to find some reproducible setup. I'll take a look in the upcoming days :)

[negahama]

@msujew Thank you for being interested this issue...

Do you have some problems to run my sample?
I think you've already solved the setup problem because you're expert, If not, let me know

Anyway, I've fortunately solved this issue.

It's about fragment Arguments. I've changed my grammar from

MemberCall infers Expression:
  FeatureCall ({infer MemberCall.previous=current} 
    "." element=[Method:ID] (explicitCall?='(' Arguments? ')')?
  )*;

fragment Arguments:
  args+=Expression (',' args+=Expression)*;

to

MemberCall infers Expression:
  FeatureCall ({infer MemberCall.previous=current} 
    "." element=[Method:ID] (explicitCall?='(' (args+=Expression (',' args+=Expression)*)? ')')?
  )*;

as you see, Arguments was replaced it's context.
featureCall rule was also changed with the same way

As I did this, error messages was

  There are validation errors:
  line 2: Could not resolve reference to Method named 'c'. [c]
  line 3: Could not resolve reference to Method named 'c'. [c]

BUT I don't know why it occurred until now.
So I really want you to explain to me how could it be.

I thought I knew Langium's grammar a lot, but I lost a lot of confidence.
I hope your perfect explanation... :)
Thanks in advance...

[msujew]

Ok, so the grammar part seems to be working as expected now. As far as I can tell, you're returning a number from fn. Since numbers don't have any callable members (at least in the lox type system), the scope will just return empty. With that in mind, the linker will not be able to resolve c when called from fn(1).

Does that explain the behavior you're seeing? Or have I misunderstood your question?

[negahama]

I'm not English native speaker. although I did my best, I couldn't make you be understand what I meant. sorry for that.

Please I hope you to pay more attention to code rather than what I'm saying..

OK.. let's make it more simple

grammar MyLanguage

entry Model: codes+=(Method | MemberCall)*;

Method: 'def' name=ID;

MemberCall:
    FeatureCall ({infer MemberCall.previous=current} 
    "." element=[Method:ID] '(' Arguments? ')'
    )*;

FeatureCall: {infer MemberCall}
    element=[Method:ID] '(' Arguments? ')';

fragment Arguments:
    arg=Literal;

Literal: value=NUMBER | value=ID;

terminal ID: /[_a-zA-Z][\w_]*/;
terminal NUMBER: /[0-9]+(\.[0-9]+)?/;
hidden terminal WS: /\s+/;

my-language-scope.ts
I think this part is the most important.
Here are a lot of codes, but most of them is console log, real code is few and flow is very simple.

export class MyLanguageScopeProvider extends DefaultScopeProvider {
  constructor(services: LangiumServices) {
    super(services);
  }

  override getScope(context: ReferenceInfo): Scope {
    const scopeId = `${context.container.$type}.${context.property} = ${context.reference.$refText}`;
    console.log(`Enter getScope: ${scopeId}`);

    if (context.property !== "element") {
      console.log(`Exit1 getScope: ${scopeId}`);
      return super.getScope(context);
    }

    const memberCall = context.container as MemberCall;
    if (memberCall.arg == undefined) {
      console.log("    MemberCall.args is undefined");
    }

    const previous = memberCall.previous;
    if (!previous) {
      console.log("    MemberCall.previous is undefined");
      const scope = super.getScope(context);
      console.log(`Exit2 getScope: ${scopeId}`);
      return scope;
    }

    console.log(`    MemberCall.previous is ${previous.$type}, ${previous.element?.$refText}`);
    if (isMemberCall(previous)) {
      console.log("    BEFORE ref:");
      const element = previous.element?.ref;
      console.log("    AFTER ref:", element?.name);
    }

    console.log(`Exit3 getScope: ${scopeId}`);
    return super.getScope(context);
  }
}

Ok... let's test below code

def fn
fn().c()

Here is debug message

Enter getScope: MemberCall.element = c
    MemberCall.args is undefined
    MemberCall.previous is MemberCall, fn
    BEFORE ref:
Enter getScope: MemberCall.element = fn
    MemberCall.args is undefined
    MemberCall.previous is undefined
Exit2 getScope: MemberCall.element = fn
    AFTER ref: fn
Exit3 getScope: MemberCall.element = c

There are validation errors:
line 2: Could not resolve reference to Method named 'c'. [c]

It's normal process that was expected

BUT next test code is not normal

def fn
fn(anyid).c()

Here is debug message

Enter getScope: MemberCall.element = c
    MemberCall.args is undefined
    MemberCall.previous is MemberCall, fn
    BEFORE ref:
Enter getScope: MemberCall.element = fn
    MemberCall.args is undefined
    MemberCall.previous is undefined
Enter getScope: MemberCall.element = fn
    MemberCall.previous is undefined
Exit2 getScope: MemberCall.element = fn

There are validation errors:
line 2: An error occurred while resolving reference to 'c': Error: AST node has no document. [c]

Do you notice that there isn't AFTER ref In this message?
And MemberCall's arg property is undefined although test code is fn(anyid)
And getScope() for fn was called twice

How could this be possible?

But this phenomenon becomes all normal if you change the grammar like below.

FeatureCall: {infer MemberCall}
	element=[Method:ID] '(' (arg=Literal)? ')';

My explanation is clear?
Thanks

[msujew]

Yes, I understand exactly what you're saying. I've already implied in #1638 (comment) that this is a bug that I had trouble reproducing. Whether you write the args as a fragment or not shouldn't matter. I'll likely get to it next week.

[negahama]

Hi.. everyone.
It's been a long time since I visited here last.
@msujew I think you've changed your profile's picture... you look better than previous

Anyway... I want to discuss this issue again because my grammar is so complex, so I want to use the fragment grammar of Langium but it doesn't work as I expected

So I'll explain this problem one more time in other way

grammar MyLanguage

entry Model: codes+=(Method | MemberCall)*;

Method: 'def' name=ID;

MemberCall:
  Start ({infer MemberCall.previous=current} "." element=[Method:ID]
  (isArray?='[' ']')? )*;

Start:
  element=[Method:ID]
  (isArray?='[' ']')?;

terminal ID: /[_a-zA-Z][\w_]*/;
hidden terminal WS: /\s+/;

I want to process my code below by using above grammar

def a
def b
a[].b

It works successfully

Now that I change the grammar to below to reduce the complexity of grammar.

MemberCall:
  Start ({infer MemberCall.previous=current} "." element=[Method:ID]
  ArrayCallSignature? )*;

Start:
  element=[Method:ID]
  ArrayCallSignature?;

fragment ArrayCallSignature:
  (isArray?='[' ']');

As you can see, (isArray?='[' ']') has been changed fragment ArrayCallSignature. that's all

It works successfully too

Until this moment, everything work as we expected...

I append new ScopeProvider that is

export class MyLanguageScopeProvider extends DefaultScopeProvider {
  constructor(services: LangiumServices) {
    super(services)
  }

  override getScope(context: ReferenceInfo): Scope {
    const call = context.container as ast.MemberCall
    const previous = call.previous
    if (previous) {
      // console.log('ref:', previous.element.ref?.name, previous.element.ref?.$type)
    }
    return super.getScope(context)
  }
}

It works well until I active the console.log comment..

So My point is...

When I use element.ref in getScope(), ref work abnormally in case using fragment in grammar.

The Langium throws exception Error: AST node has no document when fragment and ref are used in the same time

This is absolutely not a irreproducible issue.
Rather, I'm just wondering how this problem can remain until now.

As usual, I will cut it short in anticipation of quick action.
Thank you.

[negahama]

In addition, a.b instead of a[].b works successfully...

In my humble opinion... It might be...
a[] is one, not a + [] when fragment is used in Start rule.
So Langium Parser couldn't recognize the a in the precomputedScope stage, so didn't set ref enough.

ps) I have to use previous.element.ref to determine type of previous element

[msujew]

I see. I was able to reproduce this. It seems to be a very specific set of circumstances: Action + Fragment + References. I'm currently looking at a fix for this.

[msujew]

See fix and test case in #1762.

[negahama]

Great...

Most of all, I am very glad that my problem has been accurately conveyed.
I saw comment in #1762. It's the same as what I've experienced.
I bet you know the exact cause, and although new code hasn't been merged yet, I believe this issue will be completely resolved soon.

It reminds me of a time when I had a hard time trying to solve this problem.

I hope you, your company, and Langium all the best.