Get Language Server to run with Eclipse (and other editors)

[borisbrodski]

I'm migrating my Xtext-projects to Langium.

I have created the "hello world" Langium project, build it without any change and then added it to Eclipse 2024-6-R (or 2024-3-R) using LSP4E (starting "node" with "{path}/out/language/main.js". I get following:

• No syntax highlighting
• Diagnostics are coming ok at first, but only the saved file get parsed. Here, how it works starting from file with no errors:
  • Change file adding a parse error (no diagnostics are shown, because file is not saved yet)
  • Save file to the disk (no diagnostics are shown, because LSP-message get only sent on file change)
  • Change file further (diagnostics for the saves state of the file are showing up)
• Show References is working
• Jump to declaration is working
• Rename locally is working

If I start Language Server for *.langium in Eclipse in the same way, the same problems are there.

I also tested Language Servers for *.langium and *.hello in NeoVim. It works almost exactly the same, as in Eclipse. A difference is only, that NeoVim sends LSP message on save making it much more usaful, since I don't need to change a file in order to see diagnostics.

Also NeoVim gets langium files completely broken while saving actual changes to the grammar (probably by formatter, not sure, if related).

Here is the initialization phase:

[2024-08-06T18:39:00.188465400+02:00] LSP4E_TO_LANGUAGE_SERVER LSP hello world:
Content-Length: 2136


[2024-08-06T18:39:00.232479200+02:00] LSP4E_TO_LANGUAGE_SERVER LSP hello world:
{"jsonrpc":"2.0","id":"1","method":"initialize","params":{"processId":6468,"rootPath":"/D:/Coding/ws/workspace-2024-06-R/Test/","rootUri":"file:///D:/Coding/ws/workspace-2024-06-R/Test/","capabilities":{"workspace":{"applyEdit":true,"workspaceEdit":{"documentChanges":true,"resourceOperations":["create","delete","rename"],"failureHandling":"undo","changeAnnotationSupport":{"groupsOnLabel":true}},"symbol":{"dynamicRegistration":true},"executeCommand":{"dynamicRegistration":true},"workspaceFolders":true,"configuration":true,"codeLens":{"refreshSupport":true}},"textDocument":{"synchronization":{"willSave":true,"willSaveWaitUntil":true,"didSave":true},"completion":{"completionItem":{"snippetSupport":true,"documentationFormat":["markdown","plaintext"],"resolveSupport":{"properties":["documentation","detail","additionalTextEdits"]},"insertTextModeSupport":{"valueSet":[1,2]}},"contextSupport":true,"completionList":{"itemDefaults":["commitCharacters","editRange","insertTextFormat","insertTextMode"]}},"hover":{"contentFormat":["markdown","plaintext"]},"signatureHelp":{},"references":{},"documentHighlight":{},"documentSymbol":{"symbolKind":{"valueSet":[18,17,5,14,9,10,22,24,8,1,12,11,20,6,2,3,21,16,19,25,4,7,15,23,26,13]},"hierarchicalDocumentSymbolSupport":true},"formatting":{"dynamicRegistration":true},"rangeFormatting":{},"definition":{"linkSupport":true},"typeDefinition":{"linkSupport":true},"codeAction":{"codeActionLiteralSupport":{"codeActionKind":{"valueSet":["quickfix","refactor","refactor.extract","refactor.inline","refactor.rewrite","source","source.organizeImports"]}},"dataSupport":true,"resolveSupport":{"properties":["edit"]},"dynamicRegistration":true},"codeLens":{},"documentLink":{},"colorProvider":{},"rename":{"prepareSupport":true},"publishDiagnostics":{},"foldingRange":{},"selectionRange":{},"inlayHint":{}},"window":{"workDoneProgress":true,"showMessage":{},"showDocument":{"support":true}}},"clientInfo":{"name":"Eclipse IDE","version":"0.18.10.202405280703"},"trace":"off","workspaceFolders":[{"uri":"file:///D:/Coding/ws/workspace-2024-06-R/Test/","name":"Test"}]}}
[2024-08-06T18:39:01.207765200+02:00] LANGUAGE_SERVER_TO_LSP4E LSP hello world:
{"jsonrpc":"2.0","id":"1","result":{"capabilities":{"workspace":{"workspaceFolders":{"supported":true}},"textDocumentSync":{"change":2,"openClose":true,"save":false,"willSave":false,"willSaveWaitUntil":false},"completionProvider":{},"referencesProvider":true,"documentSymbolProvider":true,"definitionProvider":true,"typeDefinitionProvider":false,"documentHighlightProvider":true,"codeActionProvider":false,"documentFormattingProvider":false,"documentRangeFormattingProvider":false,"foldingRangeProvider":true,"hoverProvider":true,"renameProvider":{"prepareProvider":true},"implementationProvider":false,"declarationProvider":false,"workspaceSymbolProvider":{"resolveProvider":false}}}}
[2024-08-06T18:39:01.224768200+02:00] LSP4E_TO_LANGUAGE_SERVER LSP hello world:
Content-Length: 52


[2024-08-06T18:39:01.226772600+02:00] LSP4E_TO_LANGUAGE_SERVER LSP hello world:
{"jsonrpc":"2.0","method":"initialized","params":{}}
[2024-08-06T18:39:01.234776+02:00] LSP4E_TO_LANGUAGE_SERVER LSP hello world:
Content-Length: 330


[2024-08-06T18:39:01.235775600+02:00] LSP4E_TO_LANGUAGE_SERVER LSP hello world:
{"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"file:///D:/Coding/ws/workspace-2024-06-R/Test/src/my.hello","languageId":"hello","version":1,"text":"person Aaa\r\nperson abc\r\n\r\n\r\nHello abc!\r\nHello Aaa!\r\nHello abc!\r\n\r\nperson Xxxx\r\nperson yyyy\r\n\r\nHello abc\r\n\r\n"}}}
[2024-08-06T18:39:01.236773+02:00] LANGUAGE_SERVER_TO_LSP4E LSP hello world:
{"jsonrpc":"2.0","id":0,"method":"client/registerCapability","params":{"registrations":[{"id":"2f7c6427-51b7-419a-ab1a-631f548f8d2c","method":"workspace/didChangeConfiguration","registerOptions":{"section":["hello-world"]}}]}}
[2024-08-06T18:39:01.238775+02:00] LSP4E_TO_LANGUAGE_SERVER LSP hello world:
Content-Length: 168


[2024-08-06T18:39:01.239774200+02:00] LSP4E_TO_LANGUAGE_SERVER LSP hello world:
{"jsonrpc":"2.0","id":"2","method":"textDocument/foldingRange","params":{"textDocument":{"uri":"file:///D:/Coding/ws/workspace-2024-06-R/Test/src/my.hello"}}}
[2024-08-06T18:39:01.241772900+02:00] LSP4E_TO_LANGUAGE_SERVER LSP hello world:
Content-Length: 38

Highlighting LSP messages:

[2024-08-06T18:58:39.957849900+02:00] LSP4E_TO_LANGUAGE_SERVER LSP hello world:
{"jsonrpc":"2.0","id":"48","method":"textDocument/documentHighlight","params":{"textDocument":{"uri":"file:///D:/Coding/ws/workspace-2024-06-R/Test/src/my.hello"},"position":{"line":0,"character":13}}}
[2024-08-06T18:58:39.962858400+02:00] LANGUAGE_SERVER_TO_LSP4E LSP hello world:
{"jsonrpc":"2.0","id":"48","result":[{"range":{"start":{"character":7,"line":0},"end":{"character":10,"line":0}}},{"range":{"start":{"character":6,"line":5},"end":{"character":9,"line":5}}}]}
[2024-08-06T18:58:40.366047700+02:00] LSP4E_TO_LANGUAGE_SERVER LSP hello world:
Content-Length: 171

Doesn't get shown neither in Eclipse nor in NeoVim.

"textDocument/didChange" messages:

[2024-08-06T18:58:39.059482900+02:00] LSP4E_TO_LANGUAGE_SERVER LSP hello world:
{"jsonrpc":"2.0","method":"textDocument/didChange","params":{"textDocument":{"version":5,"uri":"file:///D:/Coding/ws/workspace-2024-06-R/Test/src/my.hello"},"contentChanges":[{"range":{"start":{"line":0,"character":10},"end":{"line":0,"character":10}},"rangeLength":0,"text":" "}]}}
[2024-08-06T18:58:39.067492900+02:00] LANGUAGE_SERVER_TO_LSP4E LSP hello world:
{"jsonrpc":"2.0","method":"textDocument/publishDiagnostics","params":{"uri":"file:///d%3A/Coding/ws/workspace-2024-06-R/Test/src/my.hello","diagnostics":[{"severity":1,"range":{"start":{"line":11,"character":9},"end":{"line":11,"character":9}},"message":"Expecting token of type '!' but found ``.","data":{"code":"parsing-error"},"source":"hello-world"},{"message":"Person name should start with a capital.","range":{"start":{"character":7,"line":1},"end":{"character":10,"line":1}},"severity":2,"source":"hello-world"},{"message":"Person name should start with a capital.","range":{"start":{"character":7,"line":9},"end":{"character":11,"line":9}},"severity":2,"source":"hello-world"}]}}

Needless to say, that in VS Code everything works perferctly fine :)

Any ideas, what am I doing wrong?

I need to get this to work. If needed, (after some guadance) I can implement and contribute required fixes in langium.

Thank you in advance!

[cdietrich]

which langium version do you use? (there we some recent bugfixes around cancellation and build)
how did you configure lsp4e?
how does your semantic highlighting impl look like? (for clientside only highlighting/no semantic on server you need to configure tm4e)
maybe windows related.

[borisbrodski]

Langium: 3.1.0 (Just installed everything fresh using official tutorial)
Eclipse: 2024-06 (4.32.0) "Java package", fresh downloaded, no additional plugins, new workspace.
Node: v20.11.0
Windows 10

Eclipse - Run configuration "Run a program":

• Name: LSP hello world
• Location: D:\Software\node-v20.11.0-win-x64\node.exe
• Args:
  • D:\tmp\hello-world\out\language\main.js
  • --stdio

LSP configuration:

• Content type "hello" (under Text) added, file associations: *.hello
• Language Servers:
  • Content type: hello, launch configuration "LSP hello world", Launch Mode: run

(sorry, can't post screenshots from my corporate environment... )

[cdietrich]

• 3.1.0 exact or by range 3.1.2? (~)
• my run config looks the almost same with mac path. but i use the cjs so ...../out/language/main.cjs
• you either need to implement highlighting https://github.com/cdietrich/langium-did-close-support/blob/main/hello-world/src/language/hello-semantic-tokens.ts
• or you point tm4e to your textmate json.

[borisbrodski]

• You are right. Langium 3.1.2 is actually used.
• I tried both main.js and main.cjs
• Highlighting: it totally makes sence. Thank you!

[cdietrich]

Bildschirmaufnahme.2024-08-06.um.19.56.06.mov

[cdietrich]

with semantic highlighting implemented it also works

[borisbrodski]

Looks amazing! I hope, I will get it to work also on Windows (see my next post in this thread)

PS
Why actually I need to implement sematic highlighting to see regular keywords highlighted?
Will it be fixed in the future or is it kind of feature?

[borisbrodski]

It is a Windows related issue.

I did exactly the same setup on Linux and 2 Windows systems (Windows 10 corporate, Windows 11 private). Results:

• Linux: works (as on your video)
• Both Windows: exactly the same problem

I'm new to langium code base, but I tried to debug the didChange event. In document-update-handlers.js:50:

    didChangeContent(change) {
        this.fireDocumentUpdate([URI.parse(change.document.uri)], []);
    }

On key stroke in the Eclipse editor, the change.document._content contains updated document, but only change.document.uri get evaluated here, so the current state of the document is not used. Maybe I'm on some wrong place.

[cdietrich]

looking at the tests in xtext i linked above it seems intentional ;)

[cdietrich]

Langium must evaluate it somewhere in the code, since it's working on Linux.

i assume the

const documents = services.workspace.TextDocuments;
documents.listen(connection);

makes sure the TextDocuments reacts to did change

[msujew]

Could you point me to the code doing it?
Thank you!

I've linked it already earlier: This evaluates the current document content on document update:

langium/packages/langium/src/workspace/documents.ts

Lines 271 to 272 in b2c40e3

	const textDocument = this.textDocuments?.get(document.uri.toString());
	const text = textDocument ? textDocument.getText() : await this.fileSystemProvider.readFile(document.uri);

Evaluating the idea with URI encoding and going through logs, I found, that on some places, Langium LS responds with broken URI:

It responds with whatever VS Code would provide/expect (see also the Xtext tests linked above, which do the exact same). The URI issue inside of language servers is surprisingly complicated, given that we don't know what kind of URI is expected by the language client in the first place. We simply use whatever vscode-uri provides us with, which is an exact replica of the URIs used inside of VS Code. However, it is known that VS Code's URIs don't align to the RFC for URIs.

@cdietrich Do you perhaps know how other language servers deal with this?

[cdietrich]

i just remember we had to do 50 iterations over these so that the uri util as tested in the test can deal with all kind of client. (also vscode)

[cdietrich]

i assume lsp4e does not do the same
and thus might not be able to idenitfy the file

[borisbrodski]

Found the error:

 const textDocument = this.textDocuments?.get(document.uri.toString()); 
 const text = textDocument ? textDocument.getText() : await this.fileSystemProvider.readFile(document.uri);

document.uri.toString() evaluates to

• 'file:///d%3A/Coding/ws/workspace-2024-06-R2/Test/src/test.hello'

while this.textDocuments contains document with uri:

• 'file:///D:/Coding/ws/workspace-2024-06-R2/Test/src/test.hello'

Note also d vs D in the drive name.

So the textDocument is always undefined and get read from the FS.

[cdietrich]

const u = URI.parse("file:///D:/Coding/ws/workspace-2024-06-R2/Test/src/test.hello")
    console.log(u.toString())

=>

console.log
    file:///d%3A/Coding/ws/workspace-2024-06-R2/Test/src/test.hello

[cdietrich]

console.log(u.toString(true)) => file:///d:/Coding/ws/workspace-2024-06-R2/Test/src/test.hello

[msujew]

So its exactly what I assumed in #1619 (reply in thread). However, that's pretty non-trivial to fix. We would need to infer the URI encoding from the client messages and then respond appropriately.

[cdietrich]

wonder if a wrapper on textdocuments that does a URI.parse(original).toString() would help (assuming lsp4e is able to decode)

[msujew]

Yeah, assuming lsp4e is able to decode (which is seems to do correctly, since the diagnostics are displayed as per original post), that should probably help.

[borisbrodski]

Here is my VERY QUICK AND DIRTY fix:

In default-lsp-module.ts, function createDefaultSharedLSPModule:

        workspace: {
            TextDocuments: () => {
                const textDocuments = new TextDocuments(TextDocument);
                return new Proxy(textDocuments, {
                    get(target, property, receiver) {
                        if (property === 'get') {
                            return function(uri: string) {
                                const fixedWindowsURL = URI.parse(uri).toString(true).replace(/(file:\/\/\/)([a-z])(:\/)/, (_all, a, b, c) => a + b.toUpperCase() + c);
                                return target.get(fixedWindowsURL);
                            };
                        }
                        return Reflect.get(target, property, receiver);
                    }
                });
            }
        }

Tested only on Windows. Also npm run test is green.

[borisbrodski]

What do you think about this fix?
It's working for me locally.

Should I create an issue and then pull request?

[msujew]

Sure, a pull request would be appreciated :)

[msujew]

@borisbrodski I've provided a PR for this here: #1660. Your fix works for Eclipse only by the way, it breaks support for VS Code. The actual fix is a bit more elaborate. Our tests don't flag this, because we don't have automated VS Code tests.

[borisbrodski]

Thank you very much! Just tested it.

Your solution is much more involved indeed :)