gopls/internal/golang: semtok: use type information consistently

Previously there were two competing mechanisms for annotating
identifiers: the syntactic traversal and type information.
This meant that identifiers in different grammatical places
were annotated differently.

This change annotates identifiers using type information exclusively.
In addition to making things consistent, it also means that
types are reported independently (using modifiers) to symbol kind,
so, for example, a "var x func()" is a Variable with modifier
Signature, indicating that its type is a function.
Also, the rules for "defaultLibrary", "readonly", and so on
are more simply and consistently enforced.

The "deprecated" modifier is however lost as a consequence,
as it relied on the syntax.

+ Tests for both issues.

Also, use an enum for all the Modifiers.

Also, document the complete current set of token types
and modifiers that gopls returns.

Fixes golang/go#66809
Fixes golang/go#70251

Change-Id: I15e59d10f5a9269bc6be87f30e3502a9054d88e7
Reviewed-on: https://go-review.googlesource.com/c/tools/+/626279
Reviewed-by: Robert Findley <[email protected]>
Reviewed-by: Peter Weinberger <[email protected]>
LUCI-TryBot-Result: Go LUCI <[email protected]>

Author: adonovan

gopls/doc/features/passive.md

@@ -211,9 +211,46 @@ a portion of it.
 The client may use this information to provide syntax highlighting
 that conveys semantic distinctions between, for example, functions and
 types, constants and variables, or library functions and built-ins.
-Gopls also reports a modifier for the top-level constructor of each symbols's type, one of:
-`interface`, `struct`, `signature`, `pointer`, `array`, `map`, `slice`, `chan`, `string`, `number`, `bool`, `invalid`.
-The client specifies the sets of types and modifiers it is interested in.
+
+The client must specify the sets of types and modifiers it is interested in.
+
+Gopls reports the following token types:
+
+- `"comment"`: a comment
+- `"function"`: a function
+- `"keyword"`: a keyword
+- `"label"`: a control label (not an LSP standard type)
+- `"macro"`: text/template tokens
+- `"method"`: a method
+- `"namespace"`: an imported package name
+- `"number"`: a numeric literal
+- `"operator"`: an operator
+- `"parameter"`: a parameter variable
+- `"string"`:  a string literal
+- `"type"`: a type name (plus other uses)
+- `"typeParameter"`: a type parameter
+- `"variable"`: a var or const (see `readonly` modifier)
+
+Gopls also reports the following standard modifiers:
+
+- `"defaultLibrary": predeclared symbols
+- `"definition"`: the declaring identifier of a symbol
+- `"readonly"`: for constants
+
+plus these non-standard modifiers each representing the top-level
+constructor of each symbols's type:
+
+- `"array"`
+- `"bool"`
+- `"chan"`
+- `"interface"`
+- `"map"`
+- `"number"`
+- `"pointer"`
+- `"signature"`
+- `"slice"`
+- `"string"`
+- `"struct"`
 
 Settings:
 - The [`semanticTokens`](../settings.md#semanticTokens) setting determines whether

gopls/internal/cmd/integration_test.go

@@ -818,9 +818,9 @@ const c = 0
 		got := res.stdout
 		want := `
 /*⇒7,keyword,[]*/package /*⇒1,namespace,[]*/a
-/*⇒4,keyword,[]*/func /*⇒1,function,[definition]*/f()
-/*⇒3,keyword,[]*/var /*⇒1,variable,[definition]*/v /*⇒3,type,[defaultLibrary number]*/int
-/*⇒5,keyword,[]*/const /*⇒1,variable,[definition readonly]*/c = /*⇒1,number,[]*/0
+/*⇒4,keyword,[]*/func /*⇒1,function,[definition signature]*/f()
+/*⇒3,keyword,[]*/var /*⇒1,variable,[definition number]*/v /*⇒3,type,[defaultLibrary number]*/int
+/*⇒5,keyword,[]*/const /*⇒1,variable,[definition readonly number]*/c = /*⇒1,number,[]*/0
 `[1:]
 		if got != want {
 			t.Errorf("semtok: got <<%s>>, want <<%s>>", got, want)