IntellIJ doesn't recognize generated grpc/protobuf for golang

[justinsb]

Sorry for what is a very similar issue to the ones for Java & Python. Not sure if I'm doing something wrong, but IntelliJ won't recognize protobuf-generated types in my go projeccts.

I have an example project that is open source: https://github.com/kopeio/etcd-manager/blob/master/pkg/apis/etcd/BUILD.bazel

For reference:

proto_library(
    name = "etcd_proto",
    srcs = ["etcdapi.proto"],
    visibility = ["//visibility:public"],
)

go_grpc_library(
    name = "etcd_go_proto",
    importpath = "kope.io/etcd-manager/pkg/apis/etcd",
    proto = ":etcd_proto",
    visibility = ["//visibility:public"],
)

Not sure if I'm not setting up my project correctly in IntelliJ (GOPATHs? Marking as generated sources?) The workaround I've found is to symlink to ./bazel-bin/pkg/apis/etcd/etcd_go_proto~/kope.io/etcd-manager/pkg/apis/etcd/etcdapi.pb.go.

Any better suggestions appreciated! Or if this should "just work" let me know if I can provide more diagnostic information.

Go plugin version: 173.3727.144
Bazel plugin version: 2017.11.20.0.4
Bazel version: 0.8.1
IntelliJ version: IU-173.4127.27 (aka 2017.3.2)

[schmitch]

actually another workaround is setting a $GOPATH and creating the "generated" files into the protobuf.
I currently "play" with bazel and do not use a $GOPATH at all, but have it set to a completly different path than my project and generate the protos there for syntax highlighting. intellij with goglang will detect it than.

[hexaglow]

Just wondering if there has been any progress on this. I am also encountering this issue, on Bazel 0.16.1 + IntelliJ 2018.1.6 + plugin 2018.08.20.0.3.

I have created a repository at https://github.com/hexaglow/bazel-intellij-194 that demonstrates the issue. After importing the project, the generated code cannot be resolved by IntelliJ causing incorrect error highlighting

[chaoren]

Sorry, lost track of this.

Does go_proto_library instead of go_grpc_library work?

Go protobuf support is already implemented, but the go rules in blaze is sufficiently different from bazel's rules_go that there's a bunch of these issues causing our aspect to not be able to locate go sources with bazel.

I think this should be caused by something similar to #360, where go_grpc_library does not contain the provider that we need.

[hexaglow]

Using only go_proto_library also does not work, same problem. I have added a branch protobuf-only to the repository listed above that demonstrates this.

[hexaglow]

It seems go_grpc_library is just a macro wrapper that calls go_proto_library, specifying the GRPC compiler. So both should be returning the same providers.
I suspect that information on generated sources is not exposed, or perhaps it is exposed but not in the way the plugin expects. Unfortunately my Bazel knowledge is not really good enough to investigate further.

[chaoren]

Okay, same problem as #214.

We'll need load("@io_bazel_rules_go//go:def.bzl", "GoSource") to use the providers.
I don't think conditional load is a thing yet.

[mtsgrd]

I have little experience with both Go and this plugin, but managed to get generated proto_library files included using GoSource. Is this commit somewhere along the lines of what you had in mind @chaoren?

    ...
    elif ctx.rule.kind == 'go_proto_library':
        proto_files = getattr(target[GoSource], "srcs", [])
        proto_sources = [f for f in proto_files if f.basename.endswith(".pb.go")]
        if not proto_sources:
            return False
        sources += proto_sources
        generated += proto_sources
    ...

[chaoren]

Yes that does work. But then our aspect would not be able to support any project that does not use go.
We could also just generate the .pb.go file paths from .proto file locations in the plugin.

[ceason]

In lieu of conditional loads we have at least two solutions that could work today:

1. Updating rules_go to return the legacy-style provider that the intellij plugin is already expecting. Downside is using a deprecated API, but upside is rules_go is on a much faster release cycle than the intellij plugin.
2. Update rules_go and bazelbuild/intellij to expose and consume, respectively, an appropriately named output group (perhaps 'generated_srcs' or 'go_generated_srcs'?)

Additionally, to pick up everything (eg go_proto_library embedded in go_library), the intellij aspect needs to propagate along the 'embed' attr (though I suspect this needs to happen independent of this use case).

@chaoren what do you think about option 1 vs 2?

[chaoren]

We talked to the rules_go team about option 1. No chance of that happening.

Option 2 makes some sense. Since the .pb.go file is indeed generated, and you can conceivably call that an output. But I think the rules_go team is still expecting this issue to be solved via the current providers.

[mattnathan]

I'm not sure if changes are partially done, but it seems that the *intellij-info generated for a go_proto_library includes the .pb.go file in the metadata

build_file_artifact_location {
  is_external: false
  is_new_external_version: true
  is_source: true
  relative_path: "protobuf/playground/grpc-test/helloworld/BUILD.bazel"
  root_execution_path_fragment: ""
}
go_ide_info {
  import_path: "example.com/protobuf/playground/grpc-test/helloworld"
  library_kind: "go_proto_library"
  library_label: "//protobuf/playground/grpc-test/helloworld:helloworld_go_proto"
  sources {
    is_external: false
    is_new_external_version: true
    is_source: false
    relative_path: "protobuf/playground/grpc-test/helloworld/windows_amd64_stripped/helloworld_go_proto%/example.com/protobuf/playground/grpc-test/helloworld/helloworld.pb.go"
    root_execution_path_fragment: "bazel-out/x64_windows-fastbuild/bin"
  }
}
key {
  label: "//protobuf/playground/grpc-test/helloworld:helloworld_go_proto"
}
kind_string: "go_proto_library"

However the intellij plugin doesn't seem to take it into account.

Using rules_go 0.17.1

[chaoren]

So I tried out
https://github.com/kopeio/etcd-manager/blob/master/pkg/apis/etcd/BUILD.baze, after upgrading to 0.17.1, and reverting pkg/apis/etcd back to a911a31098fa6e462435c418be75a2ff0347eb9d~.
It works if the go_library import path is not exactly the same the as go_proto_library's import path.
Does rules_go actually allow you to merge multiple targets into single go packages? We have a special casing for tests that share go packages with their libraries, but I didn't know people did this with arbitrary targets.

[chaoren]

For reference in blaze (internal bazel), the import paths are derived from the full target name, and there's no explicit importpath field, so it's literally impossible for this kind of conflict to happen there.

[mattnathan]

While creating an example project (https://github.com/vanti-public/ijwb_194) I stumbled across the cause of the issue: Vgo support.

Turning vgo support off in the IDE allows the generated proto files to resolve as expected. Turning it on causes the issues I've been having.

[mstg]

Has anyone found any fix for this? The issue persist only in some packages. Even if it builds and runs correctly some packages won't recognize the generated proto file (even if it is to be found in bazel-out) in the IDE. The inconsistency is weird. I'm trying to look into the plugin to debug this and will post an update if I find anything.

As #194 (comment) points out changing the importpath seems to fixing the issue. And as I've understood it only happens when there is other sources in the same package with generated sources and both targets have the same importpath

[mstg]

Good news! I don't know how correct this is but this patch https://gist.github.com/mstg/051d0bf16941df49312b3a80f868903b seems to have worked!

Apply it with patch -p1 to this codebase and build with bazel build //ijwb:ijwb_bazel_zip --define=ij_product=intellij-ue-latest

I can create a PR if this is a desired fix. Please advise.

[awkr]

I've tried a lot, but GoLand still doesn't recognize generated pb.go file, although Bazel can run target successfully. I think using symlink or writing bzl maybe not the perfect solution, because this should be the plugin's work, and project should only contain business logic.

[ppmathis]

I am having the same issue as described by @mattnathan in #194 (comment) - as soon as Go Modules (vgo) support is being enabled in IntelliJ IDEA Ultimate Edition 2019.3.4, the import paths for the generated proto files no longer resolve and show the warning Build constraints exclude all files in <path>.

Disabling Go Modules support in IntelliJ works, however this is not really a fortunate solution. The patch provided by @mstg in #194 (comment) unfortunately did not resolve this issue either.

[iamricard]

Hi! This seems to still be an issue unfortunately. As folks mentioned earlier this seems to stem from a conflict with GoLand's Go Modules support, and turning support for it does in fact get rid of the issue. However, that creates another issue. Because Go Modules support is turned off GoLand doesn't pick up any new modules if you update the go.mod file (a workaround seems to be to toggle go modules support, but that's less than ideal!).

I'm not familiar enough with how the plugin system works to fix it, but if someone can coach me through it I'm happy to work on a fix!

Update:

I added a comment in https://youtrack.jetbrains.com/issue/GO-9040#focus=Comments-27-4879464.0-0 because I'm assuming it's IntelliJ's Go Modules integration that is breaking this. I have also created an up-to-date repro at https://github.com/iamricard/goland-bazel-mods-repro.

Also, none of the fixes/patches above fix this issue. I think this is because the go mod integration is the culprit here. My assumptions is they'd need to expose something that would let the bazel plugin list the generated files as part of the module.

[c16a]

Disabling Go modules also makes sense because Bazel doesn't use it anyway. I'm settling with that.

[loeffel-io]

Hi! This seems to still be an issue unfortunately. As folks mentioned earlier this seems to stem from a conflict with GoLand's Go Modules support, and turning support for it does in fact get rid of the issue. However, that creates another issue. Because Go Modules support is turned off GoLand doesn't pick up any new modules if you update the go.mod file (a workaround seems to be to toggle go modules support, but that's less than ideal!).

I'm not familiar enough with how the plugin system works to fix it, but if someone can coach me through it I'm happy to work on a fix!

Update:

I added a comment in https://youtrack.jetbrains.com/issue/GO-9040#focus=Comments-27-4879464.0-0 because I'm assuming it's IntelliJ's Go Modules integration that is breaking this. I have also created an up-to-date repro at https://github.com/iamricard/goland-bazel-mods-repro.

Also, none of the fixes/patches above fix this issue. I think this is because the go mod integration is the culprit here. My assumptions is they'd need to expose something that would let the bazel plugin list the generated files as part of the module.

Thanks ❤️

[timduly4]

Disabling Go modules also makes sense because Bazel doesn't use it anyway. I'm settling with that.

I had this option disabled but am still running into this issue -- any hints?

[jschaf]

After upgrading to IntelliJ 2023.1.3, IntelliJ insists on opening my project with Bazel, creating the .ijwb project if I remove it. I've been using a regular non-Bazel project folder, .idea, for the last few years to avoid some of the rough edges of Intellij with Bazel.

The good news is that almost everything works, including debugging! The bad news is:

1. I can't coerce IntelliJ to open my project with the normal project directory .idea. Eventually, I uninstalled the Bazel plugin and restarted IntelliJ.

2. If I stick with the .ijwb project, I can't figure out how to make Go protobufs resolve (this issue, the same as @timduly4). I've tried disabling Go modules and resyncing a few times.

[blorente]

Protobuf should be supported for go as of #6030 .