IntelliJ does not attach sources for generated Java files or srcjars unless an in-source file is also present

[JohnnyMorganz]

Description of the bug:

When trying to debug a failing Bazel Java test, one may want to step into the relevant Java file to explore and attach breakpoints. For java_test targets where the sources are solely composed of generated files, this is not possible.

For example, in such a test, the test file is not link-able in the exception:

Selecting the test class in the side panel also does not direct us to the source file

However, if we also have Java source files (that may be completely irrelevant) as part of the java_test target, then Intellij shows a link and allows us to click through:

Clicking the test on the side panel also works.

This happens both when the generated files are directly added to the srcs, or if they are packaged into a .srcjar then the srcjar is added to the srcs.

Which category does this issue belong to?

Intellij

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Create a build file as below:

genrule(
    name = "generated-test-class",
    outs = ["com/foo/TestClass.java"],
    cmd = "\n".join([
        'echo "',
        "package com.foo;",
        "import org.junit.*;",
        "public class TestClass {",
        "   @Test",
        "   public void exampleTest() {",
        "       throw new RuntimeException();",
        "   }",
        "}",
        '" > $@',
    ]),
)

java_test(
    name = "test",
    srcs = [
        # "Dummy.java",
        ":generated-test-class",
    ],
    test_class = "com.foo.TestClass",
)

Run bazel sync, then try running the target. You will see no link available.

Next, create a Dummy.java file and uncomment the "Dummy.java" line, then re-run Bazel sync

package com.foo;

public class Dummy {
}

Try running the target, you will see links now appear and TestClass is accessible.

Writing the generated file into a .srcjar also reproduces the same issue

Which Intellij IDE are you using? Please provide the specific version.

IntelliJ IDEA 2024.1 Build #JBC-241.14494.240

What programming languages and tools are you using? Please provide specific versions.

Java 11, Bazel 7.2.1

What Bazel plugin version are you using?

2024.06.18.0.1-api-version-241

Have you found anything relevant by searching the web?

I've looked in the past issues, these look relevant but are not entirely the same problem. Linking in case it helps

• Generated files should be added to sources #205 - note that when we attach Dummy.java as a source it works, so the issue is not related to packages not matching. The issue also happens with srcjars
• -filtered-gen.jar and -filtered-gen-src.jar generated for packages without any .java files #6407

Any other information, logs, or outputs that you want to share?

If it helps, here is the intellij-info before and after the Dummy.java file is added. (A filtered gen jar is also created only when the sources are present, probably because of this -

intellij/aspect/intellij_info_impl.bzl

Lines 687 to 695 in d55c435

	if java_sources and (gen_java_sources or srcjars):
	filtered_gen_jar, filtered_gen_resolve_files = _build_filtered_gen_jar(
	ctx,
	target,
	java_outputs,
	gen_java_sources,
	srcjars,
	)
	resolve_files += filtered_gen_resolve_files

)

Before

build_file_artifact_location {
  is_external: false
  is_source: true
  relative_path: "test/BUILD"
  root_execution_path_fragment: ""
}
deps {
  dependency_type: 0
  target {
    label: "@@bazel_tools//tools/cpp:optional_current_cc_toolchain"
  }
}
deps {
  dependency_type: 0
  target {
    label: "@@rules_java~//toolchains:current_java_toolchain"
  }
}
features: "cpp23"
features: "lto16"
features: "nolto"
features: "sysroot"
features: "werror"
java_ide_info {
  jars {
    jar {
      is_external: false
      is_source: false
      relative_path: "test/test.jar"
      root_execution_path_fragment: "bazel-out/k8-dbg--cd/bin"
    }
    source_jar {
      is_external: false
      is_source: false
      relative_path: "test/test-src.jar"
      root_execution_path_fragment: "bazel-out/k8-dbg--cd/bin"
    }
    source_jars {
      is_external: false
      is_source: false
      relative_path: "test/test-src.jar"
      root_execution_path_fragment: "bazel-out/k8-dbg--cd/bin"
    }
  }
  jdeps {
    is_external: false
    is_source: false
    relative_path: "test/test.jdeps"
    root_execution_path_fragment: "bazel-out/k8-dbg--cd/bin"
  }
  main_class: ""
  sources {
    is_external: false
    is_source: false
    relative_path: "test/com/foo/TestClass.java"
    root_execution_path_fragment: "bazel-out/k8-dbg--cd/bin"
  }
  test_class: "com.foo.TestClass"
}
key {
  label: "//test:test"
}
kind_string: "java_test"
test_info {
  size: "medium"
}

After

build_file_artifact_location {
  is_external: false
  is_source: true
  relative_path: "test/BUILD"
  root_execution_path_fragment: ""
}
deps {
  dependency_type: 0
  target {
    label: "@@bazel_tools//tools/cpp:optional_current_cc_toolchain"
  }
}
deps {
  dependency_type: 0
  target {
    label: "@@rules_java~//toolchains:current_java_toolchain"
  }
}
features: "cpp23"
features: "lto16"
features: "nolto"
features: "sysroot"
features: "werror"
java_ide_info {
  filtered_gen_jar {
    jar {
      is_external: false
      is_source: false
      relative_path: "test/test-filtered-gen.jar"
      root_execution_path_fragment: "bazel-out/k8-dbg--cd/bin"
    }
    source_jar {
      is_external: false
      is_source: false
      relative_path: "test/test-filtered-gen-src.jar"
      root_execution_path_fragment: "bazel-out/k8-dbg--cd/bin"
    }
  }
  jars {
    jar {
      is_external: false
      is_source: false
      relative_path: "test/test.jar"
      root_execution_path_fragment: "bazel-out/k8-dbg--cd/bin"
    }
    source_jar {
      is_external: false
      is_source: false
      relative_path: "test/test-src.jar"
      root_execution_path_fragment: "bazel-out/k8-dbg--cd/bin"
    }
    source_jars {
      is_external: false
      is_source: false
      relative_path: "test/test-src.jar"
      root_execution_path_fragment: "bazel-out/k8-dbg--cd/bin"
    }
  }
  jdeps {
    is_external: false
    is_source: false
    relative_path: "test/test.jdeps"
    root_execution_path_fragment: "bazel-out/k8-dbg--cd/bin"
  }
  main_class: ""
  package_manifest {
    is_external: false
    is_source: false
    relative_path: "test/test.java-manifest"
    root_execution_path_fragment: "bazel-out/k8-dbg--cd/bin"
  }
  sources {
    is_external: false
    is_source: true
    relative_path: "test/Dummy.java"
    root_execution_path_fragment: ""
  }
  sources {
    is_external: false
    is_source: false
    relative_path: "test/com/foo/TestClass.java"
    root_execution_path_fragment: "bazel-out/k8-dbg--cd/bin"
  }
  test_class: "com.foo.TestClass"
}
key {
  label: "//test:test"
}
kind_string: "java_test"
test_info {
  size: "medium"
}

[tpasternak]

@JohnnyMorganz can you please take a look into project structure and check what's there?

1. Double-press shift
2. "Project Structure"
3. Libraries
4. Try to look at the contents. I think it should change in the background once you try to navigate to the class files for the first time, so you could also try before/after

[JohnnyMorganz]

Hey @tpasternak

With the Dummy.java line commented out, the "Libraries" panel is completely empty

If I then uncomment the line and resync, I see the following:

I don't see any changes after navigating to the class files

[tpasternak]

Could you please check this out? bazelbuild/bazel#22254

[tpasternak]

What I can recommend, is just to restructure the directories in a way they would match the heuristic described in the bazel ticket

[JohnnyMorganz]

Thanks, although I'm not sure if that issue is exactly the same

I created this minimal repro from a larger example. In that example, we are generating a .srcjar, which already is structured appropriately with the path matching the package path:

unzip -l bazel-bin/path/to/generated.srcjar
Archive:  bazel-bin/path/to/generated.srcjar
  Length      Date    Time    Name
---------  ---------- -----   ----
   198108  01-01-1980 00:00   com/foo/TestClass.java
---------                     -------
   198108                     1 file

In my minimal example, I also have Dummy.java written to test/Dummy.java (instead of test/src/main/com/foo/Dummy.java). And including it in srcs causes the classes to resolve successfully. So I don't think the path of the files are impacting it.

[JohnnyMorganz]

For completeness, with the following sources jar, the generated source file resolves correctly in IntelliJ:

$ bazel build //test:test-src.jar //test:test.jar
$ unzip -l bazel-bin/test/test-src.jar
Archive:  bazel-bin/test/test-src.jar
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  01-01-2010 00:00   META-INF/
       43  01-01-2010 00:00   META-INF/MANIFEST.MF
        0  01-01-2010 00:00   test/
       45  01-01-2010 00:00   test/Dummy.java
        0  01-01-2010 00:00   test/com/
        0  01-01-2010 00:00   test/com/foo/
      148  01-01-2010 00:00   test/com/foo/TestClass.java
---------                     -------
      236                     7 files

If we remove test/Dummy.java and then rebuild, it does not resolve correctly.

In the Intellij libraries panel, it lists the test-filtered-gen-src.jar.

If I remove test/Dummy.java, this filtered gen src jar is not generated, and there is nothing added to libraries.

[JohnnyMorganz]

Also just to simplify things a bit further, I changed the minimal example slightly so that the source paths match exactly with the Java package path. You can still repro the issue in this case.

# test/BUILD.bazel
genrule(
    name = "generated-test-class",
    outs = ["TestClass.java"],
    cmd = "\n".join([
        'echo "',
        "package test;",
        "import org.junit.*;",
        "public class TestClass {",
        "   @Test",
        "   public void exampleTest() {",
        "       throw new RuntimeException();",
        "   }",
        "}",
        '" > $@',
    ]),
)

java_test(
    name = "test",
    srcs = [
        #        "Dummy.java",
        ":generated-test-class",
    ],
    test_class = "test.TestClass",
)

$ unzip -l bazel-bin/test/test-src.jar
Archive:  bazel-bin/test/test-src.jar
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  01-01-2010 00:00   META-INF/
       43  01-01-2010 00:00   META-INF/MANIFEST.MF
        0  01-01-2010 00:00   test/
      145  01-01-2010 00:00   test/TestClass.java
---------                     -------
      188                     4 files

[tpasternak]

So I just tried to change java_test to java_library and reference it from yet another library. All the indexing was done correctly I could reference the symbols.

[tpasternak]

Screen.Recording.2024-08-27.at.16.15.52.mov

[JohnnyMorganz]

Interesting thanks, I indeed see the same behaviour that you see when I switch to java_library, however that isn't exactly the same structure that I had in the original issue.

Do you experience the behaviour I mentioned originally when using a single java_test? (EDIT: or alternatively 2 targets with no in-source files in the second target, see video 4) I've attached some screen recordings showing my setup and what I see. Please let me know if there is anything else I should test:

I cleaned my bazel output and restarted IntelliJ to ensure I was in a clean state. I'm using the blue links in terminal to indicate the correct source file was resolved or not, as well as checking the Libraries panel.

Before with no Dummy file synced:

NoDummyFile.mov

After we sync with the Dummy file:

WithDummyFile.mov

To double check, using a separate java library with a source file, like you mention:

Screen.Recording.2024-08-27.at.17.35.29.mov

Also, I removed the Dummy.java from the second build target and we get back to the same issue as before (however I did have to switch from deps to runtime_deps as there are no srcs and Bazel complains, so maybe that is a separate thing):

Screen.Recording.2024-08-27.at.17.41.58.mov

For context, the example I am minimising this from is a java test with only generated sources, no in-source files (i.e., video 1 and 4 above)

[tpasternak]

Ok, thanks, now I understand the problem. However still, I don't have capacity to work on it

[JohnnyMorganz]

That's fair enough, I think we can make do with the workaround by including a Dummy in-source file as part of the generated target. Thanks for taking the time to investigate!