typechecking fails when package declaration and source location not in sync

[scabug]

This is a presentation compiler bug we have known for a really long time in the Scala IDE, but I just realised we have never opened a ticket here (despite talking about the issue with several members of the Scala team).

Basically, the presentation compilers fails to resolve symbols from source whose package declaration is not in sync with the source filesystem location.

a/b/A.scala
package a // notice here it's `a` and not `a.b`

class A

// Foo.scala
object Foo {
  new a.A()
}

Typechecking source Foo.scala alone reports the following error

type A is not a member of package a

The problem goes away if source a/b/A.scala is also typechecked.

We have a number of tickets in the Scala IDE issue tracker that are linked to issue reported here, e.g., https://scala-ide-portfolio.assembla.com/spaces/scala-ide/tickets/1001154

[scabug]

Imported From: https://issues.scala-lang.org/browse/SI-9309?orig=1
Reporter: @dotta
Affected Versions: 2.11.6

[scabug]

@retronym said:
Here is a test case:

//package scala

import java.io.File

import scala.reflect.internal.util.BatchSourceFile
import scala.reflect.io.Directory
import scala.tools.nsc._
import scala.tools.nsc.interactive
import scala.tools.nsc.reporters.StoreReporter

object t9309 {

  def main(args: Array[String]) {
    val sourceDir = Directory.makeTemp()
    val a = (sourceDir / "a").createDirectory()
    a.jfile.mkdirs()
    val b = (a / "b").createDirectory()
    b.jfile.mkdirs()
    val A = (b / "A.scala").createFile()
    A.createDirectory()
    A.writeAll("package a; class A")

    val settings = new Settings(sys.error(_))

    settings.sourcepath.value = sourceDir.jfile.getAbsolutePath
    settings.uniqid.value = true
    settings.stopAfter.value = List("typer")

    val foo = new BatchSourceFile("Foo.scala", "object Foo { new a.A() }")

    val tester = new interactive.tests.Tester(1, Array(foo), settings)

    val storeReporter = new StoreReporter
    tester.compiler.reporter = storeReporter
    tester.compiler.ask { () =>
      val run = new tester.compiler.Run
      run.compileSources(List(foo))
      assert(storeReporter.infos.isEmpty, storeReporter.infos)
      // println(tester.compiler.show(run.units.toList.head.body, printIds = true))
    }
    println("done.")
  }
}

Caused by: java.lang.AssertionError: assertion failed: Set(pos: RangePosition(Foo.scala, 17, 19, 20) type A is not a member of package a#75 ERROR)
	at scala.Predef$.assert(Predef.scala:165)

[scabug]

@retronym said:
The current behaviour is basically an optimization to only parse a subset of source files related to a package that is referred to in the current source file. AFAICT, to support arbitrary file layouts, we would need to eagerly perform an outline parser of all files in the source path at the start of each compilation run. Outline parsing is cheaper than full parsing (very few AST nodes are allocated), but is still not free.

Here's a rough prototype of that approach that makes that test pass:

https://github.com/retronym/scala/compare/ticket/9309

A more refined approach would be possible if the IDE could use its knowledge of file modification times to maintain a data structure to record which source files contain top level definitions in each package. The presentation compiler could use this to selectively perform outline parsing of relevant source files when completing a package symbol.

[scabug]

@SethTisue said:
details on some efforts to push Jason's WIP branch further along are at scala/scala-dev#56