Skip to content

Unstable compilation output of src/library due to stray references to Predef.class before Predef.scala is named #12086

New issue
New issue
Open
Open
Unstable compilation output of src/library due to stray references to Predef.class before Predef.scala is named#12086
Assignees
retronym
Labels
compiler determinism
Milestone
Backlog
@retronym

Description

@retronym
retronym
opened on Jul 17, 2020
Member

reproduction steps

using Scala 2.12.12, and the extended version of DeterminismTest I'm working on in retronym/scala#96, the pickle for scala.Symbol.class differs depending on the order/subset of source files being compiled:

sbt:root> junit/test:runMain scala.tools.nsc.DeterminismTester /Users/jz/code/scala/src/library

--- /var/folders/22/g1sv634d11j1d_lqlnhz9p2r0000gn/T/reference1538611564431125389/scala/Symbol.class
+++ /var/folders/22/g1sv634d11j1d_lqlnhz9p2r0000gn/T/recompileOutput2404148374994820649/scala/Symbol.class
@@ -1,17 +1,17 @@
 // class version 52.0 (52)
 // access flags 0x31
 public final class scala/Symbol implements scala/Serializable {

   // compiled from: Symbol.scala

-  @Lscala/reflect/ScalaSignature;(bytes="\u0006\u0001!4Aa\u0004\u0009\u0003'!A1\u0004\u0001BC\u0002\u0013\u0005A\u0004\u0003\u0005)\u0001\u0009\u0005\u0009\u0015!\u0003\u001e\u0011\u0015I\u0003\u0001\"\u0003+\u0011\u0015i\u0003\u0001\"\u0011/\u0011\u0015y\u0003\u0001\"\u00031\u0011\u0015\u0009\u0005\u0001\"\u0011C\u0011\u00151\u0005\u0001\"\u0011H\u000f\u0015i\u0005\u0003#\u0001O\r\u0015y\u0001\u0003#\u0001P\u0011\u0015I\u0013\u0002\"\u0001T\u0011\u0015!\u0016\u0002\"\u0011V\u0011\u00159\u0016\u0002\"\u0005Y\u0011\u0015Q\u0016\u0002\"\u0005\\\u0011\u001dy\u0013\"!A\u0005\n\u0005\u0014aaU=nE>d'\"A\u0009\u0002\u000bM\u001c\u0017\r\\1\u0004\u0001M\u0019\u0001\u0001\u0006\r\u0011\u0005U1R\"\u0001\u0009\n\u0005]\u0001\"AB!osJ+g\r\u0005\u0002\u00163%\u0011!\u0004\u0005\u0002\r'\u0016\u0014\u0018.\u00197ju\u0006\u0014G.Z\u0001\u0005]\u0006lW-F\u0001\u001e!\u0009qRE\u0004\u0002 GA\u0011\u0001\u0005E\u0007\u0002C)\u0011!EE\u0001\u0007yI|w\u000e\u001e \n\u0005\u0011\u0002\u0012A\u0002)sK\u0012,g-\u0003\u0002'O\u000911\u000b\u001e:j]\u001eT!\u0001\n\u0009\u0002\u000b9\u000cW.\u001a\u0011\u0002\rqJg.\u001b;?)\u0009YC\u0006\u0005\u0002\u0016\u0001!)1d\u0001a\u0001;\u0005AAo\\*ue&tw\rF\u0001\u001e\u0003-\u0011X-\u00193SKN|GN^3\u0015\u0003E\u0002\"!\u0006\u001a\n\u0005M\u0002\"aA!os\"\u001aQ!\u000e!\u0011\u0007U1\u0004(\u0003\u00028!\u00091A\u000f\u001b:poN\u0004\"!\u000f \u000e\u0003iR!a\u000f\u001f\u0002\u0005%|'\"A\u001f\u0002\u0009)\u000cg/Y\u0005\u0003�i\u0012Qc\u00142kK\u000e$8\u000b\u001e:fC6,\u0005pY3qi&|gnI\u00019\u0003!A\u0017m\u001d5D_\u0012,G#A\"\u0011\u0005U!\u0015BA#\u0011\u0005\rIe\u000e^\u0001\u0007KF,\u0018\r\\:\u0015\u0005![\u0005CA\u000bJ\u0013\u0009Q\u0005CA\u0004C_>dW-\u00198\u0009\u000b1;\u0001\u0019A\u0019\u0002\u000b=$\u0008.\u001a:\u0002\rMKXNY8m!\u0009)\u0012bE\u0002\n!b\u0001B!F)\u001eW%\u0011!\u000b\u0005\u0002\u0010+:L\u0017/^3oKN\u001c8)Y2iKR\u0009a*A\u0003baBd\u0017\u0010\u0006\u0002,-\")1d\u0003a\u0001;\u0005aa/\u00197vK\u001a\u0013x.\\&fsR\u00111&\u0017\u0005\u000671\u0001\r!H\u0001\rW\u0016LhI]8n-\u0006dW/\u001a\u000b\u00039~\u00032!F/\u001e\u0013\u0009q\u0006C\u0001\u0004PaRLwN\u001c\u0005\u0006A6\u0001\raK\u0001\u0004gflG#\u00012\u0011\u0005\r4W\"\u00013\u000b\u0005\u0015d\u0014\u0001\u00027b]\u001eL!a\u001a3\u0003\r=\u0013'.Z2u\u0001")

Diffing extra trace output added to the pickler:

image

analysis

We end up with two symbols representing type alias Predef.String = java.lang.String. One of these symbols is created when the Predef.class is loaded from the classpath when loading the scala.package package object

java.lang.Throwable
	at scala.reflect.internal.Types$AliasTypeRef.$init$(Types.scala:1957)
	at scala.reflect.internal.Types$AliasNoArgsTypeRef.<init>(Types.scala:2411)
	at scala.reflect.internal.Types$TypeRef$.apply(Types.scala:2423)
	at scala.reflect.internal.pickling.UnPickler$Scan.readType(UnPickler.scala:415)
	at scala.reflect.internal.pickling.UnPickler$Scan.$anonfun$readTypeRef$1(UnPickler.scala:656)
	at scala.reflect.internal.pickling.UnPickler$Scan.at(UnPickler.scala:190)
	at scala.reflect.internal.pickling.UnPickler$Scan.readTypeRef(UnPickler.scala:656)
	at scala.reflect.internal.pickling.UnPickler$Scan.readNonEmptyTree(UnPickler.scala:605)
	at scala.reflect.internal.pickling.UnPickler$Scan.readTree(UnPickler.scala:619)
	at scala.reflect.internal.pickling.UnPickler$Scan.$anonfun$readAnnotArg$1(UnPickler.scala:467)
	at scala.reflect.internal.pickling.UnPickler$Scan.at(UnPickler.scala:190)
	at scala.reflect.internal.pickling.UnPickler$Scan.readAnnotArg(UnPickler.scala:467)
	at scala.reflect.internal.pickling.UnPickler$Scan.readAnnotationInfo(UnPickler.scala:501)
	at scala.reflect.internal.pickling.UnPickler$Scan.readSymbolAnnotation(UnPickler.scala:513)
	at scala.reflect.internal.pickling.UnPickler$Scan.$anonfun$run$2(UnPickler.scala:108)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.reflect.internal.pickling.UnPickler$Scan.runAtIndex(UnPickler.scala:90)
	at scala.reflect.internal.pickling.UnPickler$Scan.run(UnPickler.scala:108)
	at scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:47)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.unpickleOrParseInnerClasses(ClassfileParser.scala:1182)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.parseClass(ClassfileParser.scala:466)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$2(ClassfileParser.scala:160)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$2$adapted(ClassfileParser.scala:146)
	at scala.reflect.internal.util.ReusableInstance.using(ReusableInstance.scala:30)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$1(ClassfileParser.scala:146)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.pushBusy(ClassfileParser.scala:129)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.parse(ClassfileParser.scala:145)
	at scala.tools.nsc.symtab.SymbolLoaders$ClassfileLoader.doComplete(SymbolLoaders.scala:333)
	at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:240)
	at scala.reflect.internal.Symbols$Symbol.completeInfo(Symbols.scala:1542)
	at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
	at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:355)
	at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:410)
	at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:303)
	at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:240)
	at scala.reflect.internal.Symbols$Symbol.completeInfo(Symbols.scala:1542)
	at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
	at scala.reflect.internal.Types$TypeRef.decls(Types.scala:2285)
	at scala.tools.nsc.typechecker.Namers$Namer.enterPackage(Namers.scala:765)
	at scala.tools.nsc.typechecker.Namers$Namer.dispatch$1(Namers.scala:288)
	at scala.tools.nsc.typechecker.Namers$Namer.standardEnterSym(Namers.scala:301)
	at scala.tools.nsc.typechecker.AnalyzerPlugins.pluginsEnterSym(AnalyzerPlugins.scala:479)
	at scala.tools.nsc.typechecker.AnalyzerPlugins.pluginsEnterSym$(AnalyzerPlugins.scala:478)
	at scala.tools.nsc.Global$$anon$5.pluginsEnterSym(Global.scala:483)
	at scala.tools.nsc.typechecker.Namers$Namer.enterSym(Namers.scala:279)
	at scala.tools.nsc.typechecker.Analyzer$namerFactory$$anon$1.apply(Analyzer.scala:50)
	at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:454)
	at scala.tools.nsc.Global$GlobalPhase.run(Global.scala:402)

This is referred to when computing the root imports:

java.lang.Throwable
	at scala.reflect.internal.Symbols$AliasTypeSymbol.<init>(Symbols.scala:3059)
	at scala.reflect.internal.Symbols$Symbol.createAliasTypeSymbol(Symbols.scala:1331)
	at scala.reflect.internal.Symbols$Symbol.newNonClassSymbol(Symbols.scala:1397)
	at scala.reflect.internal.pickling.UnPickler$Scan.readSymbol(UnPickler.scala:346)
	at scala.reflect.internal.pickling.UnPickler$Scan.$anonfun$run$1(UnPickler.scala:98)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.reflect.internal.pickling.UnPickler$Scan.runAtIndex(UnPickler.scala:90)
	at scala.reflect.internal.pickling.UnPickler$Scan.run(UnPickler.scala:98)
	at scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:47)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.unpickleOrParseInnerClasses(ClassfileParser.scala:1182)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.parseClass(ClassfileParser.scala:466)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$2(ClassfileParser.scala:160)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$2$adapted(ClassfileParser.scala:146)
	at scala.reflect.internal.util.ReusableInstance.using(ReusableInstance.scala:30)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$1(ClassfileParser.scala:146)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.pushBusy(ClassfileParser.scala:129)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.parse(ClassfileParser.scala:145)
	at scala.tools.nsc.symtab.SymbolLoaders$ClassfileLoader.doComplete(SymbolLoaders.scala:333)
	at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:240)
	at scala.reflect.internal.Symbols$Symbol.completeInfo(Symbols.scala:1542)
	at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
	at scala.reflect.internal.Symbols$Symbol.tpeHK(Symbols.scala:1468)
	at scala.reflect.internal.Types$Type.computeMemberType(Types.scala:735)
	at scala.reflect.internal.Types$Type.memberType(Types.scala:732)
	at scala.reflect.internal.TreeGen.mkAttributedSelect(TreeGen.scala:248)
	at scala.reflect.internal.TreeGen.mkAttributedRef(TreeGen.scala:174)
	at scala.reflect.internal.TreeGen.mkAttributedStableRef(TreeGen.scala:209)
	at scala.tools.nsc.ast.TreeGen.mkImportFromSelector(TreeGen.scala:43)
	at scala.tools.nsc.ast.TreeGen.mkWildcardImport(TreeGen.scala:33)
	at scala.tools.nsc.typechecker.Contexts.$anonfun$rootContext$1(Contexts.scala:111)
	at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
	at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
	at scala.collection.immutable.List.foldLeft(List.scala:91)
	at scala.collection.TraversableOnce.$div$colon(TraversableOnce.scala:179)
	at scala.collection.TraversableOnce.$div$colon$(TraversableOnce.scala:179)
	at scala.collection.AbstractTraversable.$div$colon(Traversable.scala:108)
	at scala.tools.nsc.typechecker.Contexts.rootContext(Contexts.scala:111)
	at scala.tools.nsc.typechecker.Contexts.rootContext$(Contexts.scala:110)
	at scala.tools.nsc.Global$$anon$5.rootContext(Global.scala:483)
	at scala.tools.nsc.typechecker.Analyzer$namerFactory$$anon$1.apply(Analyzer.scala:50)
	at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:454)
	at scala.tools.nsc.Global$GlobalPhase.run(Global.scala:402)
	at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1511)
	at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1495)
	at scala.tools.nsc.Global$Run.compileSources(Global.scala:1488)
	at scala.tools.nsc.DeterminismTester.compile$1(DeterminismTester.scala:56)
	at scala.tools.nsc.DeterminismTester.$anonfun$test$7(DeterminismTester.scala:97)
	at scala.tools.nsc.DeterminismTester.$anonfun$test$7$adapted(DeterminismTester.scala:94)
	at scala.collection.immutable.List.foreach(List.scala:431)
	at scala.tools.nsc.DeterminismTester.test(DeterminismTester.scala:94)
	at scala.tools.nsc.DeterminismTester$.main(DeterminismTester.scala:34)
	at scala.tools.nsc.DeterminismTester.main(DeterminismTester.scala)

Later, the namer enters a source file loader for scala.Predef, running through this code:

def enterClassSymbol(tree: ClassDef): Symbol = {
      val existing = context.scope.lookup(tree.name)
      val isRedefinition = (
           existing.isType
        && existing.isTopLevel
        && context.scope == existing.owner.info.decls
        && currentRun.canRedefine(existing)
      )
      val clazz: Symbol = {
        if (isRedefinition) {
          updatePosFlags(existing, tree.pos, tree.mods.flags)
          setPrivateWithin(tree, existing)
          clearRenamedCaseAccessors(existing)
          existing
        }
        else enterInScope(assignMemberSymbol(tree)) setFlag inConstructorFlag
      }
      clazz match {
        case csym: ClassSymbol if csym.isTopLevel => enterClassSymbol(tree, csym)
        case _                                    => clazz
      }
    }

updatePosFlags resets the existing symbolOf[Predef.type] to represent the source file. But it seems we've already performed some typechecking sees (potentially stale) information from Predef.class. The error mode here isn't related to stateness, but rather to having two distinct identical symbols for the same entity, a detail which shows up in the number of entries of in the pickle.

We should see if this problem generalizes to user code (that isn't redefining Predef from source). Even if the problem is ours alone, a fix is desirable as it would let us use src/library as a regression test for determism without needed to doctor to the test to pass Predef.scala first on the source file list or somesuch.

Activity

retronym
added
compiler determinism
on Jul 17, 2020
retronym
self-assigned this
on Jul 17, 2020
retronym
added this to the 2.12.13 milestone on Jul 17, 2020
som-snytt

som-snytt commented on Jul 17, 2020

@som-snytt
som-snytt
on Jul 17, 2020

I wonder if 😕 plus ❤️ adequately conveys the feeling, in the absence of 😿 .

retronym

retronym commented on Jul 18, 2020

@retronym
retronym
on Jul 18, 2020 via email
MemberAuthor
retronym

retronym commented on Jul 20, 2020

@retronym
retronym
on Jul 20, 2020 · edited by retronym
MemberAuthor

Here's a variant in userland, where the package object does not participate in root imports.

  @Test def testPackageObjectUserLand(): Unit = {
    def code = List[SourceFile](
      source("package.scala", "package userland; object `package` { type Throwy = java.lang.Throwable }"),
      source("th.scala", "package userland; class th[T <: Throwy](cause: T = null)")
    )
    test(code :: Nil, permuter = _.reverse :: Nil)
  }

When th.scala appears before package.scala, naming the PackageDef in th.scala with:

    def enterPackage(tree: PackageDef) {
      val sym = createPackageSymbol(tree.pos, tree.pid)
      tree.symbol = sym
      newNamer(context.make(tree, sym.moduleClass, sym.info.decls)) enterSyms tree.stats
    }

... forces the info of the the package class symbol, including entering the members of the package.class. Again, we end up with two symbols representing the same entity which results in a different (albeit semantically equivalent) pickle. I'm pretty sure there is also a correctness bug here in that references to members of package.class that no longer in package.scala might not result in "no such symbol" errors.

retronym
mentioned this on Jul 22, 2020
dwijnand
modified the milestones: 2.12.13, 2.12.14 on Oct 20, 2020
dwijnand

dwijnand commented on Oct 20, 2020

@dwijnand
dwijnand
on Oct 20, 2020
Member

Pushed as I assumed you wouldn't work on this in the next 10 days...

SethTisue
modified the milestones: 2.12.14, Backlog on May 18, 2021
retronym

retronym commented on Jun 10, 2021

@retronym
retronym
on Jun 10, 2021
MemberAuthor

Possibly helpful is this idea for lazier package object loading: scala/scala#9664

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

compiler determinism

Type

No type

Projects

No projects

Milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @retronym@SethTisue@dwijnand@som-snytt

      Issue actions
        Unstable compilation output of src/library due to stray references to Predef.class before Predef.scala is named · Issue #12086 · scala/bug