sbt bootstrap Part 2: Revenge of the sbt bootstrap #2312

smarter · 2017-04-26T17:33:19Z

This is a major overhaul of how the sbt bootstrap works, the main goal is to be able to publish a usable bootstrapped compiler, this mean having artefacts with names like dotty-*_0.1 instead of dotty-*-bootstrapped_2.11, this is important to be able to cross-compile with dotty like any other Scala version (I will rewrite sbt-dotty to take advantage of this soon. The other important change in this PR is that the non-bootstrapped compiler no longer needs to be published to compile the bootstrapped compiler, only dotty-sbt-bridge needs to be published (I haven't investigated whether this is something we could avoid too). It is still possible to bootstrap from published artefacts using the bootstrapFromPublishedJars setting I added, see the commit messages.

felixmulder

LGTM,

I have some reservations with merging this however. If I've understood correctly this will make the distributed version of Dotty bootstrapped as well.

Has the infinite loop issue been resolved for the REPL in this case? If not, I would prefer if we fix that first - otherwise people who try out Dotty will be met with a REPL that does not work. Not very good...

smarter · 2017-04-27T09:45:23Z

Has the infinite loop issue been resolved for the REPL in this case? If not, I would prefer if we fix that first - otherwise people who try out Dotty will be met with a REPL that does not work. Not very good...

I've asked @OlivierBlanvillain to look at it ;).

sjrd

Looks nice overall. Just a couple minor details.

sjrd · 2017-04-27T15:47:04Z

build.sbt

@@ -16,5 +16,3 @@ val `scala-library` = Build.`scala-library`
 val `scala-compiler` = Build.`scala-compiler`
 val `scala-reflect` = Build.`scala-reflect`
 val scalap = Build.scalap
-
-inThisBuild(Build.thisBuildSettings)


This appears to be in the wrong commit.

Good catch, will fix

sjrd · 2017-04-27T16:05:30Z

project/Build.scala

    settings(commonBootstrappedSettings).
    settings(
-      publishArtifact := false
+      moduleName := "dotty"


You can automate this by having the following in commonBootstrappedSettings:

moduleName ~= { _.stripSuffix("-bootstrapped") }

sjrd · 2017-04-27T16:14:02Z

sbt-bridge/test/xsbt/TestDriver.scala

+import dotty.tools.dotc._
+import core.Contexts._
+
+class TestDriver extends Driver {


Do you really need to make this public? Can't in private inside the companion object of ScalaCompilerForUnitTesting?

Sure, but it's a test so it doesn't matter much, and I'd like to avoid changing ScalaCompilerForUnitTesting too much since it's adapted from sbt and I synchronize it with the sbt version sometimes.

smarter · 2017-04-30T20:28:20Z

I've opened #2335 to track the bootstrapped REPL issue.

…ettings No need to have multiple ways to define settings common to all projects, and thisBuildSettings doesn't work for keys which are defined in every project but not globally.

By default, sbt will try to rewrite dependencies on scala-{library,compiler,reflect} and scalap artifacts. This can have very weird consequences for us since we define our own versions of these projects.

Not having publishSettings set does not prevent publishing, we should set publishArtifact := false on such projects instead.

It is automatically set correctly for dotty by sbt 0.13.15

The regular dotty-sbt-bridge depends on `dotty-compiler` but we'd like to be able to publish only bootstrapped artifacts without depending on any non-bootstrapped artifacts.

There are four kinds of projects in our build: - Projects compiled only with Scala 2 (dotty-bot, as well as dotty-interfaces since it's Java-only) - Projects that are part of dotty and compiled with Scala 2 - Projects that are part of dotty and compiled with a non-bootstrapped dotty - Dummy projects for scala{-library,-compiler,-reflect,p}, see the comment above `commonDummySettings` They are now distinguished using four sets of settings. The first and second set are identical but this will change with the next commit.

This make them easier to distinguish and harder to accidentally use. This also means that in the future we could compile dotty with `crossPath := false`, but this will require changing the hardcoded dotty dependencies in sbt.

- The bootstrapped dotty artifacts are now named dotty-*_0.1 instead of dotty-*-bootstrapped_2.11, which means they can be used in sbt projects. - The dummy scala{-library,-reflect,-compiler,p} aritfacts are now made to work with the bootstrapped dotty. - Compiling the bootstrapped dotty no longer requires publishing the non-bootstrapped dotty, only dotty-sbt-bridge needs to be published, so you can do: ;dotty-sbt-bridge/publishLocal;dotty-bootstrapped/test

Since the previous commit, we bootstrap dotty from the current build output of the non-bootstrapped compiler, but when hacking on the compiler it might be useful to bootstrap from a stable artefact, this is now possible by setting `bootstrapFromPublishedJars` to true. Usage: \# Publish the non-bootstrapped compiler > dotty/publishLocal \# Enable the option > set bootstrapFromPublishedJars in ThisBuild := true \# Bootstrap the compiler > ;dotty-bootstrapped/clean;dotty-bootstrapped/compile

Otherwise the test won't compile with dotty.

Dotty crashes when trying to compile them currently.

Also move them to their own project `dotty-sbt-scripted-tests`, because they need to be in a Scala 2.10 project to work correctly, since sbt uses 2.10. The scripted tests can now be run with: > dotty-sbt-scripted-tests/scripted A few of them are broken right now, I'll fix them later (we don't run them on the CI yet).

This is needed so that dotty-bootstrapped/publishLocal publishes everything needed to use dotty.

These `publishLocal` calls should be run _before_ `scriptedTask`, but before this commit they were run in parallel, leading to horrible failures.

Before this commit, `dotty-sbt-bridge` had to be published locally to be able to compile bootstrapped projects. This is no longer the case, instead the local `dotty-sbt-bridge` is used, this also requires calling `cleanSbtBridge` before packaging `dotty-sbt-bridge` instead of before publishing it.

smarter · 2017-05-01T18:13:55Z

I managed to get the bootstrapped dotty compilable without publishing anything! See last commit. I'll merge this soon now that the REPL is fixed because this is a blocker for other things, but I'll still take into account further comments of course.

smarter · 2017-05-01T20:01:59Z

@DarkDimius @felixmulder I've updated our Jenkins nightly build to run dotty-bootstrapped/publishSigned and successfully ran it once.

Dotty is now published with scalaBinaryVersion 0.1, see scala/scala3#2312

smarter force-pushed the publish-bootstrap branch 2 times, most recently from a3e0d36 to 0bed0bd Compare April 26, 2017 17:35

smarter requested review from sjrd and felixmulder April 26, 2017 17:55

felixmulder approved these changes Apr 27, 2017

View reviewed changes

smarter force-pushed the publish-bootstrap branch from 0bed0bd to a2f3b9e Compare April 27, 2017 15:56

sjrd approved these changes Apr 27, 2017

View reviewed changes

smarter force-pushed the publish-bootstrap branch from a2f3b9e to b84e4f2 Compare April 27, 2017 19:10

smarter mentioned this pull request Apr 28, 2017

Major rework of the sbt-dotty plugin for cross-compilation support lampepfl/sbt-dotty#4

Merged

smarter force-pushed the publish-bootstrap branch from b84e4f2 to d9157b6 Compare April 28, 2017 12:27

smarter added 17 commits May 1, 2017 17:38

Build.scala: Replace thisBuildSettings and sourceStructure by commonS…

168efa7

…ettings No need to have multiple ways to define settings common to all projects, and thisBuildSettings doesn't work for keys which are defined in every project but not globally.

Prevent sbt from rewriting our dependencies

eaedfa7

By default, sbt will try to rewrite dependencies on scala-{library,compiler,reflect} and scalap artifacts. This can have very weird consequences for us since we define our own versions of these projects.

Add publishSettings to commonSettings

991b198

Not having publishSettings set does not prevent publishing, we should set publishArtifact := false on such projects instead.

Remove scalaCompilerBridgeSource setting

693f3f7

It is automatically set correctly for dotty by sbt 0.13.15

Add dotty-sbt-bridge-bootstrapped project

8d4015a

The regular dotty-sbt-bridge depends on `dotty-compiler` but we'd like to be able to publish only bootstrapped artifacts without depending on any non-bootstrapped artifacts.

ScalaCompilerForUnitTesting: do not use structural types

c0da276

Otherwise the test won't compile with dotty.

dotty-sbt-bridge-bootstrapped: Disable tests

03fa3d5

Dotty crashes when trying to compile them currently.

Make dotty-boostrapped aggregate dotty-interfaces

86d6dfa

This is needed so that dotty-bootstrapped/publishLocal publishes everything needed to use dotty.

Fix the source-dependencies/binary test

72230c8

Fix the compilerReporter/simple test

b822cb0

Fix race condition in scripted

6e28fee

These `publishLocal` calls should be run _before_ `scriptedTask`, but before this commit they were run in parallel, leading to horrible failures.

smarter force-pushed the publish-bootstrap branch from d9157b6 to de5d34a Compare May 1, 2017 18:12

smarter merged commit b19d1fb into scala:master May 1, 2017

smarter added a commit to scala/scala3-example-project that referenced this pull request May 1, 2017

Upgrade to bootstrapped dotty

f5f2593

Dotty is now published with scalaBinaryVersion 0.1, see scala/scala3#2312

allanrenucci deleted the publish-bootstrap branch December 14, 2017 19:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

sbt bootstrap Part 2: Revenge of the sbt bootstrap #2312

sbt bootstrap Part 2: Revenge of the sbt bootstrap #2312

Uh oh!

smarter commented Apr 26, 2017

Uh oh!

felixmulder left a comment

Uh oh!

smarter commented Apr 27, 2017

Uh oh!

sjrd left a comment

Uh oh!

sjrd Apr 27, 2017

Uh oh!

smarter Apr 27, 2017

Uh oh!

sjrd Apr 27, 2017

Uh oh!

smarter Apr 27, 2017

Uh oh!

sjrd Apr 27, 2017

Uh oh!

smarter Apr 27, 2017

Uh oh!

smarter commented Apr 30, 2017

Uh oh!

smarter commented May 1, 2017

Uh oh!

smarter commented May 1, 2017

Uh oh!

Uh oh!

sbt bootstrap Part 2: Revenge of the sbt bootstrap #2312

sbt bootstrap Part 2: Revenge of the sbt bootstrap #2312

Uh oh!

Conversation

smarter commented Apr 26, 2017

Uh oh!

felixmulder left a comment

Choose a reason for hiding this comment

Uh oh!

smarter commented Apr 27, 2017

Uh oh!

sjrd left a comment

Choose a reason for hiding this comment

Uh oh!

sjrd Apr 27, 2017

Choose a reason for hiding this comment

Uh oh!

smarter Apr 27, 2017

Choose a reason for hiding this comment

Uh oh!

sjrd Apr 27, 2017

Choose a reason for hiding this comment

Uh oh!

smarter Apr 27, 2017

Choose a reason for hiding this comment

Uh oh!

sjrd Apr 27, 2017

Choose a reason for hiding this comment

Uh oh!

smarter Apr 27, 2017

Choose a reason for hiding this comment

Uh oh!

smarter commented Apr 30, 2017

Uh oh!

smarter commented May 1, 2017

Uh oh!

smarter commented May 1, 2017

Uh oh!

Uh oh!