Plugin Direction

[ajoberstar]

Recently, @cmf, @jszakmeister, and @ajoberstar had a call with @eriwen from Gradle regarding language plugin development.

Following are my notes and open questions about the next steps.

Audience

Clojure

2016 State of Clojure has some data useful to what we would target as baselines. See the Cognitect blog and full results. Note that most of these questions allowed multiple answers, so they add up to more than 100%. There's obviously also some self-selection bias in the survey.

• Clojure:
  • 33% were already targeting pre-releases of 1.9
  • 83% were targeting 1.8
  • No more than 17% target 1.7 or lower
• Java:
  • 95% target Java 8
  • No higher than 15% target 1.7 or lower
• Gradle is only used by 2% of respondents

Gradle

On the call, Gradle indicated:

• Using Java 8 as a baseline is pretty common nowadays
• Many plugins will be moving to a Gradle 4.2 baseline due to the new plugin-oriented features coming out
• Additionally the vast majority of their downloads (I think he mentioned 70%) where Gradle 3.3 (released in January 2017) or higher, so users are probably willing to move to newer versions without huge trouble.
• While we should minimize internal API usage, we may be stuck using some until they've built out more of the plugin-oriented public API.

Potential Users

• Existing Gradle users who want to start using Clojure or standardize their existing Clojure builds onto Gradle.
• Existing Lein or Boot users who:
  • Want to build projects using multiple JVM languages (most likely Java + Clojure)
  • Are just generally curious about Gradle

Compelling features gradle-clojure and Gradle itself provide:

• Easier polyglot development (wouldn't be limited to just Java + Clojure)
• Easier development of multiple projects (both Gradle's multi-project and composite builds features are helpful here)
• Access to Gradle's plugin ecosystem

Expectations Clojure users will have:

• Easy availability of REPL
• Figwheel or something like that for CLJS development
• Publishing to and getting dependencies from Clojars
• Running Clojure-based test frameworks (clojure.test, test.check, etc)
• Uberjar support (generally leveraging Gradle's shadow plugin instead of building ourselves)

Expectations Gradle users will have:

• Clojure support feeling native (i.e. leveraging existing Gradle features as much as possible)
• Lazy/reactive evaluation of build logic
• Incremental builds -- not necessarily incremental compilation, but running a build twice should result in "UP-TO-DATE" messages and no actions taken by the build
• Plugin structure (base layer vs opinionated/conventional layer)
• Dedicated task classes (likely extending relevant abstract types in Gradle)
• Test integration that leverages test reporting/filtering/etc

Potential Contributors

• People familiar with Gradle would likely have backgrounds in Java (and possibly Groovy). There will be growth of Kotlin in the Gradle ecosystem (particularly with the availability of the Kotlin DSL), but not sure how much there is right now.
• People familiar with Clojure would likely also have familiarity with Java.

Next steps

A bunch of them are logistical.

• Need to decide on a home for the plugin:
  • Reusing gradle-clojure (either under the cursive org or its own) -- gradle-clojure is closest to what we'll need to implement going forward and is the only current Gradle Clojure plugin
  • New repo in its own org -- Provides a clean start, might be useful if the existing gradle-clojure will need to be maintained for a while (maybe for Colin's purposes with Cursive development).
  • Graclj is pretty useless given the direction away from the software model. I have no concerns about abandoning this.
• Need to decide a Java baseline
• Need to decide a Gradle baseline
• Need to decide what language the plugin will be implemented in
  • gradle-clojure is currently Kotlin
• Need to decide what language tests will be written in:
  • If we did Java impl, we might want something less verbose for tests (maybe Clojure)
  • Regardless of language the Gradle TestKit will be a major part of the testing and verifying compatibility across Gradle versions
• Need to decide scope of first release
  • Might be just applying Gradle idioms to gradle-clojure with the current feature set

[ajoberstar]

My votes on some of these questions:

• Java 8 baseline
• Gradle 4.2 baseline (with a willingness to go higher if more helpful plugin APIs are introduced, I'd rather not work around deficiencies in Gradle in the case of backwards compatibility, particularly until there's a significant userbase)
• Clojure 1.8+ target (though I expect it will be pretty easy to support older versions)
• Java implementation -- While I'm interested in learning Kotlin (so I'm OK if the water runs that way), I think it will be easier to get contributions to a Java-based plugin from Clojure users than Kotlin.

[aadrian]

Need to decide a Java baseline

+1 for Java 8

Need to decide a Gradle baseline

+1 for Gradle 4.2

Need to decide what language the plugin will be implemented in

Groovy ?

[ajoberstar]

@aadrian Groovy has the advantage of being more common in Gradle plugins (and allowing some dynamic behavior around API compatibility between Gradle versions) but similar to Kotlin may be less familiar to Clojure programmers. Similar to Kotlin, I'm also OK if Groovy is the direction. Personally, with Lambdas and Streams in Java 8, I've had little need for anything Groovy provides recently.

[aadrian]

@ajoberstar What about using all of the above languages? Since the main purpose of the plug-in is to easily allow many languages in one project that has Clojure :) .

[ajoberstar]

@aadrian We could certainly emphasize the polyglot possibilities that way 😄.

While supporting that kind of development is a goal, it's not the primary one (for me, personally). I would expect most people won't have more than a two language mix, and I hope that we can provide something compelling enough to be considered for Clojure-only development too. I think it just reduces the barrier to entry for someone reading or contributing to the plugin source to reduce the number of languages in play.

[cursive-ghost]

I agree that Java 8 is a fine baseline - even if companies aren't all using it in prod, it's less of an ask to expect them to upgrade their build environments.

I suspect Gradle 4.2. is probably more of an issue, but I have no idea really. Perhaps @pmoriarty or @mbossenbroek have more feedback.

As for language, I'm not keen on Groovy. gradle-clojure contains one Groovy file, but that's just to do API switching based on the Gradle version - that has to be in a dynamic language, and Groovy is already available in Gradle so I used it there. This seems to be the approach recommended by the Gradle team for handling API differences across Gradle versions. I'm fine with Java if Kotlin isn't a popular choice - it is the only language other than Groovy which is officially supported for build files though, so I suspect it will become more popular. Reducing friction for contributions is important though.

I think that migrating the plugin to a gradle-clojure org here on Github is probably the best move. Realistically I don't have a lot of bandwidth for this, the current plugin works for my needs so while I'm happy to be involved in discussion I may not have much time for actually working on it. I'd like to see this being a community effort, I feel bad that @pbzdyl has had to wait so long for his PRs to be merged.

Other misc notes from the meeting:

• Just to be clear, the software model approach is not going anywhere, which is what graclj is based on. However the Gradle team did suggest using some of the details of how it is implemented, see below.
• In terms of how to deal with polyglot projects, we should take the simplest approach to begin with. We'll need to look at Added support for multiple source sets (issue #11) and basic support for Gradle incremental tasks (issue #10) #12 to see what to take from it - definitely multiple source sets, definitely compiling Clojure to a separate directory, definitely not different source sets per language. If users need some more complexity in their build files (i.e. to specify if their compilation order for multiple languages) that's ok for now, they can do that to begin with by ordering tasks in their afterEvaluate block.
• Console REPL support is basically impossible in Gradle as it stands because of the daemon. Users will have to start a REPL headlessly and then connect to it somehow. We could provide a task which starts a REPL and perhaps a small command-line task which connects REPLy to it somehow?

These were the recommendations from the Gradle team:

• Consider reacting to java plugin applied instead of applying manually
• Create a task of type JavaCompile and configure with some default settings
• Setup a new ClojureCompile task like this
• Setup a ClojureSourceSet
• Separate ClojureScriptPlugin, perhaps building on top of ClojurePlugin

I don't fully understand what is meant by all of those, but I haven't looked at the code in detail since the meeting - we may have to go back to them for more clarification. Their recommended sources of inspiration are https://github.com/gradle-guides/gradle-site-plugin for general plugin best practices, and possibly https://github.com/golo-lang/gradle-golo-plugin for some inspiration on the JVM language front.

In terms of a first release, I'd suggest:

• Starting a gradle-clojure org, and copying gradle-site-plugin there and deleting all the code (@eriwen suggested this as a starting point).
• Bring the code from gradle-clojure over, either as-is if we decide to stick with Kotlin, or porting it to Java if we decide to do that.
• Possibly bringing in the changes that we're clear we want from Added support for multiple source sets (issue #11) and basic support for Gradle incremental tasks (issue #10) #12.
• Possibly writing some tests to get a bit more confidence in the existing code.

That's still quite a lot of work, so I don't think we should do much more for a v1.

If this is going to be under its own org, I'd like this to be a community effort but I think it's probably a good idea to have a nominated maintainer even if we have several collaborators. If he's up for it I'd like to nominate @ajoberstar since it seems he has by far the best knowledge of Gradle of any of us.

[ajoberstar]

If no one else had their eyes set on it, I'm willing to be the maintainer. I agree about wanting it to be a community effort; even if people don't have a lot of time to write code, it would be helpful to have a few willing to test and provide feedback on feature direction, etc.

[ajoberstar]

Naming question... Right now the Gradle plugin's ID is "cursive.clojure". Not sure if @cursive-ide had a preference, but maybe we want to change the name anyway.

The main downside to "gradle-clojure" is translating it into a plugin ID (which has to be 2+ segments). Best option there would probably be "io.github.gradle-clojure".

I do still have a "graclj" organization and domain (that I would otherwise jut let expire), so I'll offer that as an option too.

[pmoriarty]

We would be happy to move to Gradle 4.2 (once it's released) for this.

[aadrian]

Naming question... Right now the Gradle plugin's ID is "cursive.clojure"
The main downside to "gradle-clojure" is translating it into a plugin ID (which has to be 2+ segments). Best option there would probably be "io.github.gradle-clojure".

@ajoberstar why not just clojure and clojurescript as IDs ?

[cursive-ghost]

why not just clojure and clojurescript as IDs ?

Because the names need to be 2+ segments, i.e. something.something.

I think "io.github.gradle-clojure" is fine, or perhaps "gradle-clojure.gradle-clojure"? I don't think it matters too much, as long as it's likely to be unique.

I prefer gradle-clojure to graclj just because I know how to pronounce it :-), but I'm not bothered if everyone wants to use graclj.

[jszakmeister]

My votes on some of these questions:

Java 8 baseline
Gradle 4.2 baseline (with a willingness to go higher if more helpful plugin APIs are introduced, I'd rather not work around deficiencies in Gradle in the case of backwards compatibility, particularly until there's a significant userbase)
Clojure 1.8+ target (though I expect it will be pretty easy to support older versions)
Java implementation -- While I'm interested in learning Kotlin (so I'm OK if the water runs that way), I think it will be easier to get contributions to a Java-based plugin from Clojure users than Kotlin.

/me agrees. And io.github.gradle-clojure sounds like a good idea to me too.

[pbzdyl]

I am very happy to see more interest in Gradle Clojure plugin. I understand that it is not a top priority for @cursive-ide and thus we are maintaining our fork with PR #12 integrated.

I am not a Gradle expert but I would like to offer my help and I would more than happy if features from #12 become incorporated into the new plugin (with code from the PR or a new implementation). Support for incremental builds and multiple source sets are the top priority for me.

As for the ClojureScript plugin, I would suggest to take a look at how Duck framework integrates Figwheel as part of the application code (as an Integrant component). I think we could use the same way for supporting figwheel in Gradle ClojureScript plugin. Support for simple ClojureScript compilation would be even easier as it would require just calling the CLJS Build API.

[ajoberstar]

I think "io.github.gradle-clojure" is fine, or perhaps "gradle-clojure.gradle-clojure"?

"gradle-clojure.clojure" and "gradle-clojure.clojurescript"?

[ajoberstar]

I am not a Gradle expert but I would like to offer my help and I would more than happy if features from #12 become incorporated into the new plugin (with code from the PR or a new implementation). Support for incremental builds and multiple source sets are the top priority for me.

Glad to have you involved. Haven't looked through that PR yet, but I'm guessing it would make sense to get that integrated before anything else.

As for the ClojureScript plugin, I would suggest to take a look at how Duck framework integrates Figwheel as part of the application code (as an Integrant component). I think we could use the same way for supporting figwheel in Gradle ClojureScript plugin. Support for simple ClojureScript compilation would be even easier as it would require just calling the CLJS Build API.

Yeah, this looks promising. Will be nice to base this off other usages of figwheel to make sure we provide similar support.