Support Spark 3.3/JDK8

[xumingming]

We are currently using Spark 3.3/JDK8 in production, don't whether is it possible to support Spark 3.3/JDK8 in comet?

[mbutrovich]

Hi @xumingming! We chose to drop JDK 8 support to keep up with a more recent release of Arrow Java, among other reasons. You can see the discussion at the time here:
#1775

[coderfender]

@xumingming thank you . We are always looking for new contributors to the comet project and there are a bunch of 'good first issues' I can guide you to work on.

[mbutrovich]

@andygrove @mbutrovich one more question, for "shift resources particularly for 3.5 and 4.0 support", the "resource" here means "developing/developer resource" right? If that is case, if I'd like to support spark 3.3 related developing/reviewing staff(After a glance at #1529, seems the main code is not too much), is it possible to add it back to Comet master?

The resource is not only developers/reviewers, but also CI resources. Our CI matrix is already quite large, and running Spark's test suites for 3.4, 3.5. and 4.0 is typically the bottleneck.

The difference in Spark behavior between 3.3 and 4.0 (which we'd like to start treating as a primary target for Comet) is also such that maintaining that many shims and conditional codepaths is concerning to me.

Lastly, would you require JDK8 support or are you just asking about 3.3 with JDK 11+? AFAIK Comet cannot support JDK8 anymore as Arrow Java has dropped support for it, and we'd have to backport critical behavior and bug fixes to an older fork of Arrow Java, which seems like a non-starter to me.

[andygrove]

@xumingming Unfortunately, I have to agree with @mbutrovich that adding support for Spark 3.3 at this point, given the current number of maintainers, just isn't realistic. If we had more maintainers, then I think it could make sense. Have you tested with a version of Comet that does support 3.3? I would suggest starting there to see if Comet is even viable for you.

Adding support for JDK 8 is not possible because the Java implementation of Apache Arrow no longer supports it.

Out of interest, what are the main issues preventing you from upgrading to Spark 3.4 or later?

[andygrove]

It is also worth noting that Spark 3.3 reached EOL more than 2 years ago, and Spark 3.5 reaches EOL in about 3 months. I would suggest upgrading to pick up important bug fixes and performance improvements.

[xumingming]

@andygrove the main issues for upgrading Spark is that we are not justing using community Spark 3.3 directly, we have made many changes to support customized SQL grammar(to be compatible with another engine), to support customized data source etc. The upgrading itself involves a lot of coordination and work. Your advice of trying an old version of Comet that support Spark 3.3 is inspiring and I will try it to see if Comet gives a good perf result.

@mbutrovich @andygrove I get that supporting JDK8 is a not easy thing and seems impossible to maintain for Comet. Adding support for Spark 3.3 + JDK11 alone is also desirable and would help a lot.