[jvm-packages] Support spark connect

[wbo4958]

This PR is to make xgboost jvm package run on spark connect. Currently, I put the xgboost4j python wrapper into jvm-packages/xgboost4j-spark/python, but I can change it to other place. @WeichenXu123 @trivialfis let's discuss it on this thread.

To Do,

• add read/write
• add XGBoostRegressor
• add XGBoostRanker

[trivialfis]

I think the Python code should be part of the Python package, we can reuse a lot of code there and it also helps unify the interfaces. In addition, Python packaging is not trivial, @hcho3 has been working on related projects and I can feel the difficulty there. Let's not do it for another package.

[trivialfis]

These can be removed, we dropped suport for all previous versions in 3.0

[jjayadeep06]

Will this be a new class that needs to be used for both normal and spark connect invocation ? Can we not modify _fit method in SparkXGBClassifier to use try_remote_fit decorator or will it be a big change ?

[wbo4958]

Thx @jjayadeep06 for your reply. Well, when involving connect, things is becoming complicated. You know, we could make the existing xgboost-pyspark support spark connect by changing the RDD operations to Dataframe without using any try_remote_xxxx. Yes, we have a plan to do that.

While this PR is to make xgboost jvm package to support connect by introducing a light-weight python wrapper. If we add the python wrapper over xgboost jvm package to the existing xgboost-python-pyspark, then it's going to raise an issue which backends (xgboost jvm package or python package) will be chose when running xgboost over connect?

[jjayadeep06]

Got It

[wbo4958]

Hi @jjayadeep06, @WeichenXu123, @trivialfis, I propose integrating the python wrapper over xgboost jvm package into the xgboost python package as an optional module, placed in a subdirectory such as xgboost/jvm/ or xgboost/4j where the estimators will reside.

This would allow users to install jvm wrapper independently using the following command without requiring the full xgboost package.

pip install xgboost[4j]

[trivialfis]

I don't think it's easy to merge the package into the existing one but split it up during deployment. If it's merged, it's first class. You can put an experimental flag if you want.

I just want to say the design of the connect ML sets a very high bar for anyone who wants to dig in. One needs to know four different programming languages and some familiarity with spark connect ML to poke around, debugging native code is another level. I'm curious what leads to this design.

Lastly, we need CI.

[jjayadeep06]

Is the thought here that when someone wants to use spark connect then they can install only the python jvm wrapper instead of entire xgboost package? If this is the case then we can use the existing python folder structure with optional dependencies wherein if we do pip install xgboost it will install the current xgboost code and if we do pip install xgboost[connect] then it will install the jvm only python wrapper ?

[trivialfis]

The XGBoost JVM package still needs to live on the server side with a matching version.

and if we do pip install xgboost[connect] then it will install the jvm only python wrapper ?

Based on my understanding of Python wheel packaging, xgboost[connect] means xgboost + connect specific dependencies. Like xgboost[dask] will install xgboost in addition to all dask-related dependencies.

[jjayadeep06]

The XGBoost JVM package still needs to live on the server side with a matching version.

In Spark Connect the server is always a JVM server so i don't think we need the python jvm package on the server side. Also, this PR is using the standard scala/java APIs which are used by other SparkML models and therefore there is no additional deployment needed. @wbo4958 - please correct me.

and if we do pip install xgboost[connect] then it will install the jvm only python wrapper ?
Based on my understanding of Python wheel packaging, xgboost[connect] means xgboost + connect specific dependencies. Like xgboost[dask] will install xgboost in addition to all dask-related dependencies.

You are correct it will install the additional dependencies connect along with the core dependency xgboost. If we want to have two separate packaging structure then we might have to alter the project structure.