Skip to content

Commit a3e51cc

Browse files
Brennon Yorkpwendell
Brennon York
authored and
pwendell
committed
[SPARK-4501][Core] - Create build/mvn to automatically download maven/zinc/scalac
Creates a top level directory script (as `build/mvn`) to automatically download zinc and the specific version of scala used to easily build spark. This will also download and install maven if the user doesn't already have it and all packages are hosted under the `build/` directory. Tested on both Linux and OSX OS's and both work. All commands pass through to the maven binary so it acts exactly as a traditional maven call would. Author: Brennon York <[email protected]> Closes apache#3707 from brennonyork/SPARK-4501 and squashes the following commits: 0e5a0e4 [Brennon York] minor incorrect doc verbage (with -> this) 9b79e38 [Brennon York] fixed merge conflicts with dev/run-tests, properly quoted args in sbt/sbt, fixed bug where relative paths would fail if passed in from build/mvn d2d41b6 [Brennon York] added blurb about leverging zinc with build/mvn b979c58 [Brennon York] updated the merge conflict c5634de [Brennon York] updated documentation to overview build/mvn, updated all points where sbt/sbt was referenced with build/sbt b8437ba [Brennon York] set progress bars for curl and wget when not run on jenkins, no progress bar when run on jenkins, moved sbt script to build/sbt, wrote stub and warning under sbt/sbt which calls build/sbt, modified build/sbt to use the correct directory, fixed bug in build/sbt-launch-lib.bash to correctly pull the sbt version be11317 [Brennon York] added switch to silence download progress only if AMPLAB_JENKINS is set 28d0a99 [Brennon York] updated to remove the python dependency, uses grep instead 7e785a6 [Brennon York] added silent and quiet flags to curl and wget respectively, added single echo output to denote start of a download if download is needed 14a5da0 [Brennon York] removed unnecessary zinc output on startup 1af4a94 [Brennon York] fixed bug with uppercase vs lowercase variable 3e8b9b3 [Brennon York] updated to properly only restart zinc if it was freshly installed a680d12 [Brennon York] Added comments to functions and tested various mvn calls bb8cc9d [Brennon York] removed package files ef017e6 [Brennon York] removed OS complexities, setup generic install_app call, removed extra file complexities, removed help, removed forced install (defaults now), removed double-dash from cli 07bf018 [Brennon York] Updated to specifically handle pulling down the correct scala version f914dea [Brennon York] Beginning final portions of localized scala home 69c4e44 [Brennon York] working linux and osx installers for purely local mvn build 4a1609c [Brennon York] finalizing working linux install for maven to local ./build/apache-maven folder cbfcc68 [Brennon York] Changed the default sbt/sbt to build/sbt and added a build/mvn which will automatically download, install, and execute maven with zinc for easier build capability
1 parent 080ceb7 commit a3e51cc

File tree

16 files changed

+330
-166
lines changed

16 files changed

+330
-166
lines changed

‎.gitignore

+5-2
Original file line numberDiff line numberDiff line change
@@ -8,16 +8,19 @@
88
*.pyc
99
.idea/
1010
.idea_modules/
11-
sbt/*.jar
11+
build/*.jar
1212
.settings
1313
.cache
14+
cache
1415
.generated-mima*
15-
/build/
1616
work/
1717
out/
1818
.DS_Store
1919
third_party/libmesos.so
2020
third_party/libmesos.dylib
21+
build/apache-maven*
22+
build/zinc*
23+
build/scala*
2124
conf/java-opts
2225
conf/*.sh
2326
conf/*.cmd

‎build/mvn

+132
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
#!/usr/bin/env bash
2+
3+
# Determine the current working directory
4+
_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
5+
# Preserve the calling directory
6+
_CALLING_DIR="$(pwd)"
7+
8+
# Installs any application tarball given a URL, the expected tarball name,
9+
# and, optionally, a checkable binary path to determine if the binary has
10+
# already been installed
11+
## Arg1 - URL
12+
## Arg2 - Tarball Name
13+
## Arg3 - Checkable Binary
14+
install_app() {
15+
local remote_tarball="$1/$2"
16+
local local_tarball="${_DIR}/$2"
17+
local binary="${_DIR}/$3"
18+
19+
# setup `curl` and `wget` silent options if we're running on Jenkins
20+
local curl_opts=""
21+
local wget_opts=""
22+
if [ -n "$AMPLAB_JENKINS" ]; then
23+
curl_opts="-s"
24+
wget_opts="--quiet"
25+
else
26+
curl_opts="--progress-bar"
27+
wget_opts="--progress=bar:force"
28+
fi
29+
30+
if [ -z "$3" -o ! -f "$binary" ]; then
31+
# check if we already have the tarball
32+
# check if we have curl installed
33+
# download application
34+
[ ! -f "${local_tarball}" ] && [ -n "`which curl 2>/dev/null`" ] && \
35+
echo "exec: curl ${curl_opts} ${remote_tarball}" && \
36+
curl ${curl_opts} "${remote_tarball}" > "${local_tarball}"
37+
# if the file still doesn't exist, lets try `wget` and cross our fingers
38+
[ ! -f "${local_tarball}" ] && [ -n "`which wget 2>/dev/null`" ] && \
39+
echo "exec: wget ${wget_opts} ${remote_tarball}" && \
40+
wget ${wget_opts} -O "${local_tarball}" "${remote_tarball}"
41+
# if both were unsuccessful, exit
42+
[ ! -f "${local_tarball}" ] && \
43+
echo -n "ERROR: Cannot download $2 with cURL or wget; " && \
44+
echo "please install manually and try again." && \
45+
exit 2
46+
cd "${_DIR}" && tar -xzf "$2"
47+
rm -rf "$local_tarball"
48+
fi
49+
}
50+
51+
# Install maven under the build/ folder
52+
install_mvn() {
53+
install_app \
54+
"http://apache.claz.org/maven/maven-3/3.2.3/binaries" \
55+
"apache-maven-3.2.3-bin.tar.gz" \
56+
"apache-maven-3.2.3/bin/mvn"
57+
MVN_BIN="${_DIR}/apache-maven-3.2.3/bin/mvn"
58+
}
59+
60+
# Install zinc under the build/ folder
61+
install_zinc() {
62+
local zinc_path="zinc-0.3.5.3/bin/zinc"
63+
[ ! -f "${zinc_path}" ] && ZINC_INSTALL_FLAG=1
64+
install_app \
65+
"http://downloads.typesafe.com/zinc/0.3.5.3" \
66+
"zinc-0.3.5.3.tgz" \
67+
"${zinc_path}"
68+
ZINC_BIN="${_DIR}/${zinc_path}"
69+
}
70+
71+
# Determine the Scala version from the root pom.xml file, set the Scala URL,
72+
# and, with that, download the specific version of Scala necessary under
73+
# the build/ folder
74+
install_scala() {
75+
# determine the Scala version used in Spark
76+
local scala_version=`grep "scala.version" "${_DIR}/../pom.xml" | \
77+
head -1 | cut -f2 -d'>' | cut -f1 -d'<'`
78+
local scala_bin="${_DIR}/scala-${scala_version}/bin/scala"
79+
80+
install_app \
81+
"http://downloads.typesafe.com/scala/${scala_version}" \
82+
"scala-${scala_version}.tgz" \
83+
"scala-${scala_version}/bin/scala"
84+
85+
SCALA_COMPILER="$(cd "$(dirname ${scala_bin})/../lib" && pwd)/scala-compiler.jar"
86+
SCALA_LIBRARY="$(cd "$(dirname ${scala_bin})/../lib" && pwd)/scala-library.jar"
87+
}
88+
89+
# Determines if a given application is already installed. If not, will attempt
90+
# to install
91+
## Arg1 - application name
92+
## Arg2 - Alternate path to local install under build/ dir
93+
check_and_install_app() {
94+
# create the local environment variable in uppercase
95+
local app_bin="`echo $1 | awk '{print toupper(\$0)}'`_BIN"
96+
# some black magic to set the generated app variable (i.e. MVN_BIN) into the
97+
# environment
98+
eval "${app_bin}=`which $1 2>/dev/null`"
99+
100+
if [ -z "`which $1 2>/dev/null`" ]; then
101+
install_$1
102+
fi
103+
}
104+
105+
# Setup healthy defaults for the Zinc port if none were provided from
106+
# the environment
107+
ZINC_PORT=${ZINC_PORT:-"3030"}
108+
109+
# Check and install all applications necessary to build Spark
110+
check_and_install_app "mvn"
111+
112+
# Install the proper version of Scala and Zinc for the build
113+
install_zinc
114+
install_scala
115+
116+
# Reset the current working directory
117+
cd "${_CALLING_DIR}"
118+
119+
# Now that zinc is ensured to be installed, check its status and, if its
120+
# not running or just installed, start it
121+
if [ -n "${ZINC_INSTALL_FLAG}" -o -z "`${ZINC_BIN} -status`" ]; then
122+
${ZINC_BIN} -shutdown
123+
${ZINC_BIN} -start -port ${ZINC_PORT} \
124+
-scala-compiler "${SCALA_COMPILER}" \
125+
-scala-library "${SCALA_LIBRARY}" &>/dev/null
126+
fi
127+
128+
# Set any `mvn` options if not already present
129+
export MAVEN_OPTS=${MAVEN_OPTS:-"-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"}
130+
131+
# Last, call the `mvn` command as usual
132+
${MVN_BIN} "$@"

‎build/sbt

+111
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
#!/usr/bin/env bash
2+
3+
# When creating new tests for Spark SQL Hive, the HADOOP_CLASSPATH must contain the hive jars so
4+
# that we can run Hive to generate the golden answer. This is not required for normal development
5+
# or testing.
6+
for i in "$HIVE_HOME"/lib/*
7+
do HADOOP_CLASSPATH="$HADOOP_CLASSPATH:$i"
8+
done
9+
export HADOOP_CLASSPATH
10+
11+
realpath () {
12+
(
13+
TARGET_FILE="$1"
14+
15+
cd "$(dirname "$TARGET_FILE")"
16+
TARGET_FILE="$(basename "$TARGET_FILE")"
17+
18+
COUNT=0
19+
while [ -L "$TARGET_FILE" -a $COUNT -lt 100 ]
20+
do
21+
TARGET_FILE="$(readlink "$TARGET_FILE")"
22+
cd $(dirname "$TARGET_FILE")
23+
TARGET_FILE="$(basename $TARGET_FILE)"
24+
COUNT=$(($COUNT + 1))
25+
done
26+
27+
echo "$(pwd -P)/"$TARGET_FILE""
28+
)
29+
}
30+
31+
. "$(dirname "$(realpath "$0")")"/sbt-launch-lib.bash
32+
33+
34+
declare -r noshare_opts="-Dsbt.global.base=project/.sbtboot -Dsbt.boot.directory=project/.boot -Dsbt.ivy.home=project/.ivy"
35+
declare -r sbt_opts_file=".sbtopts"
36+
declare -r etc_sbt_opts_file="/etc/sbt/sbtopts"
37+
38+
usage() {
39+
cat <<EOM
40+
Usage: $script_name [options]
41+
42+
-h | -help print this message
43+
-v | -verbose this runner is chattier
44+
-d | -debug set sbt log level to debug
45+
-no-colors disable ANSI color codes
46+
-sbt-create start sbt even if current directory contains no sbt project
47+
-sbt-dir <path> path to global settings/plugins directory (default: ~/.sbt)
48+
-sbt-boot <path> path to shared boot directory (default: ~/.sbt/boot in 0.11 series)
49+
-ivy <path> path to local Ivy repository (default: ~/.ivy2)
50+
-mem <integer> set memory options (default: $sbt_mem, which is $(get_mem_opts $sbt_mem))
51+
-no-share use all local caches; no sharing
52+
-no-global uses global caches, but does not use global ~/.sbt directory.
53+
-jvm-debug <port> Turn on JVM debugging, open at the given port.
54+
-batch Disable interactive mode
55+
56+
# sbt version (default: from project/build.properties if present, else latest release)
57+
-sbt-version <version> use the specified version of sbt
58+
-sbt-jar <path> use the specified jar as the sbt launcher
59+
-sbt-rc use an RC version of sbt
60+
-sbt-snapshot use a snapshot version of sbt
61+
62+
# java version (default: java from PATH, currently $(java -version 2>&1 | grep version))
63+
-java-home <path> alternate JAVA_HOME
64+
65+
# jvm options and output control
66+
JAVA_OPTS environment variable, if unset uses "$java_opts"
67+
SBT_OPTS environment variable, if unset uses "$default_sbt_opts"
68+
.sbtopts if this file exists in the current directory, it is
69+
prepended to the runner args
70+
/etc/sbt/sbtopts if this file exists, it is prepended to the runner args
71+
-Dkey=val pass -Dkey=val directly to the java runtime
72+
-J-X pass option -X directly to the java runtime
73+
(-J is stripped)
74+
-S-X add -X to sbt's scalacOptions (-S is stripped)
75+
-PmavenProfiles Enable a maven profile for the build.
76+
77+
In the case of duplicated or conflicting options, the order above
78+
shows precedence: JAVA_OPTS lowest, command line options highest.
79+
EOM
80+
}
81+
82+
process_my_args () {
83+
while [[ $# -gt 0 ]]; do
84+
case "$1" in
85+
-no-colors) addJava "-Dsbt.log.noformat=true" && shift ;;
86+
-no-share) addJava "$noshare_opts" && shift ;;
87+
-no-global) addJava "-Dsbt.global.base=$(pwd)/project/.sbtboot" && shift ;;
88+
-sbt-boot) require_arg path "$1" "$2" && addJava "-Dsbt.boot.directory=$2" && shift 2 ;;
89+
-sbt-dir) require_arg path "$1" "$2" && addJava "-Dsbt.global.base=$2" && shift 2 ;;
90+
-debug-inc) addJava "-Dxsbt.inc.debug=true" && shift ;;
91+
-batch) exec </dev/null && shift ;;
92+
93+
-sbt-create) sbt_create=true && shift ;;
94+
95+
*) addResidual "$1" && shift ;;
96+
esac
97+
done
98+
99+
# Now, ensure sbt version is used.
100+
[[ "${sbt_version}XXX" != "XXX" ]] && addJava "-Dsbt.version=$sbt_version"
101+
}
102+
103+
loadConfigFile() {
104+
cat "$1" | sed '/^\#/d'
105+
}
106+
107+
# if sbtopts files exist, prepend their contents to $@ so it can be processed by this runner
108+
[[ -f "$etc_sbt_opts_file" ]] && set -- $(loadConfigFile "$etc_sbt_opts_file") "$@"
109+
[[ -f "$sbt_opts_file" ]] && set -- $(loadConfigFile "$sbt_opts_file") "$@"
110+
111+
run "$@"

‎sbt/sbt-launch-lib.bash renamed to ‎build/sbt-launch-lib.bash

+3-3
Original file line numberDiff line numberDiff line change
@@ -37,10 +37,10 @@ dlog () {
3737
}
3838

3939
acquire_sbt_jar () {
40-
SBT_VERSION=`awk -F "=" '/sbt\\.version/ {print $2}' ./project/build.properties`
40+
SBT_VERSION=`awk -F "=" '/sbt\.version/ {print $2}' ./project/build.properties`
4141
URL1=http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
4242
URL2=http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
43-
JAR=sbt/sbt-launch-${SBT_VERSION}.jar
43+
JAR=build/sbt-launch-${SBT_VERSION}.jar
4444

4545
sbt_jar=$JAR
4646

@@ -150,7 +150,7 @@ process_args () {
150150
-java-home) require_arg path "$1" "$2" && java_cmd="$2/bin/java" && export JAVA_HOME=$2 && shift 2 ;;
151151

152152
-D*) addJava "$1" && shift ;;
153-
-J*) addJava "${1:2}" && shift ;;
153+
-J*) addJava "${1:2}" && shift ;;
154154
-P*) enableProfile "$1" && shift ;;
155155
*) addResidual "$1" && shift ;;
156156
esac

‎dev/create-release/create-release.sh

+5-5
Original file line numberDiff line numberDiff line change
@@ -87,8 +87,8 @@ if [[ ! "$@" =~ --package-only ]]; then
8787
git commit -a -m "Preparing development version $next_ver"
8888
git push origin $GIT_TAG
8989
git push origin HEAD:$GIT_BRANCH
90-
git checkout -f $GIT_TAG
91-
90+
git checkout -f $GIT_TAG
91+
9292
# Using Nexus API documented here:
9393
# https://support.sonatype.com/entries/39720203-Uploading-to-a-Staging-Repository-via-REST-API
9494
echo "Creating Nexus staging repository"
@@ -106,7 +106,7 @@ if [[ ! "$@" =~ --package-only ]]; then
106106
clean install
107107

108108
./dev/change-version-to-2.11.sh
109-
109+
110110
mvn -DskipTests -Dhadoop.version=2.2.0 -Dyarn.version=2.2.0 \
111111
-Dscala-2.11 -Pyarn -Phive -Phadoop-2.2 -Pspark-ganglia-lgpl -Pkinesis-asl \
112112
clean install
@@ -174,7 +174,7 @@ make_binary_release() {
174174
NAME=$1
175175
FLAGS=$2
176176
cp -r spark spark-$RELEASE_VERSION-bin-$NAME
177-
177+
178178
cd spark-$RELEASE_VERSION-bin-$NAME
179179

180180
# TODO There should probably be a flag to make-distribution to allow 2.11 support
@@ -219,7 +219,7 @@ scp spark-* \
219219

220220
# Docs
221221
cd spark
222-
sbt/sbt clean
222+
build/sbt clean
223223
cd docs
224224
# Compile docs with Java 7 to use nicer format
225225
JAVA_HOME=$JAVA_7_HOME PRODUCTION=1 jekyll build

‎dev/mima

+4-4
Original file line numberDiff line numberDiff line change
@@ -24,21 +24,21 @@ set -e
2424
FWDIR="$(cd "`dirname "$0"`"/..; pwd)"
2525
cd "$FWDIR"
2626

27-
echo -e "q\n" | sbt/sbt oldDeps/update
27+
echo -e "q\n" | build/sbt oldDeps/update
2828
rm -f .generated-mima*
2929

30-
# Generate Mima Ignore is called twice, first with latest built jars
30+
# Generate Mima Ignore is called twice, first with latest built jars
3131
# on the classpath and then again with previous version jars on the classpath.
3232
# Because of a bug in GenerateMIMAIgnore that when old jars are ahead on classpath
33-
# it did not process the new classes (which are in assembly jar).
33+
# it did not process the new classes (which are in assembly jar).
3434
./bin/spark-class org.apache.spark.tools.GenerateMIMAIgnore
3535

3636
export SPARK_CLASSPATH="`find lib_managed \( -name '*spark*jar' -a -type f \) | tr "\\n" ":"`"
3737
echo "SPARK_CLASSPATH=$SPARK_CLASSPATH"
3838

3939
./bin/spark-class org.apache.spark.tools.GenerateMIMAIgnore
4040

41-
echo -e "q\n" | sbt/sbt mima-report-binary-issues | grep -v -e "info.*Resolving"
41+
echo -e "q\n" | build/sbt mima-report-binary-issues | grep -v -e "info.*Resolving"
4242
ret_val=$?
4343

4444
if [ $ret_val != 0 ]; then

0 commit comments

Comments
 (0)