Skip to content

Commit 521b7e8

Browse files
author
Grant Magdanz
committed
PYT-428 Improve build and publish process
PYT-390 dvp build help messages have some small formatting problems Reviewed at: http://reviews.delphix.com/r/49057/
1 parent 142ca78 commit 521b7e8

20 files changed

+385
-146
lines changed

‎.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,9 @@ activate*
2626
# Pipenv virtual env directory
2727
.venv/
2828

29+
# Ignore other virtual environments
30+
venv/
31+
2932
# Pytest cache
3033
.pytest_cache/
3134

‎README.md

Lines changed: 71 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -31,16 +31,11 @@ The first thing to do is to change the version number. In the root of the SDK co
3131
#### Building the JAR
3232
In order to build an SDK runtime JAR, navigate to the root directory of the repository, and execute:
3333
```
34-
./build.sh
34+
./gradlew jar
3535
```
36-
The build script will produce a few build directories and output `sdk-<version>.jar` file.
36+
This will produce the jar under `build/libs`.
3737

38-
Note that the build script does not yet do a lot of error checking. So, it is sometimes not immediately apparent if the build did not succeed. Check the timestamp on the `sdk-<version>.jar` file to make sure it's been newly-built. If not, you might have to scroll through the `build.sh` output to find an error message.
39-
40-
If you want to remove the results of the JAR build, they can be cleaned by going to the root directory of the respository and executing:
41-
```
42-
./clean.sh
43-
```
38+
Note that the build not yet do a lot of error checking. So, it is sometimes not immediately apparent if the build did not succeed. Check the timestamp on the `sdk-<version>.jar` file to make sure it's been newly-built. If it's not up to date, you might need to modify the script `bin/build.sh` to produce more output, rerun the jar task, and look at the failure messages.
4439

4540
#### Building the Python distribution
4641

@@ -76,7 +71,7 @@ We need to put the local SDK build somewhere that the appgate code can access it
7671

7772
#### Making Local SDK Build Available
7873

79-
This step is easy. Simply copy `sdk-<version>.jar` to `appliance/lib` in the `app-gate`.
74+
This step is easy. Simply copy `build/libs/sdk-<version>.jar` to `appliance/lib` in the `app-gate`.
8075

8176
Don't forget to check this change into git if you plan on using `git dx-test` or `git appliance-deploy` to test. (Note: you will **not** be pushing this! We'll undo this change later.)
8277

@@ -86,49 +81,49 @@ Don't forget to check this change into git if you plan on using `git dx-test` or
8681
Delphix Engine and IntelliJ both use gradle to build. So, we have to ensure that the gradle build knows how to find and use our local SDK build. This is a two-step process.
8782

8883

89-
**Step 1** We need to tell gradle to look for jars in the `lib` directory. In order to do that, we will have to add the following code to `appliance/gradle-lib/java-common.gradle`:
90-
91-
```
92-
flatDir {
93-
dirs "${gradleLibDir}/../lib/"
94-
}
95-
```
96-
97-
The above entry has to be added in the list of external repositories. Here is a more complete listing.
98-
99-
```
100-
/*
101-
* External repositories we fetch jars from. This should never include a repository
102-
* that is not managed by Delphix. Third party repos should be mirrored through
103-
* http://artifactory.delphix.com/.
104-
*/
105-
repositories {
106-
/*
107-
* Legacy location for jars that were checked directly into the app-gate.
108-
*/
109-
ivy {
110-
ivyPattern "${gradleLibDir}/../lib/[module]/ivy-[revision].xml"
111-
artifactPattern "${gradleLibDir}/../lib/[module]/[artifact]-[revision].[ext]"
112-
}
113-
...
114-
...
115-
flatDir {
116-
dirs "${gradleLibDir}/../lib/"
117-
}
118-
}
119-
```
120-
121-
**Step 2** We have to tell gradle to actually use our local SDK where applicable. We have two modules that need to see the SDK: `appdata` and `workflow`. So, we have to edit both `appliance/server/appdata/build.gradle` and `appliance/server/workflow/build.gradle`).
122-
123-
We need to add the following line:
124-
```
125-
compile name: "sdk-<version>"
126-
```
127-
128-
We also need to remove (or comment out) this line so that gradle will not try to use an artifactory build:
129-
```
130-
implementation group: "com.delphix.virtualization", name: "sdk", version: virtualizationSdkVer
131-
```
84+
1. We need to tell gradle to look for jars in the `lib` directory. In order to do that, we will have to add the following code to `appliance/gradle-lib/java-common.gradle`:
85+
86+
```
87+
flatDir {
88+
dirs "${gradleLibDir}/../lib/"
89+
}
90+
```
91+
92+
The above entry has to be added in the list of external repositories. Here is a more complete listing.
93+
94+
```
95+
/*
96+
* External repositories we fetch jars from. This should never include a repository
97+
* that is not managed by Delphix. Third party repos should be mirrored through
98+
* http://artifactory.delphix.com/.
99+
*/
100+
repositories {
101+
/*
102+
* Legacy location for jars that were checked directly into the app-gate.
103+
*/
104+
ivy {
105+
ivyPattern "${gradleLibDir}/../lib/[module]/ivy-[revision].xml"
106+
artifactPattern "${gradleLibDir}/../lib/[module]/[artifact]-[revision].[ext]"
107+
}
108+
...
109+
...
110+
flatDir {
111+
dirs "${gradleLibDir}/../lib/"
112+
}
113+
}
114+
```
115+
116+
2. We have to tell gradle to actually use our local SDK where applicable. We have two modules that need to see the SDK: `appdata` and `workflow`. So, we have to edit both `appliance/server/appdata/build.gradle` and `appliance/server/workflow/build.gradle`).
117+
118+
We need to add the following line:
119+
```
120+
compile name: "sdk-<version>"
121+
```
122+
123+
We also need to remove (or comment out) this line so that gradle will not try to use an artifactory build:
124+
```
125+
implementation group: "com.delphix.virtualization", name: "sdk", version: virtualizationSdkVer
126+
```
132127
133128
Once you complete the above two steps, IntelliJ should notice the changes to your build files and rebuild the project. If you don't have the auto-rebuild option turned on, refresh the gradle build.
134129
@@ -155,46 +150,39 @@ In addition, it's customary to publish a "provisional" appgate review, so that p
155150
156151
## Pushing and Deploying SDK Code
157152
158-
### Preparation
159153
160-
1. Make sure you have both the JAR and the Python distribution built, as described above. Double check that you have the correct version in `build.gradle`.
154+
### Publishing
161155
162-
2. You will need your Artifactory API key, which you can get by navigating to http://artifactory.delphix.com/artifactory/webapp/#/profile (make sure you're logged in).
156+
There are two Gradle tasks that do publishing: `publishDebug` and `publishProd`. They differ in two ways:
163157
164-
3. Make sure that you have `twine` installed (run `pip install twine`), and make sure that your `~/.pypirc` file has your credentials:
165-
```bash
166-
[~] cat ~/.pypirc
167-
[distutils]
168-
index-servers =
169-
local
158+
1. They publish the Python distributions to separate repositories on Artifactory (the jar is always published to the same one.). `publishDebug` uploads to `dvp-local-pypi`. This is a special repository that has been setup to test the SDK. It falls back our our production PyPI repository, but artifacts uploaded to `dvp-local-pypi` do not impact production artifacts. This should be used for testing. `publishProd` does upload the Python distributions to our production Artifactory PyPI repository, `delphix-local`.
170159
171-
[local]
172-
repository: http://artifactory.delphix.com/artifactory/api/pypi/delphix-local
173-
username: user
174-
password: pass
175-
```
160+
2. `publishProd` runs tests, formatting, and linting while `publishDebug` does not.
176161
177-
### Publishing
162+
NOTE: The external release to `pypi.org` is done outside of the build system.
178163
179-
We do not yet have an autobuild/autodeploy mechanism for SDK changes, and so you must manually perform three related steps separately
164+
#### Setup
180165
181-
1. Push your source changes to our git server with `git push`.
182-
!!! note
183-
Depending on what you've changed, you may find a variety of "lock files" that have been changed. These need to be checked in and pushed alongside your source changes. These guarantee consistent builds from developer to developer.
166+
1. There are three environment variables that need to be set in order to publish: `ARTIFACTORY_PYPI_USER`, `ARTIFACTORY_PYPI_PASS`, and `ARTIFACTORY_JAR_KEY`.
184167
185-
2. Publish the Jar to our Artifactory server with this command:
186-
```
187-
curl -H 'X-JFrog-Art-Api: <Artifactory-API-key>' -T /path/to/virtualization-sdk/sdk-<version>.jar "http://artifactory.delphix.com/artifactory/virtualization-sdk/com/delphix/virtualization/sdk/<version>/sdk-<version>.jar"
188-
```
168+
`ARTIFACTORY_PYPI_USER` and `ARTIFACTORY_PYPI_PASS` are one set of credentials used to upload the Python distributions to our internal PyPI repositories. The credentials are the same for both internal PyPI repositories mentioned above.
169+
`ARTIFACTORY_JAR_KEY`
189170
190-
3. Publish the Python distribution to our Pypi server with these commands:
191-
```bash
192-
twine upload -r local tools/build/python-dist/*
193-
twine upload -r local libs/build/python-dist/*
194-
twine upload -r local platform/build/python-dist/*
195-
twine upload -r local common/build/python-dist/*
196-
twine upload -r local dvp/build/python-dist/*
197-
```
171+
- `ARTIFACTORY_PYPI_USER` and `ARTIFACTORY_PYPI_PASS` is the username/password combo given to you by whoever setup your Artifactory pypi account. If you do not have one, please reach out to the `#appdata-core` channel. These used to upload the Python distributions to our internal PyPI repositories. The credentials are the same for both internal PyPI repositories mentioned above.
172+
- `ARTIFACTORY_JAR_KEY` is your Artifactory API key and is used to upload the jar. It can be retreived from http://artifactory.delphix.com/artifactory/webapp/#/profile. You may have to login. This is different from the PyPI credentials because the artifacts are being uploaded to different repositories on Artifactory.
173+
174+
2. `twine` needs to be installed. This is a Python package that is used to upload Python distributions. If it's not installed, install it by running `pip install twine`.
175+
176+
177+
#### Debug Publishing
178+
179+
Run `./gradlew publishDebug`. This will build the jar, every Python distribution, and upload them to Artifactory with the Python distributions going to our testing repository, `dvp-local-pypi`.
180+
181+
You can install `dvp` from this repository with the command `pip install -i https://artifactory.delphix.com/artifactory/api/pypi/dvp-local-pypi/simple dvp==<version>`.
182+
183+
#### Final Publishing
184+
185+
Once you are absolutely certain all changes have been made run `./gradlew publishProd`. This will run checks, build the jar, create the Python distributions, and upload all of them to Artifactory with the Python distributions going to `delphix-local`.
198186
199187
## Using Newly-Deployed SDK Build
200188

‎build.sh renamed to ‎bin/build.sh

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,15 @@
33
# Copyright (c) 2018, 2019 by Delphix. All rights reserved.
44
#
55

6+
SCRIPT_DIR=`dirname $0`
7+
source ${SCRIPT_DIR}/common.sh
8+
69
# This script must be executed from the root directory of the virtualization-sdk repo.
7-
cd "$(git rev-parse --show-toplevel)"
10+
ROOT=`git rev-parse --show-toplevel`
11+
cd $ROOT
12+
13+
mkdir -p build/libs
14+
cd build/libs
815

916
echo "Copying Virtualization SDK binaries from NAS..."
1017
scp -r delphix@support-tools:/nas/engineering/fdrozdowski/virtualization-sdk/bin .
@@ -22,9 +29,9 @@ mkdir -p ${JAR_DIRECTORY}/dlpx/virtualization/
2229
touch ${JAR_DIRECTORY}/dlpx/__init__.py
2330
touch ${JAR_DIRECTORY}/dlpx/virtualization/__init__.py
2431

25-
cp common/src/main/proto/dlpx/virtualization/common.proto ${JAR_DIRECTORY}/dlpx/virtualization/common.proto
26-
cp platform/src/main/proto/dlpx/virtualization/platform.proto ${JAR_DIRECTORY}/dlpx/virtualization/platform.proto
27-
cp libs/src/main/proto/dlpx/virtualization/libs.proto ${JAR_DIRECTORY}/dlpx/virtualization/libs.proto
32+
cp ${ROOT}/common/src/main/proto/dlpx/virtualization/common.proto ${JAR_DIRECTORY}/dlpx/virtualization/common.proto
33+
cp ${ROOT}/platform/src/main/proto/dlpx/virtualization/platform.proto ${JAR_DIRECTORY}/dlpx/virtualization/platform.proto
34+
cp ${ROOT}/libs/src/main/proto/dlpx/virtualization/libs.proto ${JAR_DIRECTORY}/dlpx/virtualization/libs.proto
2835

2936
cd ${JAR_DIRECTORY}
3037
cp -r ./../bin .
@@ -39,13 +46,18 @@ java -jar bin/jython-standalone-2.7.1.jar -Dcpython_cmd=python -m py_compile dlp
3946
java -jar bin/jython-standalone-2.7.1.jar -Dcpython_cmd=python -m py_compile dlpx/virtualization/libs_pb2.py
4047

4148
echo "Compiling Java source files to Java classes..."
42-
javac -d . -classpath bin/protobuf-java-3.6.1.jar com/delphix/virtualization/common/*java com/delphix/virtualization/platform/*java com/delphix/virtualization/libs/*java
49+
javac -d . -classpath bin/protobuf-java-3.6.1.jar com/delphix/virtualization/common/*java com/delphix/virtualization/platform/*java com/delphix/virtualization/libs/*java > /dev/null
4350

44-
rsync -av --progress ./../platform/src/main/python/dlpx/ dlpx/
45-
rsync -av --progress ./../libs/src/main/python/dlpx/ dlpx/
51+
rsync -av --progress ${ROOT}/platform/src/main/python/dlpx/ dlpx/ > /dev/null
52+
rsync -av --progress ${ROOT}/libs/src/main/python/dlpx/ dlpx/ > /dev/null
4653
rm -r bin
4754

55+
VERSION=`cat "${ROOT}/build.gradle" | grep '^\s*version\s*=\s*"*"'| sed -E 's/.*"(.*)".*/\1/g'`
56+
[ -z "$VERSION" ] && die "Failed to retrieve SDK version from build.gradle."
57+
4858
echo "Creating a Virtualization SDK jar..."
49-
JAR_FILE_NAME=sdk-\<version\>.jar
50-
jar cvf ${JAR_FILE_NAME} .
51-
mv ${JAR_FILE_NAME} ./../
59+
JAR_FILE_NAME="sdk-$VERSION.jar"
60+
jar cvf ${JAR_FILE_NAME} . > /dev/null
61+
mv ${JAR_FILE_NAME} ./..
62+
63+
exit 0

‎bin/check_version_bump.sh

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
#!/bin/bash
2+
#
3+
# Copyright (c) 2019 by Delphix. All rights reserved.
4+
#
5+
6+
# This script checks the diff of 'build.gradle' and fails if the version has not been changed. This is a crude
7+
# implementation of this check and should not be kept around for long. Instead, the version should probably be managed
8+
# primarily by our CI pipeline so no manual action is needed.
9+
10+
ROOT=`git rev-parse --show-toplevel`
11+
SCRIPT_DIR=`dirname $0`
12+
CWD=`pwd`
13+
14+
source ${SCRIPT_DIR}/common.sh
15+
16+
cd ${ROOT}
17+
18+
# Diff 'build.gradle' with origin/master. This works since we don't have any backport branches. Grep that diff
19+
# for a line that starts with a '+' that signifies an addition in git. The line should then have 'version =' followed
20+
# by a something in quotes that starts with x.y.z where x, y, and z are digits. Anything is allowed to follow that
21+
# until the quotes end.
22+
VERSION=`git diff origin/master -- build.gradle | grep '^\+\s*version\s*=\s*"[0-9]\.[0-9]\.[0-9].*"'`
23+
[[ -z ${VERSION} ]] && die "The SDK version has not been increased. Please increment the version in <virtualization-sdk>/build.gradle."
24+
25+
cd ${CWD}
26+
27+
exit 0

‎bin/common.sh

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
#!/bin/bash
2+
#
3+
# Copyright (c) 2019 by Delphix. All rights reserved.
4+
#
5+
6+
# A helper function that takes an arbitrary number of arguments, prints them to stdout and exits with a non-zero exit
7+
# code.
8+
die() { printf "$*\n" 1>&2; exit 1; }

‎bin/upload.sh

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
#!/bin/bash
2+
#
3+
# Copyright (c) 2019 by Delphix. All rights reserved.
4+
#
5+
6+
# This script uploads the SDK jar as well as all Python distributions created by the SDK. It assumes that all artifacts
7+
# already exist. This is not intended to be a long term solution. Instead, this is a dirty way to abstract away
8+
# some of this logic. It should instead be rolled directly into the Gradle build and our future CI pipeline.
9+
#
10+
# This script can upload to both our internal dev and prod PyPI repositories. It defaults to dev.
11+
12+
SCRIPT_DIR=`dirname $0`
13+
source ${SCRIPT_DIR}/common.sh
14+
15+
USAGE="Usage: upload.sh [--prod]"
16+
17+
# Validate usage is correct and expected environment variables are set.
18+
if [[ $# -gt 1 ]]; then
19+
die $USAGE
20+
elif [[ $# -eq 1 && $1 != "--prod" ]]; then
21+
die $USAGE
22+
elif [[ -z ${ARTIFACTORY_PYPI_USER} || -z ${ARTIFACTORY_PYPI_PASS} || -z ${ARTIFACTORY_JAR_KEY} ]]; then
23+
die "ARTIFACTORY_PYPI_USER, ARTIFACTORY_PYPI_PASS, and/or ARTIFACTORY_JAR_KEY environment variables are not set. Set them or pass them in as arguments to upload.sh."
24+
fi
25+
26+
# dvp-local-pypi is used for testing and is the default. delphix-local is our internal production PyPI repository and
27+
# should only be used when publishing final versions.
28+
if [[ $# -eq 0 ]]; then
29+
REPO="http://artifactory.delphix.com/artifactory/api/pypi/dvp-local-pypi"
30+
else
31+
die "GOING TO DO A PROD UPLOAD"
32+
REPO="http://artifactory.delphix.com/artifactory/api/pypi/delphix-local"
33+
fi
34+
35+
# Check early that 'twine' is on the path.
36+
command -v twine 2>&1 >/dev/null || die "'twine' is either not install or not on PATH. To install 'twine' run 'pip install twine'"
37+
38+
# All the file paths need to be relative to the root of the git repo
39+
ROOT=`git rev-parse --show-toplevel`
40+
41+
# Get the SDK version from build.gradle in the root of the SDK. This essentially just looks for a line that has
42+
# 'version =' and pulls the value from the quotes after it. Nothing too sophisticated and fairly error prone.
43+
VERSION=`cat "${ROOT}/build.gradle" | grep '^\s*version\s*=\s*"*"'| sed -E 's/.*"(.*)".*/\1/g'`
44+
[ -z "$VERSION" ] && die "Failed to retrieve SDK version from build.gradle."
45+
46+
echo "Uploading custom build jar..."
47+
RESPONSE=`curl --silent --write-out "%{http_code}" -H "X-JFrog-Art-Api: ${ARTIFACTORY_JAR_KEY}" -T "${ROOT}/build/libs/sdk-${VERSION}.jar" "http://artifactory.delphix.com/artifactory/virtualization-sdk/com/delphix/virtualization/sdk/${VERSION}/sdk-${VERSION}.jar"`
48+
49+
# The above 'curl' command writes out "${http_code}" so the last three characters of the output will be the HTTP
50+
# response code. If that response code is not "201", it is a failure so die and then print the response. ${REPONSE%????}
51+
# prints $REPONSE without the last 4 characters which are the HTTP exit code and an 'n'.
52+
[ ${RESPONSE: -3} -ne "201" ] && die "Failed to upload ${ROOT}/build/libs/sdk-${VERSION}.jar to artifactory:\n" ${RESPONSE%????}
53+
54+
echo "Uploading 'common' Python distribution..."
55+
twine upload --repository-url ${REPO} -u ${ARTIFACTORY_PYPI_USER} -p ${ARTIFACTORY_PYPI_PASS} "${ROOT}/common/build/python-dist/*${VERSION}.tar.gz" > /dev/null
56+
57+
echo "Uploading 'platform' Python distribution..."
58+
twine upload --repository-url ${REPO} -u ${ARTIFACTORY_PYPI_USER} -p ${ARTIFACTORY_PYPI_PASS} "${ROOT}/platform/build/python-dist/*${VERSION}.tar.gz" > /dev/null
59+
60+
echo "Uploading 'libs' Python distribution..."
61+
twine upload --repository-url ${REPO} -u ${ARTIFACTORY_PYPI_USER} -p ${ARTIFACTORY_PYPI_PASS} "${ROOT}/libs/build/python-dist/*${VERSION}.tar.gz" > /dev/null
62+
63+
echo "Uploading 'tools' Python distribution..."
64+
twine upload --repository-url ${REPO} -u ${ARTIFACTORY_PYPI_USER} -p ${ARTIFACTORY_PYPI_PASS} "${ROOT}/tools/build/python-dist/*${VERSION}.tar.gz" > /dev/null
65+
66+
echo "Uploading 'dvp' Python distribution..."
67+
twine upload --repository-url ${REPO} -u ${ARTIFACTORY_PYPI_USER} -p ${ARTIFACTORY_PYPI_PASS} "${ROOT}/dvp/build/python-dist/*${VERSION}.tar.gz" > /dev/null
68+
69+
exit 0

0 commit comments

Comments
 (0)