Merge branch 'dev' into deploy-to-ec2

Author: landonreed

.github/CODEOWNERS

@@ -0,0 +1,4 @@
+# See https://help.github.com/articles/about-codeowners/
+
+# A Conveyal employee is required to approve PR merges
+* @conveyal/employees

.github/issue_template.md

@@ -0,0 +1,21 @@
+_**NOTE:** This issue system is intended for reporting bugs and tracking progress in software development. Although this software is licensed with an open-source license, any issue opened here may not be responded to in a timely manner. [Conveyal](https://www.conveyal.com) is unable to provide technical support for custom deployments of this software unless your company has a support contract with us. Please remove this note when creating the issue._
+
+## Observed behavior
+
+Please explain what is being observed within the application here.
+
+## Expected behavior
+
+Please explain what should happen instead.
+
+## Steps to reproduce the problem
+
+Please be as specific as possible.
+
+## Any special notes on configuration used
+
+Please describe any applicable config files that were used
+
+## Version of datatools-server and datatools-ui if applicable (exact commit hash or branch name)
+
+If using this in conjunction with [datatools-ui](https://github.com/conveyal/datatools-ui), this info can be found by clicking on the gear icon on the sidebar.

.github/pull_request_template.md

@@ -0,0 +1,14 @@
+### Checklist
+
+- [ ] Appropriate branch selected _(all PRs must first be merged to `dev` before they can be merged to `master`)_
+- [ ] Any modified or new methods or classes have helpful JavaDoc and code is thoroughly commented
+- [ ] The description lists all applicable issues this PR seeks to resolve
+- [ ] The description lists any configuration setting(s) that differ from the default settings
+- [ ] All tests and CI builds passing
+- [ ] The description lists all relevant PRs included in this release _(remove this if not merging to master)_
+- [ ] e2e tests are all passing _(remove this if not merging to master)_
+- [ ] Code coverage improves or is at 100% _(remove this if not merging to master)_
+
+### Description
+
+Please explain the changes you made here and, if not immediately obvious from the code, how they resolve any referenced issues. Be sure to include all issues being resolved and any special configuration settings that are need for the software to run properly with these changes. If merging to master, please also list the PRs that are to be included.

.travis.yml

@@ -1,44 +1,79 @@
+dist: trusty # jdk 8 not available on xenial
 language: java
-jdk:
-- oraclejdk8
+java:
+  - oraclejdk8
 install: true
 sudo: false
 # Install mongoDB to perform persistence tests
-services: mongodb
+services:
+  - mongodb
+  - postgresql
+addons:
+  postgresql: 9.6
 cache:
   directories:
-  - "$HOME/.m2"
+    - $HOME/.m2
+    - $HOME/.cache/yarn
+# Install semantic-release
+before_script:
+  - yarn global add @conveyal/maven-semantic-release semantic-release@15
+  # Create dir for GTFS+ files (used during testing)
+  - mkdir /tmp/gtfsplus
 before_install:
 #- sed -i.bak -e 's|https://nexus.codehaus.org/snapshots/|https://oss.sonatype.org/content/repositories/codehaus-snapshots/|g' ~/.m2/settings.xml
 # set region in AWS config for S3 setup
 - mkdir ~/.aws && printf '%s\n' '[default]' 'aws_access_key_id=foo' 'aws_secret_access_key=bar' 'region=us-east-1' > ~/.aws/config
 - cp configurations/default/server.yml.tmp configurations/default/server.yml
+# create database for tests
+- psql -U postgres -c 'CREATE DATABASE catalogue;'
 script:
 # package jar
 - mvn package
 after_success:
-# Upload coverage reports to codecov.io
-- bash <(curl -s https://codecov.io/bash)
-# notify slack channel of build status
+  # this first codecov run will upload a report associated with the commit set through Travis CI environment variables
+  - bash <(curl -s https://codecov.io/bash)
+  # run maven-semantic-release to potentially create a new release of datatools-server. The flag --skip-maven-deploy is
+  # used to avoid deploying to maven central. So essentially, this just creates a release with a changelog on github.
+  #
+  # If maven-semantic-release finishes successfully and the current branch is master, upload coverage reports for the
+  # commits that maven-semantic-release generated. Since the above codecov run is associated with the commit that
+  # initiated the Travis build, the report will not be associated with the commits that maven-semantic-release performed
+  # (if it ended up creating a release and the two commits that were a part of that workflow). Therefore, if on master
+  # codecov needs to be ran two more times to create codecov reports for the commits made by maven-semantic-release.
+  # See https://github.com/conveyal/gtfs-lib/issues/193.
+  #
+  # The git commands get the commit hash of the HEAD commit and the commit just before HEAD.
+  - |
+    semantic-release --prepare @conveyal/maven-semantic-release --publish @semantic-release/github,@conveyal/maven-semantic-release --verify-conditions @semantic-release/github,@conveyal/maven-semantic-release --verify-release @conveyal/maven-semantic-release --use-conveyal-workflow --dev-branch=dev --skip-maven-deploy
+    if [[ "$TRAVIS_BRANCH" = "master" ]]; then
+      bash <(curl -s https://codecov.io/bash) -C "$(git rev-parse HEAD)"
+      bash <(curl -s https://codecov.io/bash) -C "$(git rev-parse HEAD^)"
+    fi
 notifications:
+  # notify slack channel of build status
   slack: conveyal:WQxmWiu8PdmujwLw4ziW72Gc
 before_deploy:
 # get branch name of current branch for use in jar name: https://graysonkoonce.com/getting-the-current-branch-name-during-a-pull-request-in-travis-ci/
 - export BRANCH=$(if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then echo $TRAVIS_BRANCH; else echo $TRAVIS_PULL_REQUEST_BRANCH; fi)
-# copy packaged jars over to deploy dir
+# Create directory that will contain artifacts to deploy to s3.
 - mkdir deploy
+# Display contents of target directory (for logging purposes only).
+- ls target/*.jar
+# Copy packaged jars over to deploy dir.
 - cp target/dt-*.jar deploy/
-- cp "target/dt-$(git describe --always).jar" "deploy/dt-latest-${BRANCH}.jar"
+# FIXME: Do not create a branch-specific jar for now. Having a jar that changes contents but keeps the same name
+#   may cause confusion down the road and may be undesirable.
+# - cp "target/dt-$(git describe --always).jar" "deploy/dt-latest-${BRANCH}.jar"
 deploy:
   provider: s3
   skip_cleanup: true
-  access_key_id: AKIAJISY76KTZBNHS4SA
+  access_key_id: AKIAIWMAQP5YXWT7OZEA
   secret_access_key:
-    secure: a2PNYiv7kzgKxfSx6IhZxSCFBZTCjrbIAK/vmCB1KcpnlV4vTt/IL13i3u6XC8wAbUxhd6iJMtVRm4diIwmy0K7nnpp0h3cQDxYqWCmf1dHZWBJXkpurDpbfxW5G6IlL14i+EsTSCpmwalov+atOBDVyJWVGqfEYaj9c6Q1E0fiYNP3QwZQcsVuD1CRw91xzckfERwqYcz70p/hmTEPOgUwDHuyHsjFafJx+krY3mnBdRdDRLcnPavjcEtprjGkdiVbNETe3CHVNQrAVfqm187OoDA2tHTPjTFmlAdUedp4rYqLmF/WWbHZLzUkQb95FJkklx30vlwC0bIutP1TwIlr3ma5aCRFc58x3SzG07AeM+vbt/nh5A52cpdRjBnhctC2kL++QvwkJhwRy2xptl/WEd5AUagoN4ngnGzyDS4kk/taQFL0IAav5C2WH668kGyH17KNeWG/bCDd55oCvwNlppAYXH+WdbtylqiVb9Fllvs1wcIYWqqyX5zdYiyFEI8LyEQsNF/D5ekuAtLXcF25uwjNtHMjdAxQxHbAbBOeaaLwJd29os9GrKFI/2C0TVXZo2zaFLZyFaIsDHqAC+MXDBDtktimC9Uuozz7bXENCrOUBfsDEQXb46tkXLGaQNXeOhe3KwVKxlGDCsLb7iHIcdDyBm19hqUWhU3uA+dU=
+    secure: cDfIv+/+YimqsH8NvWQZy9YTqaplOwlIeEK+KEBCfsJ3DJK5sa6U4BMZCA4OMP1oTEaIxkd4Rcvj0OAYSFQVNQHtwc+1WeHobzu+MWajMNwmJYdjIvCqMFg2lgJdzCWv6vWcitNvrsYpuXxJlQOirY/4GjEh2gueHlilEdJEItBGYebQL0/5lg9704oeO9v+tIEVivtNc76K5DoxbAa1nW5wCYD7yMQ/cc9EQiMgR5PXNEVJS4hO7dfdDwk2ulGfpwTDrcSaR9JsHyoXj72kJHC9wocS9PLeeYzNAw6ctIymNIjotUf/QUeMlheBbLfTq6DKQ0ISLcD9YYOwviUMEGmnte+HCvTPTtxNbjBWPGa2HMkKsGjTptWu1RtqRJTLy19EN1WG5znO9M+lNGBjLivxHZA/3w7jyfvEU3wvQlzo59ytNMwOEJ3zvSm6r3/QmOr5BU+UHsqy5vv2lOQ9Nv10Uag11zDP1YWCoD96jvjZJsUZtW80ZweHYpDMq0vKdZwZSlbrhgHzS7vlDW7llZPUntz0SfKCjtddbRdy6T4HgsmA8EsBATfisWpmFA6roQSnYwfEZ5ooJ8IMjfOm1qGphrP1Qv8kYkqdtOyTijYErqJ3YzldjeItqaWtyD5tmHm6Wmq6XIbw4bnSfGRx9di+cG5lDEPe1tfBPCf9O5M=
   # upload jars in deploy dir to bucket
   bucket: datatools-builds
   local-dir: deploy
   acl: public_read
   on:
-    repo: catalogueglobal/datatools-server
+    repo: ibi-group/datatools-server
     all_branches: true

configurations/default/env.yml.tmp

@@ -1,10 +1,13 @@
+# This client ID refers to the UI client in Auth0.
 AUTH0_CLIENT_ID: your-auth0-client-id
 AUTH0_DOMAIN: your-auth0-domain
 # Note: One of AUTH0_SECRET or AUTH0_PUBLIC_KEY should be used depending on the signing algorithm set on the client.
 # It seems that newer Auth0 accounts (2017 and later) might default to RS256 (public key).
 AUTH0_SECRET: your-auth0-secret # uses HS256 signing algorithm
 # AUTH0_PUBLIC_KEY: /path/to/auth0.pem # uses RS256 signing algorithm
-AUTH0_TOKEN: your-auth0-token
+# This client/secret pair refer to a machine-to-machine Auth0 application used to access the Management API.
+AUTH0_API_CLIENT: your-api-client-id
+AUTH0_API_SECRET: your-api-secret-id
 DISABLE_AUTH: false
 OSM_VEX: http://localhost:1000
 SPARKPOST_KEY: your-sparkpost-key

configurations/default/server.yml.tmp

@@ -15,6 +15,9 @@ modules:
     enabled: false
   user_admin:
     enabled: true
+  # Enable GTFS+ module for testing purposes
+  gtfsplus:
+    enabled: true
   gtfsapi:
     enabled: true
     load_on_fetch: false
@@ -29,3 +32,6 @@ extensions:
     enabled: true
     api: http://api.transitfeeds.com/v1/getFeeds
     key: your-api-key
+  # Enable MTC for testing purposes
+  mtc:
+    enabled: true

jmeter/README.md

@@ -31,8 +31,8 @@ The test plan can be ran straight from the command line.  A helper script is pro
 | 1 | test plan mode | `batch`, `fetch`, `query` or `upload` | which test plan mode to use when running the jmeter script. (see notes below for more explanation of these test plan modes) |
 | 2 | number of threads | an integer greater than 0 | The number of simultaneous threads to run at a time.  The threads will have staggered start times 1 second apart. |
 | 3 | number of loops | an integer greater than 0 | the number of loops to run.  This is combined with the number of threads, so if the number of threads is 10 and the number of loops is 8, the total number of test plans to run will be 80. |
-| 4 | project name or batch csv file | string of the project name or string of file path to batch csv file | This argument is required if running the script with the `batch` test plan mode, otherwise, this argument is optional.  The jmeter script will create new projects with a project name plus the current iteration number.  The default name is "test project #".  Also, if the s3 bucket argument is also provided, the output folder will be tarred up and with this name. |
-| 5 | s3 bucket | string of an s3 bucket | OPTIONAL.  If provided, the script will tar up the output folder and attempt to upload to the specified s3 bucket.  This assumes that aws credentials have been setup for use by the `aws` command line tool. |
+| 4 | project name or batch csv file | string of the project name or string of file path to batch csv file | This argument is required if running the script with the `batch` test plan mode, otherwise, this argument is optional.<br><br>If in `fetch` or `upload` mode, the jmeter script will create new projects with a the provided project name (or "test project" if a name is not provided) plus the current iteration number.  In `fetch` or `upload` mode, the feed url and upload file is not configurable.  In `fetch` mode, the url `http://documents.atlantaregional.com/transitdata/gtfs_ASC.zip` will be used to fetch the feed to create the feed version.  In `upload` mode, the file `fixtures/gtfs.zip` will be uploaded to create the feed version.<br><br>If in `query` mode, jmeter will try to find the project matching the provided name (as long as the project name is not "test project") or a random project will be picked if this argument is not provided.  |
+| 5 | s3 bucket | string of an s3 bucket | OPTIONAL.  If provided, the script will tar up the output folder and attempt to upload to the specified s3 bucket.  This assumes that aws credentials have been setup for use by the `aws` command line tool.  If not running in batch mode and a project name has been specified, the name of this file will be `{project name}.tar.gz`.  Otherwise, the name will be `output.tar.gz`.  |
 
 Examples:
 
@@ -48,7 +48,7 @@ _Run the test plan in query mode 80 total times in 10 threads each completing 8
 
 _Run in batch mode.  Note that all feeds in the csv file will be processed in each loop.  So in the following command, each feed in the batch.csv file would be processed 6 times.  See the section below for documentation on the csv file and also see the fixtures folder for an example file._
 ```sh
-./run-tests.sh query 3 2 batch.csv my-s3-bucket
+./run-tests.sh batch 3 2 batch.csv my-s3-bucket
 ```
 
 ### Running the upload test on multiple gtfs files
@@ -124,6 +124,8 @@ This section is run under the `query` test plan mode.  This script assumes that
 
 This section is run in all test plan modes.
 
+1.  Fetch stops and a row count of stops
+1.  Make sure the number of stops matches the row count of stops
 1.  Fetch all routes
 1.  Pick a random route
 1.  Fetch all trips on selected route
@@ -133,11 +135,13 @@ This section is run in all test plan modes.
 1.  Fetch embedded stop_times from trips from a random pattern
 1.  Check that all stop_times have proper trip_id
 1.  Check that all stop_times in trips on pattern have same stop sequence as pattern
+1.  Make a GraphQL request that contains a nested query of routes, patterns and stops
+1.  Make sure that each route is present in the route within the list of patterns
 
 ## Reporting
 
 If running this script in GUI mode, it is possible to see all results in real-time by viewing the various listeners at the end of the thread group.
 
 When running the test plan from the command line in non-gui mode, reports will be saved to the `output` folder.  The outputs will contain a csv file of all requests made and an html report summarizing the results.  If the test plan mode was `batch`, `fetch` or `upload` than another csv file will be written that contains a list of the elapsed time for processing the creation of a new gtfs feed version.
 
-The csv files can be loaded into a jmeter GUI listener to view more details.
+The csv files can be loaded into a jmeter GUI to view more details.