Release script for downstream OTP forks

script/CUSTOM_RELEASE_README.md

@@ -0,0 +1,240 @@
+# README - OTP Cusom/Fork Release Scripts
+
+**Note! This describes how you can set up and release OTP in you own GitHub fork, not how OTP in the
+main repo is released.**
+
+## Introduction
+
+The scripts here can be used to release a fork of OTP. Run
+
+```
+# script/custom-release.py otp/dev-2.x
+```
+
+You can customize the release with:
+
+ - `script/custom-release-env.json` Requiered config for the main script.
+ - `script/custom-release-extension` Optional script executed by the main scrit if it exist.
+
+
+### Process overview
+
+The release is made in the `release branch` in the forked git repository. The release is created 
+based on the `base revision` - a branch, tag or commit, for example `otp/dev-2.x`. 
+
+1. The release script will start by _resetting_ the `release branch` to the given `base revision`.
+  **Nothing is kept from previous releases.**
+2. Then all pending PRs tagged with your `TEST` label is meged in. The name of the label is 
+  configured in the `custom-release-env.json` file.
+3. Then all your extention branches are meged in. These are normally branchech with your deployment
+  config and GitHub workflow files. The name of these branches is configured in the 
+  `custom-release-env.json` file.
+4. The `custom-release-extention` script is run, if it exist. For example you may delete 
+  workflow scripts comming from the upstream `base revision`. 
+5. The old release is then merged with an _empty merge_, this is done to create a continuous line
+  of releases in the release branch for easy viewing and navigation of the git history. Nothing
+  from the previous release is copied into the new release. We call this an _empty merge_.
+6. Then the script update the _otp-serialization-version-id_ and OTP _version_. Each release is 
+  given a unique version number specific to your fork, like `v2.7.0-MY_ORG-1`. The release script
+  uses both the git history and the GitHub GraphQL API (PRs labeled `bump serialization id`) to 
+  resolve the versions numbers. 
+7. The script finally push the release.
+
+Do not worry about deleting more recent versions, the release script will preserve the history so
+nothing is lost.
+
+> Tip! If something goes wrong, like a conflicting merge, you may fix the problem and resume the
+> release by running the release script again. The script will detect that it failed and ask you
+> what to do.
+
+The release script also support maining a hotfix (`--hotfix`). Simply make the needed changes 
+to any previous release. For example merge in bug-fix branches. Complete the process by running
+`script/custom-release-extension --hotfix`.
+
+
+### Advanced Git flow - roll back
+
+![Release git flow](images/release-git-flow.svg)
+
+Each release has a single commit as a base, this allows us to choose any commit as the base and
+safely **Roll-back, by rolling forward**. In the diagram above, commit (D) contained a bug. So, we
+can go back to commit (C). If we merged `dev2-x` into the release-branch then going back to this
+commit would be challenging since commit (D) was already in the release branch at this point. 
+Also note that the CI configuration is changed (c2) and this change needs to be included in the
+new release (v3). 
+
+> **Note!** OTP ignores config it does not understand it. This allows us to roll out config for new 
+> features BEFORE we roll out the new version of OTP. We can then verify that the old version of 
+> OTP works with the new config. Config is dependent on both the environment OTP run in and OTP. 
+> So, the config has its own life-cycle, independent of the OTP software.
+
+
+### Advanced Git flow - rollback and a feature branch
+
+![Release git flow with feature branch](images/release-git-flow-feature-branch.svg)
+
+When you make a new releae all open PRs tagged with your custom `TEST` label is automatically
+merged in. This is ilustrated with the _feature-branch-in-progress_ above. 
+
+## Extention branches
+
+You should create one or more branches in the local git repository where you keep your 
+deployment-specific config. Put the following in this(these) branch(es):
+
+The branch(es) must include(requiered):
+
+- The `script/custom-release-env.json`, containing the release configuration. The branch containing
+  this file must be merged into the `release brach` before the first release. If you make changes
+  to the config the branch must be merged into main again before the changes take  effect. You can
+  do this by making a release(last release with old config), and then a new release - or just merg
+  in the config before you run the script. If the release run in a CI/CD pipline the easiest is to
+  just trigger two new releases.
+
+The branch may include(optional):
+
+- Local CI build configuration.
+- OTP deployment configuration.
+- `script/custom-release-extension`, this is your custom script executed by the main  release
+  script if it exist. At entur we delete workflows from the upstream project and move some of the
+  configuration into the right place. We have all over config in a single folder to be able to
+  maintain it in a simple way and avoid merge conflicts.
+
+The config branches are merged into the release - so the best way to avoid merge conflict is to
+use a "old" commit in the **base branch/repo** as the base for your config. If the base commit
+exist in the target base revision used to create a new release, then you are guaranteed to avoid
+any merge conflicts - unless you change something - adding new files are safe.
+
+
+## Setup
+
+Create a extention branch (`main_config`) in your local fork based on a commit in the target
+repo(for example `HEAD` of `opentripplanner/OpenTripPlanner/dev-2.x`).
+
+Add the `script/custom-release-env.json` file with you config, like this:
+
+```json
+{
+  "upstream_remote": "otp",
+  "release_remote": "<organization name>",
+  "release_branch": "<release branch>",
+  "ser_ver_id_prefix": "<organization abbrivation, max 2 characters>",
+  "include_prs_label": "<organization name> Test",
+  "ext_branches" : [ "<release config branch>"]
+}
+```
+If you organization is _Curium_, then the file would look like this:
+```json
+{
+  "upstream_remote": "otp",
+  "release_remote": "curium",
+  "release_branch": "main",
+  "ser_ver_id_prefix": "CU",
+  "include_prs_label": "Curium Test",
+  "ext_branches" : [ "main_config"]
+}
+```
+
+The `<organization name>` must match the GitHub organization name in your local Git clone.
+Use `git remote -v` to list all remote repos. 
+
+```
+# git remote -v 
+curium	https://github.com/curium/OpenTripPlanner.git (fetch)
+curium	https://github.com/curium/OpenTripPlanner.git (push)
+otp	https://github.com/opentripplanner/OpenTripPlanner.git (fetch)
+otp	https://github.com/opentripplanner/OpenTripPlanner.git (push)
+```
+
+Avoid using the default `origin`, instead rename your repositories to avoid mistakes.
+
+```
+# git remote rename origin otp 
+```
+
+
+### Pending Pull Requests
+
+You may want to merge in some of the pending PRs when releasing your fork. To do so you can add a
+label to these PRs and specify the label in the _script/custom-release-env.json_ file. At Entur we
+label all PRs we want to be merged into the next release with `Entur Test`, this ensures that any
+team member can do the release. This allows us to test features at Entur before the PR is accepted
+and merged in the upstream repo. We combine this with config, and sometimes the OTPFeature toggle
+to turn _on_ new features in over test environment. When the new feature is tested ok, we can 
+enable it by changing the config. 
+
+
+## How To Make A Release
+
+Find the target branch/commit or use the branch `otp/dev-2.x`. If you use a specific branch/commit, 
+then replace `otp/dev-2.x` below with your branch/commit hash.
+
+Run the `script/custom-release.py` script:
+```
+# script/custom-release.py otp/dev-2.x
+```
+
+Use `script/custom-release.py --help` to list documentation and print options.
+
+The `--dryRun` options is used to run the script and skip pushing changes to the remote repo. Be
+aware that the local git repo is changed, and you must manually revert all changes. The `--dryRun` 
+option is used to test the script.
+
+If the script fails to rebase/compile one of the extension branches or PRS, you can resolve the
+problem/conflict and resume the script by running it again. Remember to commit you changes before
+you rerun the script.
+
+> **Tip!** If you have conflicts in documentation files, then consider running the test. The tests
+> will regenerate the correct documentation file. After the test is run and new documentation is
+> generated you mark the conflict as resolved.
+
+If a conflic happens in the CI/CD pipline it is recomended to fix the branch causing the conflict.
+The conflict can normally be fixed by rebasing or merging the extention branches or PRs. When 
+
+
+## How-to make a hot-fix release 🔥
+
+Sometimes it is necessary to roll out a new fix as fast as possible and with minimum risk. You can 
+do this by applying the new “hot-fix” on top of the an old/latest release, and then make a new 
+release. A hot-fix release can normally be rolled out without waiting for new graphs to build, 
+since the serialization version number is the same.
+
+1. Find out what the current OTP version is.
+2. Check out the `release branch`, pull the latest version. You may have to reset it to the 
+   version in the production environment. 
+3. Cherry-pick or fix the problem. 
+4. Run tests.
+5. Complete the release by running the `script/custom-release.py --hotfix`. 
+
+## How-to print a changelog summary
+
+To add a summary in the CI/CD Pipline you can run the script standing at the tagged version you
+want to produce the summary for. This will list changes in the _changelog.md_ and all PRs merged
+into the current release. This will only compare the release with the previous version, if you want 
+compare with another version you will need to use the `changelog-diff.py` script. The changelog 
+script only make a diff on the changelog, it does not include the PRs.
+
+```
+# script/custom-release.py --summary
+```
+The report look like this:
+
+> # OTP Release Summary
+>
+> ## Version
+>
+> - New version/git tag: `2.7.0-entur-68`
+> - New serialization version: `EN-0087`
+> - Old serialization version: `EN-0087`
+>
+>
+> ## Pull Requests
+> These PRs are tagged with Entur Test.
+>
+> - Fix ClassCastExcaption in Itinerary mapper [#6455](https://github.com/opentripplanner/OpenTripPlanner/pull/6455) [`Entur Test`]
+> 
+> ## Changelog v2.7.0-entur-67 vs v2.7.0-entur-68
+>
+> ### Added PRs
+> - Disable OptimizeTransfers for via search temporarily. [#6449](https://github.com/opentripplanner/OpenTripPlanner/pull/6449)
+
+

script/changelog-diff.py

@@ -0,0 +1,42 @@
+#!/usr/bin/env python3
+
+import subprocess
+import sys
+
+if len(sys.argv) != 3:
+    print("Expected arguments: <git-ref-A> <git-ref-B>", file=sys.stderr)
+    exit(1)
+
+v1 = sys.argv[1]
+v2 = sys.argv[2]
+
+cmd_args = ['git', 'diff', v1, v2, '--no-ext-diff', '-U0', '-a',  '--no-prefix',  '--',  'doc/user/Changelog.md']
+p = subprocess.run(args=cmd_args, capture_output=True, text=True, timeout=20)
+
+if p.returncode:
+    print(p.stderr, file=sys.stderr)
+    print(f"\nERROR! Execution failed: {cmd_args}", file=sys.stderr)
+    exit(p.returncode)
+
+changes = p.stdout
+print(f"\n## Changelog {v1} vs {v2} \n")
+added = []
+removed = []
+
+for line in changes.splitlines(keepends=True):
+    if line.startswith("+- "):
+        added.append(line[1:-1])
+    elif line.startswith("-- "):
+        removed.append(line[1:-1])
+
+if len(added) == 0 and len(removed) == 0:
+    print(f"No changes!")
+elif len(added) > 0:
+    print(f"### Added PRs")
+    for line in added:
+        print(line)
+elif len(removed) > 0:
+    print(f"### Removed PRs")
+    for line in removed:
+        print(line)
+