|
| 1 | +--- |
| 2 | +layout: blog |
| 3 | +title: "Automatically update README.md with Travis-CI" |
| 4 | +date: "2018-07-28" |
| 5 | +author: ["Siang Lim"] |
| 6 | +--- |
| 7 | + |
| 8 | +While working on content for the [UBC OpenChemE Initiative](https://opencheme.github.io), I got tired of manually updating the README.md file with new notebooks and decided to look for a more elegant solution. |
| 9 | + |
| 10 | +# Rewriting README.md with new links |
| 11 | +The goal is to update `README.md` automatically with links to new Jupyter notebooks whenever we make a commit on GitHub. The readme file feeds into Jekyll/GitHub Pages and generates the website that we see [here](https://opencheme.github.io/CHBE356/). |
| 12 | + |
| 13 | +We'll do this with a bash script that collects the names of all .ipynb files in the Notebooks folder and dump the contents into README.md with the right markdown formatting and nbviewer links. We'll use Travis CI to run the script every time we make a commit on GitHub. |
| 14 | + |
| 15 | +# Bash script |
| 16 | +## Skip to [this section](#final-bash-script) if you don't want the details |
| 17 | +We'll start by writing a bash script to grab all directories (`*`) in the Notebooks folder, one level up (`..`). Save this script as `deploy.sh` in a folder called `scripts` in your main project folder. |
| 18 | + |
| 19 | + |
| 20 | +```bash |
| 21 | +for d in ../Notebooks/* ; do |
| 22 | + echo "$d" |
| 23 | + # Do something with files in this folder |
| 24 | +done |
| 25 | +``` |
| 26 | + |
| 27 | +If you run that, you'll see that `$d` prints out the entire path and not just the directory name. After some Googling, I found out how to split the base paths and the name using [parameter expansion](# https://stackoverflow.com/questions/3362920/get-just-the-filename-from-a-path-in-a-bash-script). |
| 28 | + |
| 29 | +```bash |
| 30 | +for d in ../Notebooks/* ; do |
| 31 | + xpath=${d%/*} |
| 32 | + xbase=${d##*/} |
| 33 | + xfext=${xbase##*.} |
| 34 | + xpref=${xbase%.*} |
| 35 | + |
| 36 | + # Do something with files in this folder |
| 37 | +done |
| 38 | +``` |
| 39 | + |
| 40 | +Now, we want not only the directory, but also all notebook files in a particular directory, so we'll do something like this to loop through the files: |
| 41 | + |
| 42 | +```bash |
| 43 | +for d in ../Notebooks/* ; do |
| 44 | + xpath=${d%/*} |
| 45 | + xbase=${d##*/} |
| 46 | + xfext=${xbase##*.} |
| 47 | + xpref=${xbase%.*} |
| 48 | + |
| 49 | + echo "$d" |
| 50 | + |
| 51 | + for f in "$d"/*.ipynb; do |
| 52 | + fpath=${f%/*} |
| 53 | + fbase=${f##*/} |
| 54 | + ffext=${fbase##*.} |
| 55 | + fpref=${fbase%.*} |
| 56 | + |
| 57 | + echo "$f" |
| 58 | + done |
| 59 | + |
| 60 | +done |
| 61 | +``` |
| 62 | + |
| 63 | +If you run the script and look at the echo outputs, you'll see that we are getting close. |
| 64 | + |
| 65 | +We want to output the directories as Markdown `h1` headings and the files as bullet points. We also want a space between each heading for readibility. Finally, we want to write all this to the README.md file in the root directory. |
| 66 | + |
| 67 | +So here's what we'll do: |
| 68 | + |
| 69 | +```bash |
| 70 | + |
| 71 | +readme_path="../README.md" |
| 72 | + |
| 73 | +for d in ../Notebooks/* ; do |
| 74 | + xpath=${d%/*} |
| 75 | + xbase=${d##*/} |
| 76 | + xfext=${xbase##*.} |
| 77 | + xpref=${xbase%.*} |
| 78 | + |
| 79 | + echo "# $xbase" >> "$readme_path" |
| 80 | + |
| 81 | + for f in "$d"/*.ipynb; do |
| 82 | + fpath=${f%/*} |
| 83 | + fbase=${f##*/} |
| 84 | + ffext=${fbase##*.} |
| 85 | + fpref=${fbase%.*} |
| 86 | + |
| 87 | + echo "* $fpref" >> "$readme_path" |
| 88 | + done |
| 89 | + |
| 90 | + echo -e "\n" >> "$readme_path" |
| 91 | + |
| 92 | +done |
| 93 | + |
| 94 | +``` |
| 95 | + |
| 96 | +For the individual files, we want to link to nbviewer directly, and convert spaces to `%20` in the URL. Here's how to do it. The `//` in `${xbase// /%20}` means change all spaces ` ` to `%20`. |
| 97 | + |
| 98 | + |
| 99 | +```bash |
| 100 | + |
| 101 | +readme_path="../README.md" |
| 102 | +nbviewer_path="http://nbviewer.jupyter.org/github/OpenChemE/CHBE356/blob/master/Notebooks" |
| 103 | + |
| 104 | +for d in ../Notebooks/* ; do |
| 105 | + xpath=${d%/*} |
| 106 | + xbase=${d##*/} |
| 107 | + xfext=${xbase##*.} |
| 108 | + xpref=${xbase%.*} |
| 109 | + |
| 110 | + echo "# $xbase" >> "$readme_path" |
| 111 | + |
| 112 | + for f in "$d"/*.ipynb; do |
| 113 | + fpath=${f%/*} |
| 114 | + fbase=${f##*/} |
| 115 | + ffext=${fbase##*.} |
| 116 | + fpref=${fbase%.*} |
| 117 | + |
| 118 | + echo "* [$fpref]($nbviewer_path/${xbase// /%20}/${fbase// /%20})" >> "$readme_path" |
| 119 | + done |
| 120 | + |
| 121 | + echo -e "\n" >> "$readme_path" |
| 122 | + |
| 123 | +done |
| 124 | + |
| 125 | +``` |
| 126 | + |
| 127 | +# Final bash script |
| 128 | +Almost there. We want to also create a separate 'header' file for our README.md and append the links below it. We will also add `shopt -s nullglob` in case we have [empty folders](https://unix.stackexchange.com/questions/239772/bash-iterate-file-list-except-when-empty |
| 129 | +). Here's the final file: |
| 130 | + |
| 131 | +```bash |
| 132 | +#!/bin/bash |
| 133 | + |
| 134 | +# a pattern that matches nothing "disappears", rather than treated as a literal string: |
| 135 | +# https://unix.stackexchange.com/questions/239772/bash-iterate-file-list-except-when-empty |
| 136 | +shopt -s nullglob |
| 137 | + |
| 138 | +# Our paths for the readme file |
| 139 | +header_path="../header.md" |
| 140 | +readme_path="../README.md" |
| 141 | +nbviewer_path="http://nbviewer.jupyter.org/github/OpenChemE/CHBE356/blob/master/Notebooks" |
| 142 | + |
| 143 | +# Copy the header over and add a blank line |
| 144 | +cat "$header_path" > "$readme_path" |
| 145 | +echo -e "\n" >> "$readme_path" |
| 146 | + |
| 147 | +# https://stackoverflow.com/questions/3362920/get-just-the-filename-from-a-path-in-a-bash-script |
| 148 | +for d in ../Notebooks/* ; do |
| 149 | + xpath=${d%/*} |
| 150 | + xbase=${d##*/} |
| 151 | + xfext=${xbase##*.} |
| 152 | + xpref=${xbase%.*} |
| 153 | + |
| 154 | + echo "# $xbase" >> "$readme_path" |
| 155 | + |
| 156 | + for f in "$d"/*.ipynb; do |
| 157 | + fpath=${f%/*} |
| 158 | + fbase=${f##*/} |
| 159 | + ffext=${fbase##*.} |
| 160 | + fpref=${fbase%.*} |
| 161 | + |
| 162 | + echo "* [$fpref]($nbviewer_path/${xbase// /%20}/${fbase// /%20})" >> "$readme_path" |
| 163 | + done |
| 164 | + |
| 165 | + echo -e "\n" >> "$readme_path" |
| 166 | + |
| 167 | +done |
| 168 | +``` |
| 169 | + |
| 170 | +# Travis CI |
| 171 | +We're all done with the bash script at this point. Now for the Travis CI part. Travis CI is integrated with GitHub. The service allows us to run scripts every time we make a commit on GitHub. These scripts could be unit tests or any bash script. |
| 172 | + |
| 173 | +I didn't really understand the value of CI services like Travis CI or Circle CI until I had to implement unit tests for the [UBC Envision](https://www.ubcenvision.com) site to prevent our team members from unintentionally(?) breaking it with bad commits *(to be covered in a separate post)*. |
| 174 | + |
| 175 | +## Set up tokens |
| 176 | +We'll need an authentication token from GitHub to allow Travis to make changes to the repository. Run this bash command and save the value of the `token` key on your screen: |
| 177 | + |
| 178 | +```bash |
| 179 | +$ curl -u csianglim -d '{"scopes":["public_repo"],"note":"CI"}' https://api.github.com/authorizations |
| 180 | +``` |
| 181 | + |
| 182 | +Navigate to your project folder and install the `travis` gem |
| 183 | + |
| 184 | +``` |
| 185 | +$ gem install travis |
| 186 | +``` |
| 187 | + |
| 188 | +Make sure you're in your project folder, then run this command to encrypt the token. Travis will automatically add it to your .travis.yml file: |
| 189 | + |
| 190 | +``` |
| 191 | +$ travis encrypt GIT_NAME="Travis CI" --add |
| 192 | +$ travis encrypt GIT_EMAIL="[email protected]" --add |
| 193 | +$ travis encrypt GH_TOKEN=<token> --add |
| 194 | +``` |
| 195 | + |
| 196 | +Here's what my .travis.yml file looks like: |
| 197 | + |
| 198 | +``` |
| 199 | +language: ruby |
| 200 | +script: |
| 201 | +- bash ./scripts/deploy.sh |
| 202 | +branches: |
| 203 | + only: |
| 204 | + - master |
| 205 | +env: |
| 206 | + global: |
| 207 | + - secure: <encrypted_token> |
| 208 | + - secure: <encrypted_token> |
| 209 | + - secure: <encrypted_token> |
| 210 | +``` |
| 211 | + |
| 212 | +Now we just need add a few more lines to our `deploy.sh` script and tell Travis to set up git and push to the repo: |
| 213 | + |
| 214 | +``` |
| 215 | +# Setup and push to master |
| 216 | +git config --global user.email ${GIT_EMAIL} |
| 217 | +git config --global user.name ${GIT_NAME} |
| 218 | +git checkout master |
| 219 | +git add . |
| 220 | +git commit --message "Travis $TRAVIS_BUILD_NUMBER: $TRAVIS_COMMIT_MESSAGE" |
| 221 | +git remote set-url origin https://${GH_TOKEN}@github.com/OpenChemE/CHBE356.git |
| 222 | +git push origin master |
| 223 | +``` |
| 224 | + |
| 225 | +# Conclusion |
| 226 | +I am a big advocate of automation to avoid doing any repetitive and tedious work. Reducing unnecessary human input will allow us to spend time on more important and challenging problems, create consistent workflows for all team members and reduce human errors in the final results. |
0 commit comments