Use factored out testing lib (helm#5231)

* [WIP][CI] Use factored out testing lib

* Incorporate upstream changes

* Update Docker image for CircleCI

* Update Docker image

* Update Docker image

* Align with upstream changes

* Remove dummy chart and bump a few charts for testing

* Fix Dockerfile

* Align with upstream changes

* Add summary

* Update Chart.yaml

* Update Docker images

* Remove chart test bumps

* Align with upstream changes

* Bump some charts for testing

* Create v2.0.0

* Remove chart version bumps

* Revert repo-sync.sh in favor of helm#5847

* Bump image version

* Use latest chart testing image

Author: unguiculus

.circleci/config.yml

@@ -1,16 +1,28 @@
 version: 2
 jobs:
-  build:
+  lint-scripts:
     docker:
-      - image: circleci/python:3.6.4
+      - image: koalaman/shellcheck-alpine
     steps:
       - checkout
       - run:
-          name: install tools
-          command: test/circle/install.sh
+          name: lint
+          command: |
+            shellcheck -x test/build.sh
+            shellcheck -x test/e2e.sh
+            shellcheck -x test/helm-test-e2e.sh
+            shellcheck -x test/repo-sync.sh
+  lint-charts:
+    docker:
+      - image: gcr.io/kubernetes-charts-ci/test-image:v2.0.2
+    steps:
+      - checkout
       - run:
           name: lint
-          command: test/circle/lint.sh
+          command: |
+            git remote add k8s https://github.com/kubernetes/charts
+            git fetch k8s master
+            chart_test.sh --config test/.testenv --no-install
   sync:
     docker:
       - image: google/cloud-sdk
@@ -21,9 +33,10 @@ jobs:
           command: test/repo-sync.sh
 workflows:
   version: 2
-  test:
+  lint:
     jobs:
-      - build
+      - lint-scripts
+      - lint-charts
   sync:
     triggers:
       - schedule:

.gitignore

@@ -32,4 +32,6 @@ Session.vim
 .netrwhist
 
 # Chart dependencies
-**/charts/*.tgz
+**/charts/*.tgz
+
+.history

test/.dockerignore

@@ -0,0 +1,2 @@
+*
+! .testenv

test/.testenv

@@ -0,0 +1,23 @@
+# The name of the Git remote
+REMOTE=k8s
+
+# The name of the Git target branch
+TARGET_BRANCH=master
+
+# Chart directories separated by a space
+CHART_DIRS=(
+    stable
+    incubator
+)
+
+# Charts that should be skipped
+EXCLUDED_CHARTS=(
+    common
+)
+
+# Additional chart repos to add (<name>=<url>), separated by a space
+CHART_REPOS=(
+    incubator=https://kubernetes-charts-incubator.storage.googleapis.com/
+)
+
+TIMEOUT=600

test/Dockerfile

@@ -1,4 +1,4 @@
-# Copyright 2016 The Kubernetes Authors All rights reserved.
+# Copyright 2018 The Kubernetes Authors. All rights reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -12,45 +12,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM debian:8.5
+FROM gcr.io/kubernetes-charts-ci/chart-testing:v1.0.2
 
-ENV CLOUDSDK_CORE_DISABLE_PROMPTS=1 \
-  PATH=/opt/google-cloud-sdk/bin:$PATH \
-  GOOGLE_CLOUD_SDK_VERSION=154.0.0 \
-  GOOGLE_PROJECT=kubernetes-charts-ci
+ENV PATH /google-cloud-sdk/bin:$PATH
+ARG CLOUD_SDK_VERSION=202.0.0
+RUN curl -LO "https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-$CLOUD_SDK_VERSION-linux-x86_64.tar.gz" && \
+    tar xzf "google-cloud-sdk-$CLOUD_SDK_VERSION-linux-x86_64.tar.gz" && \
+    rm "google-cloud-sdk-$CLOUD_SDK_VERSION-linux-x86_64.tar.gz" && \
+    ln -s /lib /lib64 && \
+    rm -rf /google-cloud-sdk/.install/.backup && \
+    gcloud version
 
-RUN set -x \
-  && cd /opt \
-  && echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections \
-  && apt-get update \
-  && apt-get install --no-install-recommends -y jq wget python git localepurge ca-certificates \
-  && wget -q https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-${GOOGLE_CLOUD_SDK_VERSION}-linux-x86_64.tar.gz \
-  && tar zxfv google-cloud-sdk-${GOOGLE_CLOUD_SDK_VERSION}-linux-x86_64.tar.gz \
-  && ./google-cloud-sdk/install.sh \
-  && gcloud components install kubectl \
-  && gcloud config set project ${GOOGLE_PROJECT}
-RUN apt-get -y purge localepurge \
-  && apt-get clean \
-  && cd / \
-  && rm -rf \
-     /opt/google-cloud-sdk-${GOOGLE_CLOUD_SDK_VERSION}-linux-x86_64.tar.gz \
-     /opt/helm-${HELM_VERSION}-linux-amd64.tar.gz \
-     doc \
-     man \
-     info \
-     locale \
-     /var/lib/apt/lists/* \
-     /var/log/* \
-     /var/cache/debconf/* \
-     common-licenses \
-     ~/.bashrc \
-     /etc/systemd \
-     /lib/lsb \
-     /lib/udev \
-     /usr/share/doc/ \
-     /usr/share/doc-base/ \
-     /usr/share/man/ \
-     /tmp/*
+RUN gcloud config set core/disable_usage_reporting true && \
+    gcloud config set component_manager/disable_update_check true && \
+    gcloud config set metrics/environment github_docker_image
 
-VOLUME "/src"
-WORKDIR "/src"
+WORKDIR /workdir

test/README.md

@@ -27,19 +27,10 @@ The configuration of the Pull Request trigger is [in the config.json](https://gi
 
 This snippet tells Test Infra to run the [test/e2e.sh](https://github.com/kubernetes/charts/blob/master/test/e2e.sh)
 when testing is triggered on a pull request. The e2e.sh script will use the [Charts test image](https://github.com/kubernetes/charts/blob/master/test/Dockerfile)
-to run the [test/changed.sh](https://github.com/kubernetes/charts/blob/master/test/changed.sh) script. This script
+to run the [chart_test.sh](https://github.com/kubernetes-helm/chart-testing/blob/master/chart_test.sh) script. This script
 is the main logic for validation of a pull request. It intends to only test charts that have changed in this PR.
 
-The logic is as follows:
-
-1. [Get credentials for the Kubernetes cluster used for testing.](https://github.com/kubernetes/charts/blob/master/test/changed.sh#L128)
-1. [Install and initialize Helm](https://github.com/kubernetes/charts/blob/master/test/changed.sh#L143)
-1. [For any charts that have changed](https://github.com/kubernetes/charts/blob/master/test/changed.sh#L161):
-    - Download dependent charts, if any, with `helm dep build`
-    - Run `helm install` in a new namespace for this PR build
-    - Use the [test/verify-release.sh](https://github.com/kubernetes/charts/blob/master/test/verify-release.sh) to ensure that if any pods were launched that they get to the `Running` state
-    - Run `helm test` on the release
-    - Delete the release
+The testing logic has been extrated to the [chart-testing](https://github.com/kubernetes-helm/chart-testing) project. A bash library provides the required logic to lint, install, and test charts. It is provided as a Docker image and can be run by anyone on their own charts.
 
 #### Providing Custom Test Values
 

test/build.sh

@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+
+# Copyright 2018 The Kubernetes Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+readonly IMAGE_TAG=v2.0.2
+readonly IMAGE_REPOSITORY="gcr.io/kubernetes-charts-ci/test-image"
+readonly SCRIPT_DIR=$(dirname "$(readlink -f "$0")")
+
+show_help() {
+cat << EOF
+Usage: ${0##*/} <args>
+    -h, --help             Display help
+    -v, --verbose          Display verbose output
+    -p, --push             Push image to registry
+EOF
+}
+
+main() {
+    local verbose=
+    local push=
+
+    while :; do
+        case "${1:-}" in
+            -h|--help)
+                show_help
+                exit
+                ;;
+            -v|--verbose)
+                verbose=true
+                ;;
+            -p|--push)
+                push=true
+                ;;
+            -?*)
+                printf 'WARN: Unknown option (ignored): %s\n' "$1" >&2
+                ;;
+            *)
+                break
+                ;;
+        esac
+
+        shift
+    done
+
+    [[ -n "$verbose" ]] && set -o xtrace
+
+    pushd "$SCRIPT_DIR"
+
+    docker build --tag "$IMAGE_REPOSITORY:$IMAGE_TAG" .
+
+    if [[ -n "$push" ]]; then
+        docker push "$IMAGE_REPOSITORY:$IMAGE_TAG"
+    fi
+
+    popd
+}
+
+main "$@"

test/changed.sh

@@ -1,5 +1,6 @@
-#!/bin/bash
-# Copyright 2016 The Kubernetes Authors All rights reserved.
+#!/usr/bin/env bash
+
+# Copyright 2018 The Kubernetes Authors. All rights reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,40 +17,39 @@
 set -o errexit
 set -o nounset
 set -o pipefail
-set -o xtrace
-
-# TODO should we inject this.  This is creating problems bumping the Docker version
-IMAGE_VERSION="test-image:v1.11"
-CHART_ROOT=${CHART_ROOT:-$(git rev-parse --show-toplevel)}
-IMAGE_NAME=${IMAGE_NAME:-"gcr.io/kubernetes-charts-ci/${IMAGE_VERSION}"}
-
-VOLUMES="-v ${CHART_ROOT}:/src -v ${KUBECONFIG:=${HOME}/.kube/config}:/.kube/config"
-
-GKE_CREDS=""
-if [ -f $HOME/.config/gcloud/application_default_credentials.json ];then
-  GKE_CREDS="-v $HOME/.config/gcloud/application_default_credentials.json:/service-account.json:ro"
-  GKE_CREDS="${GKE_CREDS} -e KUBECONFIG=/.kube/config"
-elif [ -n ${GOOGLE_APPLICATION_CREDENTIALS:=} ];then
-  GKE_CREDS="-v ${GOOGLE_APPLICATION_CREDENTIALS}:/service-account.json:ro"
-else
-  echo "Unable to find a suitable value for GOOGLE_APPLICATION_CREDENTIALS"
-  exit 1
-fi
+
+readonly IMAGE_TAG=v2.0.2
+readonly IMAGE_REPOSITORY="gcr.io/kubernetes-charts-ci/test-image"
+readonly REPO_ROOT="${REPO_ROOT:-$(git rev-parse --show-toplevel)}"
 
 # Pull numbers are only available on presubmit jobs. When the tests are run as
 # part of a batch job (e.g., batch merge) the PULL_NUMBER is not available. Pull
 # numbers are useful for some debugging. When no PULL_NUMBER is supplied we build
 # from other info.
-PULL_INFO=${PULL_NUMBER:-}
+PULL_INFO="${PULL_NUMBER:-}"
 if [ -z "$PULL_INFO" ]; then
-  PULL_INFO=${PULL_BASE_SHA}
+    PULL_INFO="${PULL_BASE_SHA:-}"
 fi
+readonly PULL_INFO
+
+main() {
+    git remote add k8s https://github.com/kubernetes/charts
+    git fetch k8s master
+
+    local config_container_id
+    config_container_id=$(docker run -ti -d -v "$GOOGLE_APPLICATION_CREDENTIALS:/service-account.json" -v "$REPO_ROOT:/workdir" \
+        -e "BUILD_ID=$JOB_TYPE-$PULL_INFO-$BUILD_ID" \
+        "$IMAGE_REPOSITORY:$IMAGE_TAG" cat)
+
+    # shellcheck disable=SC2064
+    trap "docker rm -f $config_container_id" EXIT
+
+    docker exec "$config_container_id" gcloud auth activate-service-account --key-file /service-account.json
+    docker exec "$config_container_id" gcloud container clusters get-credentials jenkins --project kubernetes-charts-ci --zone us-west1-a
+    docker exec "$config_container_id" kubectl cluster-info
+    docker exec "$config_container_id" chart_test.sh --config /workdir/test/.testenv
+
+    echo "Done Testing!"
+}
 
-docker run ${VOLUMES} ${GKE_CREDS} \
-           -e GOOGLE_APPLICATION_CREDENTIALS=/service-account.json \
-           -e "JOB_TYPE=$JOB_TYPE" \
-           -e "PULL_INFO=$PULL_INFO" \
-           -e "BUILD_ID=$BUILD_ID" \
-           -e "VERIFICATION_PAUSE=${VERIFICATION_PAUSE:=0}" \
-           ${IMAGE_NAME} /src/test/changed.sh
-echo "Done Testing!"
+main

test/circle/README.md

@@ -1,28 +1,41 @@
 #!/usr/bin/env bash
 
+# Copyright 2018 The Kubernetes Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 set -o errexit
 set -o nounset
 set -o pipefail
 set -o xtrace
 
 # Install and initialize helm/tiller
-HELM_URL=https://storage.googleapis.com/kubernetes-helm
-HELM_TARBALL=helm-v2.9.1-linux-amd64.tar.gz
+readonly HELM_URL=https://storage.googleapis.com/kubernetes-helm
+readonly HELM_TARBALL=helm-v2.9.1-linux-amd64.tar.gz
 
-wget -q ${HELM_URL}/${HELM_TARBALL} -P ${WORKSPACE}
-tar xzfv ${WORKSPACE}/${HELM_TARBALL} -C ${WORKSPACE}
+wget -q "$HELM_URL/$HELM_TARBALL" -P "$WORKSPACE"
+tar xzfv "$WORKSPACE/$HELM_TARBALL" -C "$WORKSPACE"
 
 # Housekeeping
+pushd "$WORKSPACE/linux-amd64"
+
 kubectl -n kube-system create sa tiller
 kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
-${WORKSPACE}/linux-amd64/helm init --service-account tiller --upgrade
+./helm init --service-account tiller --upgrade
+./helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
 
-${WORKSPACE}/linux-amd64/helm repo add stable https://kubernetes-charts.storage.googleapis.com/
-${WORKSPACE}/linux-amd64/helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
+popd
 
 # Run test framework
-pushd .
-cd $GOPATH
 go get github.com/ghodss/yaml
-popd
-go run $GOPATH/src/k8s.io/charts/test/helm-test/main.go
+go run "$GOPATH/src/k8s.io/charts/test/helm-test/main.go"