[DX-1780]Generate test for tyk gateway swagger (#6827)

# OpenAPI Specification Contract Testing with Portman To ensure that the schema in our gateway's OpenAPI specification matches our API, we have implemented contract tests using Portman. Portman leverages Postman collections to automatically generate contract tests from our OpenAPI specifications. ### What the Contract Tests Are Testing: 1. **Request Body Validation**: Verifying that the request body defined in our Swagger schema matches the request body expected by our gateway API. 2. **Response Body Validation**: Ensuring that the response returned by our gateway API matches the response body defined in our Swagger schema. 3. **Content-Type Validation**: Confirming that the content types defined in the request and response schemas in our Swagger documentation match those expected and returned by our gateway API. 4. **Header Validation**: Validating that the headers sent to or returned by the gateway API are as described in our Swagger documentation. Full documentation [can be found here](https://github.com/TykTechnologies/tyk/blob/5540baeb205a9466d661b49ecf19792fa6491f99/ci/tests/schema/specs/README.md) <details open> <summary><a href="https://tyktech.atlassian.net/browse/DX-1780" title="DX-1780" target="_blank">DX-1780</a></summary> <br /> <table> <tr> <th>Summary</th> <td>Create contract tests for the tyk gateway</td> </tr> <tr> <th>Type</th> <td> <img alt="Story" src="https://tyktech.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10315?size=medium" /> Story </td> </tr> <tr> <th>Status</th> <td>In Progress</td> </tr> <tr> <th>Points</th> <td>N/A</td> </tr> <tr> <th>Labels</th> <td>-</td> </tr> </table> </details>  --- ### **User description** [DX-1780]  ## Description  ## Related Issue     ## Motivation and Context  ## How This Has Been Tested     ## Screenshots (if appropriate) ## Types of changes  - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Refactoring or add test (improvements in base code or adds test coverage to functionality) ## Checklist    - [ ] I ensured that the documentation is up to date - [ ] I explained why this PR updates go.mod in detail with reasoning why it's required - [ ] I would like a code coverage CI quality gate exception and have explained why ___ ### **PR Type** Tests, Enhancement ___ ### **Description** - Added a pre-request script for Tyk Gateway reload. - Configured Portman CLI for OpenAPI specification testing. - Introduced test cases for API and policy management. - Updated Swagger schema to allow empty string values. ___ ### **Changes walkthrough** 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>prescript.js</strong><dd><code>Add pre-request script for Tyk Gateway reload</code>                        </dd></summary> <hr> ci/tests/schema/specs/config/prescript.js <li>Added a script to reload Tyk Gateway using API key.<br> <li> Included error handling for missing API key.<br> <li> Logged responses from the Tyk Gateway reload endpoint. </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6827/files#diff-d5d32e1394f9dd0559de91dabec5b31c9a90d8e4a3f9e5db2aff9b05f82bed43">+27/-0</a>    </td> </tr> <tr> <td> <details> <summary><strong>swagger.yml</strong><dd><code>Update Swagger schema to allow empty strings</code>                          </dd></summary> <hr> swagger.yml <li>Allowed empty string values for execution mode.<br> <li> Updated version enum to include empty string. </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6827/files#diff-8f3c4cb253eee09ae2401daa7279a8bbfbfd4168bb579c3ac0ee5c672d63bb2c">+2/-1</a>      </td> </tr> </table></td></tr><tr><td><strong>Configuration changes</strong></td><td><table> <tr> <td> <details> <summary><strong>Taskfile.yml</strong><dd><code>Configure tasks for OpenAPI specification tests</code>                    </dd></summary> <hr> ci/tests/schema/specs/Taskfile.yml <li>Added tasks for running OpenAPI specification tests.<br> <li> Configured Venom and npm commands for testing. </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6827/files#diff-51702000b45f913d32d2ae4db64c31e5cbf915e241182975f938c2ad0b21ea1f">+9/-0</a>      </td> </tr> <tr> <td> <details> <summary><strong>portman-cli-options.json</strong><dd><code>Add Portman CLI configuration for testing</code>                                </dd></summary> <hr> ci/tests/schema/specs/config/portman-cli-options.json <li>Configured Portman CLI options for OpenAPI testing.<br> <li> Enabled test inclusion and Newman execution.<br> <li> Set up environment file and output paths. </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6827/files#diff-d8baf6a4141e957d52ea1f3e8589c3744e2210973ba8af02556f9d82502c1ae8">+12/-0</a>    </td> </tr> <tr> <td> <details> <summary><strong>portmanconfig.json</strong><dd><code>Define Portman configuration for OpenAPI tests</code>                      </dd></summary> <hr> ci/tests/schema/specs/config/portmanconfig.json <li>Defined contract tests for OpenAPI operations.<br> <li> Added variable assignments for API operations.<br> <li> Configured pre-request scripts and request overwrites.<br> <li> Included global settings for test execution order. </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6827/files#diff-129f9b9eb8a9bbbf3d6519a2ecee6f5d4d65e97ff4618ba99a7e8e30a6f29263">+144/-0</a>  </td> </tr> </table></td></tr><tr><td><strong>Dependencies</strong></td><td><table> <tr> <td> <details> <summary><strong>package.json</strong><dd><code>Add Portman dependency and npm script</code>                                        </dd></summary> <hr> ci/tests/schema/specs/package.json <li>Added Portman dependency for OpenAPI testing.<br> <li> Configured npm start script for Portman execution. </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6827/files#diff-bfba39305243a3cfcb0dd9e2c659f32326dbdca20e8f342f74f85725fd8d328e">+16/-0</a>    </td> </tr> </table></td></tr><tr><td><strong>Tests</strong></td><td><table> <tr> <td> <details> <summary><strong>populate_gateway_test_data.yaml</strong><dd><code>Add test cases for API and Gateway management</code>                        </dd></summary> <hr> ci/tests/schema/specs/testdata/populate_gateway_test_data.yaml <li>Added test cases for API creation and deletion.<br> <li> Included a test case for Tyk Gateway reload.<br> <li> Configured assertions for HTTP responses. </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6827/files#diff-e3a75bb64613130bff9f2ba7fa867ab0f8d2fb1a4a95403c9e2a5f92802ffd60">+61/-0</a>    </td> </tr> </table></td></tr></tr></tbody></table> ___ > 💡 **PR-Agent usage**: Comment `/help "your question"` on any pull request to receive relevant information [DX-1780]: https://tyktech.atlassian.net/browse/DX-1780?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: itachi sasuke <[email protected]> Co-authored-by: Tit Petric <[email protected]>
TykTechnologies · Feb 7, 2025 · e0b12d7 · e0b12d7
1 parent e23ca82
commit e0b12d7
Show file tree

Hide file tree

Showing 15 changed files with 810 additions and 2 deletions.
diff --git a/.github/workflows/release-tests.yml b/.github/workflows/release-tests.yml
@@ -39,6 +39,13 @@ jobs:
       - id: ecr-login
         uses: ./.github/actions/ecr-login
 
+      - name: Setup CI Tooling
+        uses: shrink/actions-docker-extract@v3
+        with:
+          image: tykio/ci-tools:latest
+          path: /usr/local/bin/.
+          destination: /usr/local/bin
+
       - name: Run /ci/tests
         shell: bash
         env:

diff --git a/.gitignore b/.gitignore
@@ -65,3 +65,10 @@ tyk_linux_*
 main
 
 /coprocess/*.pb.go-e
+ci/tests/specs/tmp
+ci/tests/specs/node_modules
+ci/tests/specs/package-lock.json
+ci/tests/specs/gateway.collection.postman.json
+ci/tests/specs/.env
+ci/tests/specs/apps
+ci/tests/specs/policies
diff --git a/ci/tests/specs/.env.example b/ci/tests/specs/.env.example
@@ -0,0 +1,2 @@
+PORTMAN_API_Key=example_gateway_secret
+GATEWAY_IMAGE=internal/tyk-gateway
diff --git a/ci/tests/specs/README.md b/ci/tests/specs/README.md
@@ -0,0 +1,55 @@
+# OpenAPI Specification Contract Testing with Portman
+
+To ensure that the schema in our gateway's OpenAPI specification matches our API, we have implemented contract tests using Portman. Portman leverages Postman collections to automatically generate contract tests from our OpenAPI specifications.
+
+## What the Contract Tests Are Testing:
+
+1. **Request Body Validation**: Verifying that the request body defined in our Swagger schema matches the request body expected by our gateway API.
+
+2. **Response Body Validation**: Ensuring that the response returned by our gateway API matches the response body defined in our Swagger schema.
+
+3. **Content-Type Validation**: Confirming that the content types defined in the request and response schemas in our Swagger documentation match those expected and returned by our gateway API.
+
+4. **Header Validation**: Validating that the headers sent to or returned by the gateway API are as described in our Swagger documentation.
+
+## Overview of the Included Files
+
+1. **config/portman-cli-options.json**: This file sets the configuration for running the Portman CLI (similar to `tyk.conf`). It also specifies the location of your `swagger.yml` file. Full configuration options can be found [here](https://github.com/apideck-libraries/portman#cli-usage).
+
+2. **config/portmanconfig.json**: [This file controls](https://github.com/apideck-libraries/portman#portman-settings) the content and how Portman generates the Postman tests. It specifies the types of tests to generate and any values to override in the generated tests. For example, to override headers or body content in the generated Postman tests, declare the overriding values in this file.
+
+3. **package.json**: Since Portman uses Node.js, this file specifies the Portman version to install and the command to run Portman.
+
+4. **testdata/populate_gateway_test_data.yaml**: A Venom test file that populates the gateway with data needed for the tests.
+
+**The following file is generated after running the tests:**
+
+- **gateway.collection.postman.json**: Contains the tests generated by Portman that Newman will execute.
+
+## How to Run It Locally
+
+To run the tests locally, install the following:
+
+1. [Venom](https://github.com/ovh/venom): Used to populate data needed for the Portman tests.
+
+2. [Portman](https://github.com/apideck-libraries/portman): Generates the contract tests from the Swagger file and converts them into Postman tests.
+
+3. [Newman](https://github.com/postmanlabs/newman): A Postman CLI tool needed to run the contract tests.
+
+Once these dependencies are installed, navigate to the `ci/tests/specs` directory and create a `.env` file using .env.example as template. Inside the `.env` file, add:
+
+```bash
+PORTMAN_API_Key=<Your Tyk Gateway secret>
+```
+
+After adding the `PORTMAN_API_Key`, run the `task` command from ci/tests/specs directory to run a gateway instance and execute portman tests.
+
+You can then stop the gateway instance by running `task down` command from the ci/tests/specs directory.
+
+## How It Is Run on the CI
+
+The GitHub Action used to run these tests is `swagger-contract-tests.yml`.
+
+- In the CI environment, we launch a live gateway using the using th image created by release.yml GitHub action on every pull request.
+
+- After that, we run the task `task tests`, which uses Portman to generate and execute all the Swagger contract tests.
diff --git a/ci/tests/specs/Taskfile.yml b/ci/tests/specs/Taskfile.yml
@@ -0,0 +1,37 @@
+version: "3"
+
+dotenv: ['.env']
+
+tasks:
+  default:
+    desc: "Run the portman tests"
+    cmds:
+      - task: up
+      - task: tests
+
+  tests:
+    desc: "Run the OpenAPI specification tests"
+    cmds:
+      - venom run testdata/populate_gateway_test_data.yaml --var bearerToken=$PORTMAN_API_Key  --stop-on-failure && rm venom*.log
+      - npm install
+      - npm start
+
+  build:
+    desc: "Build docker image"
+    cmds:
+      - docker compose build
+
+  up:
+    desc: "Bring up env"
+    cmds:
+      - docker compose up -d --wait --force-recreate || { docker compose logs gw; exit 1; }
+
+  down:
+    desc: "Shut down env"
+    cmds:
+      - docker compose down --remove-orphans
+
+  logs:
+    desc: "Tail container logs (live)"
+    cmds:
+      - docker compose logs --tail=10 -f
diff --git a/ci/tests/specs/config/certs.js b/ci/tests/specs/config/certs.js
@@ -0,0 +1,32 @@
+// certificate content
+const certContent = `
+-----BEGIN CERTIFICATE-----
+MIIDxDCCAqygAwIBAgIgEzBjeD6zmb42Re6D1cn9z2iewlkMr7nWwZkxFYphDZ8w
+DQYJKoZIhvcNAQEFBQAwcjETMBEGA1UEBhMKUGVhY2h0cmVlIDEMMAoGA1UECgwD
+dHlrMQwwCgYDVQQLDAN0eWsxDzANBgNVBAMMBnR5ay5pbzEdMBsGCSqGSIb3DQEJ
+ARYOc3VwcG9ydEB0eWsuaW8xDzANBgNVBAMMBnR5ay5pbzAeFw0yNDAzMjUwODQ2
+MzdaFw0zNDAzMjYwODQ2MzdaMGExEzARBgNVBAYTClBlYWNodHJlZSAxDDAKBgNV
+BAoMA3R5azEMMAoGA1UECwwDdHlrMQ8wDQYDVQQDDAZ0eWsuaW8xHTAbBgkqhkiG
+9w0BCQEWDnN1cHBvcnRAdHlrLmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
+CgKCAQEAjXjBHRJvcrzO8MIY2impRNRMhWoYlkKM5KIzQiyNxXJSkgXd9CoROCMr
+rQdRPusmFKONvorLRw1JEVhfMiEBG+Vmpmi07zK/STBJXdPNPCBjvaHTbDIx6git
+ezM+aKIDdnYoMDIx9T7zOukzW7AxzZ566oJETNgsz4lcleAwempqRndT91Nf5A97
+QAEL+LnDBpInE2rr36pAKxDRsMAaaBFpgM+tF2+6jd0Sh2gQCA904ZqeOOum7IqU
+KGOfYEQVjicTr+TF6V5N5EVw71jwOZt9WPO7lOh7wRmdgNp0MRFVUp1rITiIYstv
+AwdWEc7nbIj/VhMbAvX1za5hzlSNpwIDAQABo1cwVTAdBgNVHQ4EFgQUpNkYJfJr
+H5b97SmpTzdPjTGDYbMwHwYDVR0jBBgwFoAUpNkYJfJrH5b97SmpTzdPjTGDYbMw
+EwYDVR0RBAwwCoIILip0eWsuaW8wDQYJKoZIhvcNAQEFBQADggEBAIocI/ZqtXyS
+j/oI+Vlh8v/4RPzFHHCCBo8JWrSavYnWdkLl3nFF1pFaXvemlTGgUGL3c8GlujO9
+qiCKtza9XRdPPlZhbUOSTRZ0ck/KEQBZDar6juf2Ye2OgJhzeSsLyXQHD8Co6nGf
+4HE41MmhKSAUm+QIZMFRS98EkARkmtqhqehYhTvaKpvzEaRbMW+y5Dh1f6y0bYos
+Vx4NZLOF1C3/eEZywHJaPAtkPxjlhRVmgjJLDqEaj+DzwFYK2RMbRHCh8LKK8GIS
+CWeLM/p2ROlKFup25txezId1cmrhSP4m4djAX6IkgZSy1tDv3UmQQpPIrc2zdsmb
+4KKpTyHqfYo=
+-----END CERTIFICATE-----
+`;
+
+// Set the request body to the certificate content
+pm.request.body.update({
+  mode: 'raw',
+  raw: certContent
+});
diff --git a/ci/tests/specs/config/portman-cli-options.json b/ci/tests/specs/config/portman-cli-options.json
@@ -0,0 +1,12 @@
+{
+  "local": "../../../swagger.yml",
+  "baseUrl": "http://localhost:8080",
+  "output": "gateway.collection.postman.json",
+  "oaOutput": "filtered.gateway.swagger.yml",
+  "includeTests": true,
+  "syncPostman": false,
+  "runNewman": true,
+  "envFile": ".env",
+  "ignoreCircularRefs": true,
+  "portmanConfigFile":"config/portmanconfig.json"
+}
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		PORTMAN_API_Key=example_gateway_secret
		GATEWAY_IMAGE=internal/tyk-gateway