Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run conformance tests in CI #90

Open
dleehr opened this issue Jul 24, 2019 · 0 comments
Open

Run conformance tests in CI #90

dleehr opened this issue Jul 24, 2019 · 0 comments

Comments

@dleehr
Copy link
Member

dleehr commented Jul 24, 2019

The conformance tests should run automatically under a CI platform, like the other implementations listed on https://ci.commonwl.org.

Calrissian uses CircleCI for running automated tests. Through much trial and error I was able to get a minikube cluster running in CircleCI (via https://circleci.com/orbs/registry/orb/circleci/kubernetes#usage-deployment) and attempt to run the tests/collect artifacts. See branch try-circle-minikube.

Two issues are blocking progress:

  1. The CircleCI VM only has 2 CPUs, leaving <2CPUs for kubernetes pods. Some CWL conformance tests (including N1) require 2CPUs, so these pods get stuck waiting. I couldn't increase this limit or force it to a higher value
  2. Extracting the results/artifacts requires some additional resource creation/cleanup. Currently the junit XML file lives inside the cluster, so it's not available to the CircleCI environment to upload it. This is just a scripting hurdle.

I'm going to park this here for now and may explore running these tests in another mode.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant