Integration tests with multiple configuration files

[wvengen]

An attempt.

• Initial CI configuration
• Test http auth #24
• Implement -c argument for scrapyd-k8s
• Check CI configuration functions
• Check TEST_ environment variables are passed through in CI config (do they need export?)
• Find a solution for the final kill %1 not working the very last time when running scrapyd-k8s directly (both docker and k8s; one thing to try would be to store the pid in a variable right after running it in the background, and killing the pid directly; or use another signal; may also be related to the note on termination in the next point) --> just needed to wait and ignore exit code
• Find a solution for in-cluster scrapyd-k8s not coming up again soon enough in 2nd integration test (a likely cause is that scaling down scrapyd-k8s is taking a lot of time, perhaps we need to handle signals better so that it terminates earlier)
• Make sure tests pass
• Document how integration tests function, with config file adaptation, and how to run these tests easily there is no doc about this, fix later

[wvengen]

@vlerkin this is along the lines of what I thought. It keeps a better separation between the environment setup, and the actual test. And exactly the same test can be used for docker/k8s/in-cluster-k8s.

There are some small but important issues to resolve. Are you able to look into this? I think it needs a bit of Linux knowledge, let me know if you need pointers (or ask around).

[wvengen]

Ah, the k8s failure is actually very very simple: the service never comes up, because the health check is behind the auth. Bummer this issue is still open ... jpvanhal/flask-basicauth#11