Run staging CI in integration branch

[msheiny]

Feature request

Description

In a recent hack-a-thon an issue came up where CircleCI was not properly building on forks of users that already had existing CircleCI accounts. In particular, was not importing all of our environment variables and instead using the fork author's circleCI environment. See comments in #2515.

That PR also ^^^ added an additional x2 builders for a total of x4 builders per run - which I see as problematic because of the increased complexity of the CI job and potential for annoying to debug issues to get CI to successfully pass (which has happened in the past a ton).

I foresee these two issues combining to cause significant more issues for new contributors and at hackathons (where we'll have mutliple folks trying to contribute at the same time rapidly).

I'd like to see the CI workflow refactored as follows:

• PR is run against run against application-tests (if we want to wait to refactor that to containers, thats fine with me .. the current travis job is clunky). If passes, maintainers will merge that into an integration branch.
• CI will run regularly (every hour?) against integration with the full suite of tests and if failures are detected we notify ALL chat channels that CI needs to get fixed. Otherwise, if everything passes, CI will automatically merge integration --> develop.

There are definitely cons here -- but ultimately very few contributors are touching the ansible logic and yet we are spinning up a lot of flaky environments that have potential to fall over and cause grief for every single PR. This will require the team to be more aware of the pipeline and respond when CI starts failing in develop.

However as pros I see:

• ultimately it will make things easier and faster for outside contributors to jump in. Testing staging is still hard to do locally even with the vagrant workflow especially when you are only touching application code logic.
• Once we start playing with more cron like logic we can also do regular deploys to a regular staging environment on physical hardware (with real grsec tests back in)
• We can also add in the upgrade testing story at this stage see Test upgrade path in CI #1689

The big blocker for this effort that I see is that the app-tests really need to get containerized properly first. @tmc brought in PR (#2523) which aims to address this. If that goes in, I'm not sure it'll answer all pre-requisites but its certainly a great start.

User Stories

As a contributor, I want CI to be very stable, run quickly, and get potential accepted code merged efficiently by the SecureDrop maintainers.

As a maintainer, I want the CI pipeline to run in discrete chunks that is easier to respond to when integration issues arise. Having one CI job to maintain versus one for each PR to analyze.

[ghost]

IMHO there is no immediate need for this (fine) adjustment. The problem happened when a contributor tried to improve the CI. They are bound to do mistakes that use more resources than needed. And if they can't (because they have to first wait for their change to be merged in the integration branch). it may discourage them from contributing. Another concern is that adding an integration stage adds complexity to the contribution process (not much but still).

That being said, I think it would make sense to implement what is proposed when and if some malicious person sends a stream of pull requests designed to waste resources.

I don't feel strongly about this but I tend to favor not doing anything ;-)

My 2cts.

[redshiftzero]

A closely related idea that would be better is running staging tests on demand (e.g. triggered by a comment in a PR) as it:

1. Reduces CI consumption
2. Ensures we do a scan of the diff first prior to spinning up staging boxes
3. Issues are caught pre-merge into develop branch (if we don't do this staging testing pre-merge, PRs will get merged that cause breakage and maintainers will be responsible for picking up the pieces)

[redshiftzero]

@msheiny just pointed me to https://circleci.com/docs/2.0/workflows/#holding-a-workflow-for-a-manual-approval which would address the concern here: we can prevent the staging job from running on forks and just hold the staging CI job until we've looked at the diff, think it's ready for a staging test run. The workflow is that we would need to go in Circle CI and approve the job run. This way we can sidestep having to workaround the CI restrictions here, e.g. needing to separately trigger release candidate builds in another repo (freedomofpress/securedrop-builder#63).

[eloquence]

Closing; we're likely to move some jobs to nightly (e.g., #5336), but are unlikely to implement the strategy described here.