(Parallel) testing of multi-role repositories

[tumbl3w33d]

I'm quite new to molecule yet and spent too much time and some heavy shell-fu lifting to accomplish what I would have expected to be a common use case, so I'd like to present you with what I've done and get some feedback from experienced molecule users, so I might be able to simplify my approach.

I saw that you call this multi-role (or sometimes mono) repository: A repository that simply consists of role folders.

I use such as git submodule to share roles across all my ansible repositories. So, I started to add molecule tests and struggled with the discovery of my tests and their scenarios. Some blog posts pointed me to use of shell-fu to iterate through folders and execute molecule in each of them.
Since I started using the molecule-hetznercloud driver right away (because I use hcloud anyway) I quickly ran into the issue that tests were slow when executed sequentially, so I used some bash voodoo to execute everything in parallel. It works like a charm, but now we're back to my initial question: Am I doing it wrong?

I will show you my bash script that I use for local test development and in Jenkins CI:

#!/bin/bash -e

scenario="--all"
base_config="-c ../molecule/base_config_local.yml"

while getopts "c:l:s:r:p" opt; do
    case $opt in
        c)
            echo "Overriding molecule base config path with $OPTARG"
            base_config="-c $OPTARG"
            ;;
        r)
            echo "Run role $OPTARG"
            role="$OPTARG"
            ;;
        s)
            echo "Run scenario $OPTARG"
            scenario="$OPTARG"
            ;;
        p)
            echo "Will execute scenarios in parallel"
            parallel="--parallel"
            ;;
        l)
            echo "Logs will be collected in $OPTARG"
            log_dir="$OPTARG"
            [ -d $log_dir ] || mkdir $log_dir
            ;;
        \?)
            echo "Invalid option: -$OPTARG" >&2
            exit 1
            ;;
    esac
done

if [[ "$scenario" != "--all" ]] && [[ ! -z "$parallel" ]]; then
    echo "Your arguments ask to run a single scenario in parallel, but that makes no sense. Aborting…"
    exit 42
fi

if [[ "$scenario" != "--all" ]] && [[ -z "$role" ]]; then
    echo "Your arguments ask to run a certain scenario, but you did not specify the role. Aborting…"
    exit 43
fi


mcmd="molecule $base_config test $parallel"

if [[ ! -z "$role" ]]; then
    pushd $role/ > /dev/null
    if [[ "$scenario" != "--all" ]]; then
        echo "[$role] Testing scenario $scenario"
        $mcmd --scenario-name $scenario
    else
        role_scenarios=($(ls -d */*/molecule.yml | cut -d/ -f2))
        for ((i = 0; i < ${#role_scenarios[@]}; i++)); do
            role_scenario=${role_scenarios[$i]}
            echo "[$role] Testing scenario $role_scenario"
            if [[ ! -z $parallel ]]; then
                ANSIBLE_FORCE_COLOR=0 PY_COLORS=0 $mcmd --scenario-name $role_scenario > ${log_dir}molecule_${role}_$role_scenario.log 2>&1 &
                scenario_pids[$i]=$! # store bg job id
            else
                $mcmd --scenario-name $role_scenario | tee ${log_dir}molecule_${role}_$role_scenario.log 2>&1
            fi
        done
        # wait for bg jobs
        for ((j = 0; j < ${#scenario_pids[@]}; j++)); do
            pid=${scenario_pids[$j]}
            wait $pid || (echo "[$role] Scenario ${role_scenarios[$j]} failed" && exit 44)
        done
        echo "[$role] Finished waiting for scenario test jobs"
    fi
    popd > /dev/null
else
    echo "Searching for all roles with tests"
    for role_with_tests in $(ls -d */molecule | cut -d/ -f1); do
        echo "Testing role $role_with_tests"
        pushd $role_with_tests/ > /dev/null
        role_scenarios=($(ls -d */*/molecule.yml | cut -d/ -f2))
        for ((i = 0; i < ${#role_scenarios[@]}; i++)); do
            role_scenario=${role_scenarios[$i]}
            echo "[$role_with_tests] Testing scenario $role_scenario"

            if [[ ! -z $parallel ]]; then
                ANSIBLE_FORCE_COLOR=0 PY_COLORS=0 $mcmd --scenario-name $role_scenario > ${log_dir}molecule_${role_with_tests}_$role_scenario.log 2>&1 &
                scenario_pids[$i]=$! # store bg job id
            else
                $mcmd --scenario-name $role_scenario | tee ${log_dir}molecule_${role_with_tests}_$role_scenario.log 2>&1
            fi
        done
        # wait for bg jobs
        for ((j = 0; j < ${#scenario_pids[@]}; j++)); do
            pid=${scenario_pids[$j]}
            wait $pid || (echo "[$role_with_tests] Scenario ${role_scenarios[$j]} failed" && exit 45)
        done
        echo "[$role_with_tests] Finished waiting for scenario test jobs"
        unset scenario_pids
        unset role_scenario
        popd > /dev/null
    done
fi

For those who cannot read bash fluently, a little summary:

• if you pass no parameters, this simply iterates through the role folder in project root
  • if your role contains a molecule folder it's eligible for having its tests executed
  • then the role scenarios are iterated and molecule test is executed for each of them
  • this can also be done in parallel (-p) which uses bash's background job magic to fire multiple molecule processes (per scenario) and waits for the test completion
  • in case of parallel execution, logs are aggregated in files (output folder can be specified). I do this for Jenkins CI execution where I just archive these execution logs. The CI log just says which scenario failed and I can check the logs that are attached to the CI job run
• you can also specify a role name (-r) to just run tests for this one role
• you can further limit the scenario in the role (-s)
• you can override the base config (-c). I need to do that for CI, because the ansible ssh control path quickly gets too long (unix socket names are limited to 108 chars).

I'd be happy to simplify this script if you molecule gurus know some cool flags that make life easier. If it turns out this is what you have to do to achieve efficient testing of multi-role repos, then I'll just leave this script here. Maybe it saves some poor soul some hours. :)

[matthughes]

Just want to say thanks for writing this. Crazy that this isn't baked in.

[greg-hellings]

You might want to check out https://github.com/ansible-community/tox-ansible - a plugin for Tox which allows you to not only run tests in parallel and auto-discover them, but also list out the auto-discovered scenarios, run them with multiple Ansible and Python versions, and several other nice-to-haves (also finds and runs ansible-test content in addition to molecule content).

Basically, molecule doesn't natively support parallel execution because there are better tools out there that already handle that (like tox) and anyone using Molecule for CI structures should devolve that portion into the CI fan out/fan in pipeline.

[tumbl3w33d]

Thank you for the hint. I guess I should have invested more time on figuring out about tox. I guess I was put off by the thought that I need to learn a second tool just to get my ansible tests going, so I took the shortcut.

You are right, I added some more parallelization in my Jenkinsfile, because not only parallelism within a role was necessary, but I wanted to execute all roles' tests at once.