disabled capturing of stderr to allow us to see cause of failure in functional tests

[wxtim]

Closes #71

Currently when functional test fails (often when it runs cylc validate) one has locate the pytest running that code and add breakpoints, even when the test failure relates to problems which are small or tangential to the feature in question.

By adding -s to the pytest option any failures will show the stdout from the commands run.

To test, try doing anything nasty (delete the runtime section/add an illegal config item/remove icp/change the name of a jinja2 template variable) to the flow.cylc in any of the functional test workflows and running the tests.

see also Pytest docs: How to caputure stdout/err

[MetRonnie]

This works ok, but another solution is to use check=True like so

--- a/tests/functional/test_pre_configure.py
+++ b/tests/functional/test_pre_configure.py
@@ -91,9 +91,7 @@ def test_validate(tmp_path, srcdir, envvars, args):
     script = ['cylc', 'validate', str(srcdir)]
     if args:
         script = script + args
-    assert (
-        run(script, env=envvars)
-    ).returncode == 0
+    run(script, env=envvars, check=True)
 

With this, pytest captures and prints the stderr after the traceback, instead of it showing up amongst the list of tests and their statuses.

If check is true, and the process exits with a non-zero exit code, a CalledProcessError exception will be raised. Attributes of that exception hold the arguments, the exit code, and stdout and stderr if they were captured.

https://docs.python.org/3.7/library/subprocess.html#subprocess.run

[wxtim]

This works ok, but another solution is to use check=True like so

--- a/tests/functional/test_pre_configure.py
+++ b/tests/functional/test_pre_configure.py
@@ -91,9 +91,7 @@ def test_validate(tmp_path, srcdir, envvars, args):
     script = ['cylc', 'validate', str(srcdir)]
     if args:
         script = script + args
-    assert (
-        run(script, env=envvars)
-    ).returncode == 0
+    run(script, env=envvars, check=True)
 

With this, pytest captures and prints the stderr after the traceback, instead of it showing up amongst the list of tests and their statuses.

If check is true, and the process exits with a non-zero exit code, a CalledProcessError exception will be raised. Attributes of that exception hold the arguments, the exit code, and stdout and stderr if they were captured.

https://docs.python.org/3.7/library/subprocess.html#subprocess.run

Preferable, but a much bigger job: Can I suggest this, for now, but with a suggestion that we introduce your logic later.

[kinow]

Created follow-up issue: #78 Thanks @wxtim