mypy exited with status 1 - no error messages shown

[LourensVeen]

I have some type errors in my code, and would like to find them by running mypy via pytest-mypy. When I run pytest, I get:

=========================================================================== test session starts ============================================================================
platform linux -- Python 3.6.9, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- <venv>
cachedir: .pytest_cache
rootdir: <project-root>, configfile: setup.cfg, testpaths: proof_of_concept, tests
plugins: cov-2.11.1, pycodestyle-2.2.0, pydocstyle-2.2.0, mypy-0.8.1
collected 154 items                                                                                                                                                        

proof_of_concept/__init__.py::mypy PASSED
proof_of_concept/__init__.py::mypy-status FAILED

================================================================================= FAILURES =================================================================================
_______________________________________________________________________________ test session _______________________________________________________________________________
mypy exited with status 1.

This suggests that there were some errors, but it doesn't show what they were, so I cannot fix them. Running mypy separately reveals a bunch of issues in proof_of_concept/rest/registry_client.py.

Looking at the pytest-mypy source, it seems that this can happen if mypy checks a file that has not been collected. I'm importing the problem file from the file under test, and mypy gives a bunch of errors about this imported file. I guess the ::mypy test passes anyway, because the errors are not in the file under test. However, pytest-mypy then generates an error using this second mypy-status result, because mypy returned a non-zero exit code.

Ignoring the errors makes sense I think, they should show up later when the problem file itself is tested. Unfortunately, we never get to that point, because I'm running with -x, which stops the tests as soon as the first error is found, and no error messages are shown for the mypy-status failure, so I cannot fix them.

I'm a bit confused about the current intent of the pytest-mypy code. There's a comment suggesting that the ::mypy test is intended to fail even if mypy only shows errors in files that weren't collected, and that that is why there's the ::mypy-status result. On the other hand, there used to be a test that checked that no MypyStatusItem was created unless there was at least one MypyFileItem, which I think means the opposite. That was removed in cd459a4 however, which also introduced this -status result.

In my opinion, a mypy failure only because of errors in imported files should either be considered an error in the file under test, in which case the error should be printed, or it should not be considered an error in the file under test, in which case all tests related to that file should pass and the non-zero exit code ignored. The latter makes more sense to me.

[dmtucker]

Hey there! Thanks for the report 👍

At a glance, the MypyStatusCheck appears to be doing its job, but we're hiding the output it's failing on because -x prevents the MypyFileItem that would report it from running. I have a test that reproduces the issue and will try to take a closer look at a fix this week.

[LourensVeen]

Sorry, I misread, please disregard the below. If I run without -x, the output is printed, but that's because the mypy run for the offending file is run, fails, and produces the error. I'm still getting only mypy exited with status 1. for the initial mypy-status failure even without -x.

Oh! I hadn't noticed that the error was indeed printed without -x, but I've checked and I can confirm that that's the case for me as well. That's strange though. I would expect pytest to handle the first error in the same way it would do without -x, and then quit. Instead it apparently processes the first error differently.

I don't know enough about how this works to say if that makes sense, but if you think it's a pytest issue rather than a pytest-mypy issue then please let me know, I'll go make an issue on the pytest side then. (I just did a quick search there, but didn't find anything related.)

[scriptator]

I needed to debug that issue which occurred only on my CI server, so I added more detailed output to the MypyStatusError. That allowed me to find out which files were picked up by mypy that weren't collected by pytest. See my PR #134

[dmtucker]

A duplicate of this issue was reported (#196), but I think that one is a little clearer at this point.
So I'm going to close this one and continue over there...