At the moment, when Maat launches LS, it only checks the exit code:
|
def _ls_has_errors(ls: StepReport) -> bool: |
|
# Check for total errors count greater than 0. |
|
match = re.search(r"total: (\d+) errors", ls.log_str, re.M) |
|
return match and int(match.group(1)) > 0 |
This way, we only care about the process success, not the actual test. We can't verify if LS performed any actions or simply exited with status 0. We also don't know when the experiment was skipped, especially when filtering in #124.
Beyond checking the exit code, we can e.g. assert some logs in the output.
At the moment, when Maat launches LS, it only checks the exit code:
maat/src/maat/report/analysis.py
Lines 345 to 348 in 69d8acf
This way, we only care about the process success, not the actual test. We can't verify if LS performed any actions or simply exited with status 0. We also don't know when the experiment was skipped, especially when filtering in #124.
Beyond checking the exit code, we can e.g. assert some logs in the output.