What's wrong with tarantool tests?

Diff-based approach (pro et contra)

• contra: write easy, maintainance is hard
• contra: explicit test oracle, test is useless without verified .result file
• contra: fragile because depend on formatting
• contra: one cannot create a test when tested functionality is not ready, because output is a mandatory part of a test
• contra: during review you should inspect both artifacts: source code of a test and it's output. It's double efforts for reviewers.
• pro: you can easily reproduce a bug with copy-paste test source code in interactive mode

See also: MySQL, MariaDB, PostgreSQL, PostgreSQL

test-run.py and test harness limitations

• Can't use remote instances
• No support of additional test statuses like XFail, XPass. Instead we comment blocked test out and uncomment it after a fix.
• Low code reuse among different tests. Only basic and often used operations are in a test_run.lua and it's number is small.
• Pure tests isolation, test should be hermetic. Many tests doesn't restore environment state at the end of testing and new test starts in a "dirty" environment.
• Pure parametrization of tests. One cannot easily rerun async replication tests for synchronous replication.
• test-run.py lack features existed in a popular testing framework for years: just take a look to a pytest for example.
• Pure testcase management: cannot to run single testcase in a file
• One cannot use common fixtures in a number of tests, instead he should implement new functionality in a test-run.py