Discussion: how to validate regressions and add CI tests?

[gongxun0928]

I’ve been running build/run-tests.sh on the main branch and encountering several test failures. It’s currently hard to tell whether these are new regressions or if the tests/scripts themselves have become stale or outdated.

I’d like to open a discussion around two key questions:

• Should we refresh or fix the existing test script and test cases?
  • If some failures are due to environmental changes, deprecated assumptions, or drift from the current codebase, we should probably update or remove those tests first. Otherwise, it’s difficult to distinguish real breakages from false positives caused by outdated tests.
• Should we introduce a minimal, reliable CI test suite?
  • Right now, there doesn’t seem to be a stable set of tests that run consistently in CI. Would it make sense to define a core subset—e.g., successful build + a few critical unit/integration tests—to catch breaking changes early without significantly slowing down CI?

[lzjqsdd]

Regarding the first issue:

We currently rely on an internal environment for regression testing, which is not integrated with CI checks. The main reason is that debugging the GitHub Actions CI environment for tests like FUSE has proven somewhat challenging. The workflow is as follows:

We primarily use the entry scripts available at https://github.com/CurvineIO/curvine/tree/main/scripts. The regression testing is deployed on an independent test machine, where Curvine is run in standalone mode for validation. Within the scripts directory, the build-server.py file can start the regression service. This service can be triggered periodically via other task scheduling systems. Additionally, build-server.py is used to query regression test results at http://xxxx:5002/result, which includes UT (Unit Tests), cov-test (Coverage Tests), ltp-test (LTP Tests), fuse-test (FUSE Tests), and more.

Startup Instructions:

python build-server.py --project-path ~/codespace/curvine-dev --results-dir ~/curvine-dailytest/result

Triggering Tests:

curl -X POST http://localhost:5002/dailytest  # Start all tests
curl -X POST http://localhost:5002/fuse/run  # Start FUSE test individually

It’s important to note that this regression test framework relies on logic such as updating Curvine code, which needs to be customized independently based on the specific environment.

Regarding the second issue:

This is an excellent suggestion. A minimal CI test suite would enable CI checks to quickly verify whether new code breaks existing functionalities.

By the way, there are currently some known test failures in the project’s codebase. Contributions to help improve this are highly welcome!