CI tests for generator component are missing

[iakov]

Missing test for generator, but definitely some regression tests are required. I suggest implementing the following simple steps.

1. Add ASAN_OPTIONS to generation step, at least ASAN_OPTIONS="detect_stack_use_after_return=1:fast_unwind_on_malloc=0"
2. Change each CI job to have an additional step to build with freshly generated sources.
3. (Optional, arguable) add step to diff the generated_cpp sources with the currently used generated_cpp_XXX to enforce the regular update.

The last one should be done after the fixes in generator for the known non-determinism.

[mrbean-bremen]

@iakov - are you planning to make a PR for this, or should someone else look at it?

[iakov]

@YuriUfimtsev is "the chosen one". This is the issue for him, he knows, we've discussed this in person.

[iakov]

Partially done and merged: 6d90f6c
Next step: add LSan suppressions via LSAN_OPTIONS=suppression=lsan.supp with corresponding lsan.supp if it is too time-consuming to fix current leaks. Useful small step for regression tests.
@YuriUfimtsev

[iakov]

Probably, detect_leaks=1 can be turned on (with tiny suppressions for globals) after #148

[mrbean-bremen]

Probably, detect_leaks=1 can be turned on (with tiny suppressions for globals) after #148

Yes, this is the hope... let's wait and see, hopefully that branch will be merged soon.

[mrbean-bremen]

I had switched on detect_leaks=1 in a branch and noticed that there is still a lot of leaks in the generator. Personally, I'm not too worried about this, as the generator is only run on demand every once in a while, and it doesn't bother me if that takes a few MB more of RAM. I don't think it is worth the possibly large effort of fixing this, but if anyone is willing, I won't object 😀

As for the other other goals: there is currently the "Check generated_cpp" build job that runs for Qt 5.12, 5.15 and 6.5, for ubuntu and Windows (MSVC). We also may add Macos builds here, and if needed more versions. We could also change the existing jobs to compile the generated sources, and consolidated the existing tests. Here is a list of the versions currently tested in CI for reference:

Build

ubuntu 20.04  Qt 5.12.8    Python 3.8.10
ubuntu 22.04  Qt 5.15.3    Python 3.10.12
centos 7      Qt 5.9.7     Python 2.7.5
rockylinux 9  Qt 5.15.3    Python 3.9.18
MacOS 11      Qt 5.9.9     Python 3.6.15
MacOS 12      Qt 5.12.12   Python 3.11.6
MinGW 5.3     Qt 5.11.3    Python 3.6.8
MinGW 7.3     Qt 5.12.12   Python 3.10.11
MSVC 2017     Qt 5.11.3    Python 3.6.8

Check generated

MSVC 2022     Qt 5.12.12   Python 3.12.1
MSVC 2022     Qt 5.15.2    Python 3.12.1
MSVC 2022     Qt 6.5.3     Python 3.12.1
ubuntu 22.04  Qt 5.12.12   Python 3.12.1
ubuntu 22.04  Qt 5.15.2    Python 3.12.1
ubuntu 22.04  Qt 6.5.3     Python 3.12.1

As for the last point (compare generated files with checked in generated) - I'm not sure how helpful that would be, given that the checked in generated files are a snapshot of a specific Qt version, and APIs tend to change in Qt also in different patch releases. As I wrote elsewhere, we are now generating the sources ourselves and don't rely on the checked-in code, but I see that it can be helpful to have the sources already checked in.

I'm trying to wrap up a bit before a Qt 5.15 / 6.5 release soon to come, thus my ramblings...

[YuriUfimtsev]

I had switched on detect_leaks=1 in a branch and noticed that there is still a lot of leaks in the generator. Personally, I'm not too worried about this, as the generator is only run on demand every once in a while, and it doesn't bother me if that takes a few MB more of RAM. I don't think it is worth the possibly large effort of fixing this, but if anyone is willing, I won't object 😀

Yes, a bit more refactoring work and I will open PR that decreases almost all memory leaks in the generator. I will do it today or tomorrow :)

[he-hesce]

It seems Rocky Linux 9.3 has Python 3.11 as optional install. Would be cool to have an alternative rockylinyx 9 with Python 3.11 in matrix. We plan to offer support for both Python 3.9 and 3.11 on Rocky Linux 9 in our product.

[mrbean-bremen]

@YuriUfimtsev - what's the state here? Is there something left to do?
@he-hesce - is that still relevant? If yes, you can always make a PR to adapt the build matrix.

[he-hesce]

@mrbean-bremen : Yes, we are still using/supporting both Python 3.9 and Python 3.11 on Rocky Linux 9 as both Python versions are supported with 3.9 being default. Is it just to update the build matrix? Is there an image/container available with Rock9+3.11?

[mrbean-bremen]

Is there an image/container available with Rock9+3.11?

I actually have no idea, but the build should be probably similar to the current CentOS build. I have no experience there, so I thought it would be better done by somebody actually knowing the stuff :)