Update Clang toolchain from 18.0.0 to 18.1.8 #12365

catenacyber · 2024-08-16T09:48:20Z

Follow-up on #12077 by @alexcrichton cc @maflcko

Main difference is to update infra/base-images/base-runner/profraw_update.py so that oss-fuzz converts profraw version 8 to 9 (and llvm-cov seems more tolerant in older version reading cf llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp

This way, it should be more transparent for projects, that can be updated individually or not

catenacyber · 2024-08-16T09:49:31Z

This is rather a draft for discussion and testing : what is the good way to test that ?

maflcko · 2024-08-16T09:54:21Z

Not sure about authorship, but personally, I think it would be easier to follow the history, if you just took the commit 6defef2 from the other pull request. You can then rebase it on master, or just create a merge commit with master (my preference, because it leaves the commit ID exactly intact).

(Edit: Or if you want to keep the pins for now, it'd probably be commit ID 8b26362)

This will also include other useful changes, like unpinning the projects that no longer need the pin.

For testing, you can test locally (probably only works for a few spot tests), or wait for an admin to create the global infra build.

catenacyber · 2024-08-16T10:08:23Z

Not sure about authorship, but personally, I think it would be easier to follow the history, if you just took the commit 6defef2 from the other pull request.

Yes, cherry-picking the commits from Alex :-)

This will also include other useful changes, like unpinning the projects that no longer need the pin.

I am not sure I want to do that now : I want the least changes possible, only ones in infra, not ones in individual projects :

This way, if this patch is good, there should be no changes in the builds status...
Except new warnings introduced in clang 18.1 that can make some compilations fail. In this case, maybe the projects maintainers want to fix these warnings instead of ignoring them (I do as a project maintainer)
And except other surprises

maflcko · 2024-08-22T12:12:14Z

Would be nice to get an infra build here, to see if there are any build issues or other issues

catenacyber · 2024-08-25T20:09:07Z

Would be nice to get an infra build here, to see if there are any build issues or other issues

Should I do something ?

maflcko · 2024-08-26T06:29:16Z

I think only repo admins can trigger the infra trial-build

catenacyber · 2024-08-26T15:36:57Z

cc @jonathanmetzman

DonggeLiu · 2024-08-27T02:09:39Z

/gcbrun trial_build.py all

catenacyber · 2024-08-27T14:42:47Z

@DonggeLiu what is the outcome of the trial build ?

maflcko · 2024-08-27T14:58:55Z

For some reason the logs are truncated now. (Previously they weren't and it was possible to see the tail and the result)

The ones that I see are all failures. (I checked a few URLs)

logs

catenacyber · 2024-08-27T15:02:44Z

Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/gsutil
Step #1: CommandException: No URLs matched: gs://oss-fuzz-coverage/tinygltf/textcov_reports/20240827/*
Step #1: CommandException: 1 file/object could not be transferred.
Step #6: 334586aa2186: Pull complete
Step #6: 174a75ccacd5: Pull complete
Step #6: Digest: sha256:c7405fed95553b9fb9be1723f233b2c1ec42273135169d98a9c1a4da4344fafa
Step #6: Status: Downloaded newer image for gcr.io/oss-fuzz-base/base-builder-swift-testing-profraw9-update:latest
Step #6: gcr.io/oss-fuzz-base/base-builder-swift-testing-profraw9-update:latest
Finished Step #6
Finished Step #1
ERROR
ERROR: build step 1 "gcr.io/cloud-builders/gsutil" failed: step exited with non-zero status: 1

What is going on here ?

DonggeLiu · 2024-08-28T00:55:14Z

For some reason the logs are truncated now. (Previously they weren't and it was possible to see the tail and the result)
The ones that I see are all failures. (I checked a few URLs)

Thanks @maflcko. This is the full log.

I also noticed many failures.

DonggeLiu · 2024-08-28T01:09:32Z

Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/gsutil
Step #1: CommandException: No URLs matched: gs://oss-fuzz-coverage/tinygltf/textcov_reports/20240827/*
Step #1: CommandException: 1 file/object could not be transferred.
Step #6: 334586aa2186: Pull complete
Step #6: 174a75ccacd5: Pull complete
Step #6: Digest: sha256:c7405fed95553b9fb9be1723f233b2c1ec42273135169d98a9c1a4da4344fafa
Step #6: Status: Downloaded newer image for gcr.io/oss-fuzz-base/base-builder-swift-testing-profraw9-update:latest
Step #6: gcr.io/oss-fuzz-base/base-builder-swift-testing-profraw9-update:latest
Finished Step #6
Finished Step #1
ERROR
ERROR: build step 1 "gcr.io/cloud-builders/gsutil" failed: step exited with non-zero status: 1

What is going on here ?

@oliverchang, not sure if this is a known issue:
The missing report error implies that trail_build should have fetched yesterday's report, not today's.
I triggered the experiment on the morning of August 27 (Sydney), and it searches for a report in the directory 0827/. However, the report has not been generated yet.
I am pretty sure the coverage report exists now at:
https://storage.googleapis.com/oss-fuzz-coverage/tinygltf/textcov_reports/20240827/fuzz_gltf.covreport

Before we fix the bug, we should run the command after 6 PM (Syndey), given that the report was generated after 5 PM:

catenacyber · 2024-09-01T19:32:35Z

@DonggeLiu will you run the command again with the good timing ?

DonggeLiu · 2024-09-01T23:45:01Z

@DonggeLiu will you run the command again with the good timing ?

Sure, I will do it in the afternoon (hopefully I can still remember this).

DonggeLiu · 2024-09-03T12:25:14Z

/gcbrun trial_build.py all

maflcko · 2024-09-04T06:39:45Z

For reference, the output is:

trial-build

Remaining builds: 20/1007, defaultdict(<class 'list'>, {'apache-poi': ['8111fa89-ffae-406f-a0a3-9e030531718c'], 'aspell': ['2212a2a7-e36e-4d0e-bd43-8a431d6e5458'], 'binutils': ['33a8c9f8-a2c5-4c7c-b56b-4b6b7cacd2f7'], 'cloud-custodian': ['6ee0ab2c-c6da-4147-b0f0-893aba501f45'], 'cmake': ['bcbab0f6-4b6a-41a0-9595-097b5ea9b643'], 'duckdb': ['3d2f04d6-d453-4815-af78-2078c51d1e5d'], 'gitpython': ['ca5d9ed0-2494-44df-8985-26c774af3203'], 'ipaddress': ['fec49ceb-49da-4e62-a736-05c6803e7dfb'], 'jansi': ['fafb268c-e9d2-468f-86f2-2a4dd7112580'], 'kubeedge': ['878a7656-4f2f-43d5-93da-7ec019bb58d8'], 'onednn': ['5c18d180-53f2-4a89-b1e5-ce220ef40404'], 'opencsv': ['f5ebe13a-5f5a-4911-8a5d-cd9ac0e53a04'], 'openweave': ['9589d151-fac6-415e-b56f-d10534d276d3'], 'pasta': ['2d33ffc1-f266-4821-9724-fe5ceb39913a'], 'simd': ['a4389c9d-ae8a-4771-a089-d81e308b78e0', '2e0e616b-049b-41f3-9609-08f2abfe3680'], 'sqlalchemy_jsonfield': ['29357d3d-f01f-4742-ae70-6cabf3ae1704'], 'utf8proc': ['9620eec1-fbd4-41ed-870b-7e348ec8b36b'], 'xen': ['efe68fc4-1aad-497b-9980-8b5ca2bfbfb8'], 'zlib-ng': ['82469efd-1f99-4bb7-a73b-bccdfc1102b1'], 'llvm': ['e2cb5d7e-8647-482c-9936-82e859b00004']}).

maflcko · 2024-09-04T06:41:13Z

It looks like most of the failures are due to the branch being outdated.

Before a trial-build, my recommendation would be to merge the current master branch into the pull request. Otherwise, the result may be inaccurate and outdated.

DonggeLiu · 2024-09-04T12:14:06Z

/gcbrun trial_build.py all

catenacyber · 2024-09-04T12:31:41Z

Failed builds: 38/1007

Looked already pretty good, did it not ?

DonggeLiu · 2024-09-04T23:52:10Z

Here is the trial build log log.

I (or someone else) will take a look into them

maflcko · 2024-09-05T04:53:29Z

I looked at the result and I think some projects break due to the compiler bump. It would be good to pin them, or otherwise fix them.

https://github.com/google/oss-fuzz/pull/12365/checks?check_run_id=29667299742

trial-build

Remaining builds: 31/1011, defaultdict(<class 'list'>, {'bitcoin-core': ['e97dac9b-220c-45ee-8343-84e9eec6bfe1'], 'bluez': ['b1499b2f-148a-4a43-af5e-084863c2c8a9'], 'checkstyle': ['8d87523f-7b24-473e-af3a-3bdcd241c6ee'], 'cloud-custodian': ['e38a48e3-419c-4e2b-9dd7-f94f18419a20'], 'cpuinfo': ['54fa5e3d-885d-4a6e-8a77-32fd3b80213b'], 'duckdb': ['b0426bc7-3e54-403b-a847-56ee0cb9a412'], 'ffmpeg': ['ac82d2f0-d7ca-4e70-b08b-b660c9dc2a04'], 'genshi': ['d2f6169b-2ec1-47df-8557-505f95afaa7b'], 'gprof2dot': ['33736e00-d147-4d0d-b2d8-7b9f7b78e833'], 'hikaricp': ['0958c36c-12ab-4ecb-9816-19b5e45cfd10'], 'htslib': ['61e7a9d0-2b54-4d1e-a96e-e072703a63f8'], 'http-parser': ['7dc60deb-9c57-4d76-825d-5ed27c6e6d74'], 'imagemagick': ['4075bbc0-1eb4-443a-b141-828a5cea34fe'], 'jq': ['f997d55a-3b45-46ab-b6aa-7f1b8a0888ab'], 'libreoffice': ['021c4098-7db3-49a5-ade8-535674b6fa3d'], 'libxslt': ['2f7b1a83-51da-4096-a39a-5216ee44b14e'], 'libyal': ['c4323b37-191e-4289-bc75-3b3c41a7ec79'], 'muparser': ['7fd96760-4d21-44a5-ae01-cf24e3af2a60'], 'neomutt': ['ab316d4f-3f3b-44cf-a37e-2c7ce2e0a3fb'], 'onednn': ['e6448e28-040b-445b-a7aa-9e8fdb06e382'], 'openweave': ['72af842b-1ce3-45f8-b8d5-3b1202183974'], 'python-phonenumbers': ['229da2ce-22f8-4c23-a3f3-e5fda4a17b1e'], 'simd': ['0d3301bd-6bc7-461c-8722-9fa3ae920b8a', '50e5284d-c067-4800-b6fc-a66b0641243a'], 'spring-cloud-config': ['d6f33c33-d6dd-4e0c-8011-1047e1f9be71'], 'sqlalchemy_jsonfield': ['c2b8ae95-517c-4152-ae49-9bae934a762e'], 'toolbelt': ['f71f41d6-59ec-4744-9dc7-e4656c7819ef'], 'vorbis': ['7a26d044-8d1d-4b68-9327-edb71f387ef3'], 'xmlpull': ['c72c9058-a0f0-408f-98cc-5a68d2b5150d'], 'yoga': ['2ef90f49-a3f6-4036-b0d5-6f00e465cfa3'], 'zopfli': ['10bee761-eaa3-4727-974b-9315b59786f0'], 'llvm': ['5346b641-4627-48a2-aa21-cdf3b33cca3b']}).

catenacyber · 2024-09-05T14:32:54Z

First analysis :

digest : network error
envoy: new warning Wthread-safety-reference-return
freeimage: network error
icu: centipede build failure, is it related ?
librawspeed: new warning Wnan-infinity-disabled
libucl: strange, maybe due to timeouts and stack overflows in corpus for coverage
libxml2: looks like incompatibility between trial build and aarch64

I looked at the result and I think some projects break due to the compiler bump. It would be good to pin them, or otherwise fix them.

I guess the new warnings need to be fixed by each project

Only 2 out of the 7 first failures (icu and libucl) I do not understand and they may be due to this PR.
I think this PR will fix more than it breaks, would it not ?

maflcko · 2024-09-05T15:52:09Z

Only 2 out of the 7 first failures (icu and libucl) I do not understand and they may be due to this PR.

Sometimes the project starts to fail to build at the same time the infra run is scheduled. So it may or may not be related to the PR and another build has to be done in those cases (either locally, or in another infra run).

If the project is failing, I think maintainers prefer it to be pinned or fixed. Pinning can be achieved similar to commit 767c7d3. (You'll have to adjust the sha256 to be a recent one).

catenacyber · 2024-09-05T19:09:36Z

mpv : centipede again...
poppler: unrelated Could NOT find Boost
rust-lexical: build failure to investigate
samba: Wformat-truncation intended by some test
simdutf: aarch64
suricata: I guess I need to update it for rust and C compiler to match
tinyusb: aarch64

catenacyber · 2024-09-05T19:19:09Z

tomlkit: network error ?
wasmer: new rust warnings
zeek: looks like I need to fix coverage for targets generating multiple profraw in one run ?..
Hadoop: looks unrelated
lua: aarch64
tarantool: aarch64

catenacyber · 2024-09-15T19:11:05Z

@DonggeLiu do you know who from centipede could look at the icu build failure ?
@ussuri ?
The failure log is
https://oss-fuzz-gcb-logs.storage.googleapis.com/log-d8fd93ff-f605-4ffe-a61b-71e43fef6175.txt and shows

The binary should be built with clang 16 and with -fsanitize-coverage=control-flow flag.

@adisuissa would you understand the envoy build failure ?
https://oss-fuzz-gcb-logs.storage.googleapis.com/log-de0ee1b4-4d36-4890-bf8b-6c3e53824da0.txt
Why do not we have debug symbols in this use-after-poison stack trace ?

catenacyber · 2024-09-15T19:15:18Z

cf #12124 for the centipede issue

adisuissa · 2024-09-16T15:36:53Z

@adisuissa would you understand the envoy build failure ? https://oss-fuzz-gcb-logs.storage.googleapis.com/log-de0ee1b4-4d36-4890-bf8b-6c3e53824da0.txt Why do not we have debug symbols in this use-after-poison stack trace ?

I'm not sure why.
Do you know if the error is due to this change or is it because of the update of the clang-chain? If it is the former, consider splitting this change into a different PR, so we can debug just that.

catenacyber · 2024-09-16T19:13:17Z

I'm not sure why. Do you know if the error is due to this change or is it because of the update of the clang-chain? If it is the former, consider splitting this change into a different PR, so we can debug just that.

I think this is due to the clang-chain update.
The clang-chain update is the primary cause of this change because clang 18 finds this warning, see build log here https://oss-fuzz-gcb-logs.storage.googleapis.com/log-b01a6fe2-1743-4a2f-8bf0-273d44169f88.txt

To reproduce, you can, with this PR, run :

python3 infra/helper.py build_image --no-pull base-clang
python3 infra/helper.py build_image --no-pull base-builder
python3 infra/helper.py build_image --no-pull envoy
python3 infra/helper.py build_fuzzers --no-pull envoy

catenacyber · 2024-09-16T19:15:14Z

@DonggeLiu I think every project is good now except envoy...
Do you want to run more tests ?
Should I pin envoy to the current base-builder ?

DonggeLiu · 2024-09-16T22:35:12Z

@DonggeLiu I think every project is good now except envoy... Do you want to run more tests ?

Wow, thanks @catenacyber !

Should I pin envoy to the current base-builder ?

Yep, that's a good idea : )

This is done in the interest of assisting google#12075 and google#11626. Currently the Rust toolchain cannot be updated because the latest nightly uses LLVM 18.1.8 and coverage information breaks. This breakage is because LLVM 18.1.8 records coverage information with version "9" but LLVM 18.0.0 recorded coverage information with version "8". This means that the recordings created by Rust binaries use version "9" which are unreadable by the processing that OSS-Fuzz does with the 18.0.0-based toolchain using version "8". This commit updates the Clang toolchain to the latest 18.x.x release to get the two in sync so the same coverage recording version is used.

Pull in LLVM 18.1.7 with Rust to match the previous Clang update

Fixes wolfssl, muduo

catenacyber · 2024-09-17T19:38:58Z

Here it is, with a rebase on latest master.

What is your plan for this ?

DonggeLiu · 2024-09-17T23:59:50Z

@oliverchang @jonathanmetzman Shall we merge this?

maflcko · 2024-09-18T07:32:46Z

I'd feel more comfortable with another trial build before merge on current master, because some projects may have changed in the meantime from the build last week.

jonathanmetzman · 2024-09-18T15:21:33Z

/gcbrun trial_build.py all

use the standard one from oss-fuzz

As demonstrated by zeek project

current nightly finds dead code

catenacyber · 2024-09-19T07:11:27Z

Step #1: Failed builds: 15/1019, {'cert-manager': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-0ca6aa50-74b1-4ce6-bff8-bf20c3472980.txt', 'elfutils': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-cca5f000-28b3-40cb-b6d1-01ceb96ab83d.txt', 'envoy': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-7784b5d4-6886-4d12-8ff4-f943efb5e3b8.txt', 'grpc-swift': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-7885e51a-4c8e-4c61-a500-62bdb4affb8d.txt', 'gstreamer': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-10b8b2de-c97e-4291-902d-902c4b8b1742.txt', 'immer': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-faffc9b0-450d-463c-9864-8a5b96e0facf.txt', 'libfido2': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-90f5c884-354c-40f2-9b70-9f7f80a476d3.txt', 'libultrahdr': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-9f1c20fd-f678-4e48-a7a6-bd8e237f4ce4.txt', 'libxml2': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-7675453d-81f6-49d5-869f-2628dbf3f055.txt', 'ms-tpm-20-ref': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-699d434d-c40b-4574-92f8-78c73b20fa5f.txt', 'simdutf': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-0545640d-0d93-4f69-b7c2-ae3b2fbba55a.txt', 'tinyusb': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-7be51527-a9cc-4489-ae98-214c082c23c2.txt', 'vlc': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-2a7ac9b4-8b31-47a5-8a6e-5386dca189b0.txt', 'tarantool': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-45fcaa5c-d95e-4682-85e7-b7a333f0913d.txt', 'lua': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-ab3ad87e-5178-42e6-a131-8eaaccc012a2.txt'}

Looks unrelated :

cert-manager is golang, does not seem related
elfutils : file is not valid UTF-8
grpw-swift Invalid Exclude '/src/grpc-swift/FuzzTesting/Sources/EchoModel/echo.proto': File not found.
immer : ERROR: 10.526315789473685% of fuzz targets seem to be broken
libultrahdr : timeouts
libxml2 : arm64
ms-tpm-20-ref : network failure
simdutf : arm64
tinyusb : arm64
vlc : failures in corpus
tarantool : arm64
lua : arm64

Looks related :

envoy : fixing it
gstreamer : to investigate
libfido2 : to investigate

try to make it more clear which changes are needed for which version

catenacyber · 2024-09-19T20:51:02Z

So, I pushed a new version of profraw_update.py

Bug in gstreamer and libfido2 was due to the previous check v9_header.CountersDelta != (sect_prf_cnt - sect_prf_data) which does not always hold anymore, but it is not a problem as this check was meant for one case with rust and profraw version 7

catenacyber · 2024-09-20T15:02:57Z

@jonathanmetzman is it possible to run another trial during the week-end ?

Do you want me to split my last commit in 2 ? The small bug fix, and the bigger rewrite to make code clearer ?

catenacyber mentioned this pull request Aug 16, 2024

Support for LLVM 18 fuzzing #11626

Open

catenacyber force-pushed the profraw9-update branch from 171b315 to dd7bf2b Compare August 16, 2024 10:01

maflcko mentioned this pull request Aug 20, 2024

[infra] Unpin clang version #12385

Closed

maflcko mentioned this pull request Aug 27, 2024

Rust build image needs updating #12410

Open

catenacyber and others added 5 commits September 17, 2024 21:37

infra/coverage: update profraw to version 9

157cbd4

Update Rust to the latest nightly

e1b1393

Pull in LLVM 18.1.7 with Rust to match the previous Clang update

Add -Wno-error=vla-cxx-extension to default flags

3e99576

Fixes wolfssl, muduo

Fix build of samba

b7f2075

catenacyber force-pushed the profraw9-update branch from 77a4a25 to d2518e6 Compare September 17, 2024 19:38

catenacyber force-pushed the profraw9-update branch from d2518e6 to 44c28ad Compare September 17, 2024 19:39

catenacyber added 8 commits September 19, 2024 08:59

librawspeed: disable warning nan-infinity-disabled in clang 18.1

0123258

suricata: do not pin a rustup

3468381

use the standard one from oss-fuzz

infra: profraw_update works for multiple input files at once

657fa51

As demonstrated by zeek project

wasmer: pin old rust version

8e8324a

current nightly finds dead code

rust-lexical: pin old rust builder

de71c40

suricata: remove obsolete files for version 6 fuzzing

e1df9aa

icu: disable centipede with clang 18

62248e4

envoy: pin base-builder before clang18 bump

26ca77a

catenacyber force-pushed the profraw9-update branch from 44c28ad to 26ca77a Compare September 19, 2024 07:11

infra/coverage: siplification for profraw_update

f38b663

try to make it more clear which changes are needed for which version

Update Clang toolchain from 18.0.0 to 18.1.8 #12365

Are you sure you want to change the base?

Update Clang toolchain from 18.0.0 to 18.1.8 #12365

Conversation

catenacyber commented Aug 16, 2024

catenacyber commented Aug 16, 2024

maflcko commented Aug 16, 2024 • edited Loading

catenacyber commented Aug 16, 2024

maflcko commented Aug 22, 2024

catenacyber commented Aug 25, 2024

maflcko commented Aug 26, 2024

catenacyber commented Aug 26, 2024

DonggeLiu commented Aug 27, 2024

catenacyber commented Aug 27, 2024

maflcko commented Aug 27, 2024 • edited Loading

catenacyber commented Aug 27, 2024

DonggeLiu commented Aug 28, 2024

DonggeLiu commented Aug 28, 2024

catenacyber commented Sep 1, 2024

DonggeLiu commented Sep 1, 2024

DonggeLiu commented Sep 3, 2024

maflcko commented Sep 4, 2024

maflcko commented Sep 4, 2024

DonggeLiu commented Sep 4, 2024

catenacyber commented Sep 4, 2024

DonggeLiu commented Sep 4, 2024

maflcko commented Sep 5, 2024

catenacyber commented Sep 5, 2024

maflcko commented Sep 5, 2024

catenacyber commented Sep 5, 2024 • edited Loading

catenacyber commented Sep 5, 2024

catenacyber commented Sep 15, 2024

catenacyber commented Sep 15, 2024

adisuissa commented Sep 16, 2024

catenacyber commented Sep 16, 2024

catenacyber commented Sep 16, 2024

DonggeLiu commented Sep 16, 2024

catenacyber commented Sep 17, 2024

DonggeLiu commented Sep 17, 2024

maflcko commented Sep 18, 2024

jonathanmetzman commented Sep 18, 2024

catenacyber commented Sep 19, 2024 • edited Loading

catenacyber commented Sep 19, 2024

catenacyber commented Sep 20, 2024

maflcko commented Aug 16, 2024 •

edited

Loading

maflcko commented Aug 27, 2024 •

edited

Loading

catenacyber commented Sep 5, 2024 •

edited

Loading

catenacyber commented Sep 19, 2024 •

edited

Loading