Add fuzzing testing

[baentsch]

Follow the approach taken by OpenSSL or another one: Suggestions welcome below.

[silvergasp]

I'd like to suggest and champion an effort integrating liboqs with google/oss-fuzz. If you aren't familiar with it, Google offers a free (for open source) continuous fuzzing service called OSS-fuzz.

I've opened up a draft pull request to add a super basic fuzz-testing harness here #1905. It needs a little bit of tidying before it's ready to go but I thought I'd guage interest before polishing it up.

The general process would look something like this.

• Merge Add a basic fuzz testing harness for Dilithium2 #1905
• Apply for integration into oss-fuzz, this is a pretty simple PR into the repo that I can do on your behalf. All I'd need is a comment with approval from someone with write access to this repo.
• Integrate the project, this includes a Dockerfile and a shell script to build liboqs in the oss-fuzz environment.
• OSS-fuzz integration would be complete and the fuzzer would run every night for a few hours on a distributed cluster.
• Integrate clusterfuzzlite, this would run a stripped down version of oss-fuzz in github actions for X minutes (usually 10mins, but this is configurable) on every PR. The motivation behind this differs slightly in that it's intended to catch shallow bugs before they are merged. In comparison OSS-fuzz will run for significantly longer each night to attempt to find those harder to reach bugs (if they exist).

Let me know what your thoughts are on this :)

[baentsch]

I'd like to suggest and champion an effort integrating liboqs with google/oss-fuzz.

This would be very welcome, @silvergasp ! Thanks a million for the suggestion and apparent commitment!! As you seem to be an Independent Contributor like me (trying to establish that notion towards the corporate/LF folks :) I shall provide any possibly needed assistance with this, e.g., helping #1905 move to merge-ability, so please be sure to tag me when needed.

Let me know what your thoughts are on this :)

In a nutshell: LGTM :) Details to follow once this moves forward, I guess.

[silvergasp]

This would be very welcome, @silvergasp ! Thanks a million for the suggestion and apparent commitment!! As you seem to be an Independent Contributor like me (trying to establish that notion towards the corporate/LF folks :) I shall provide any possibly needed assistance with this, e.g., helping #1905 move to merge-ability, so please be sure to tag me when needed.

Cheers mate. Yeah I'm an independent contributor. I've gone ahead and opened up a draft pull request over at oss-fuzz google/oss-fuzz#12408 that will function as both the integration and the "application" process. Everything seems to be working well locally and the CI is passing. A few things I'll need to move that PR forward (in the order that they need to happen);

• Add a basic fuzz testing harness for Dilithium2 #1905 will need to be merged before the oss-fuzz PR goes through as I've currently got the oss-fuzz configuration fetching my personal fork as it contains the fuzz harness. I'll need to swap that back to the main fork before the oss-fuzz PR is merged.
• I need an email address for the primary_contact field. This will be the email that will receive updates whenever a security vulnerability or other bug is found by oss-fuzz. It will also be what you'll use to login into the bug-tracker and dashboard for various tools analyzing fuzzing bugs. It is preferable that this is a gmail/google account, otherwise you won't be able to login to the bug-tracker. I can also add many other emails that will get CC'd into these bug reports, if you would like multiple people to be notified. Please note that these emails will be stored in the oss-fuzz repo in plain text.
• You to comment on the oss-fuzz PR (once it's complete) saying that you approve of the integration.

I've gone ahead and polished up #1905 and I think it's ready for review. It's just a super-basic fuzzer that's mostly adapted from one of the examples. But the goal was just to get all the infrastructure in place so that more complex/useful fuzzer's are possible and worth the effort.

[baentsch]

I need an email address for the primary_contact field.

Ideally you'd use [email protected] (listed at https://openquantumsafe.org/liboqs/security.html#reporting-security-bugs) that different people read. Please let us know if it must be a gmail account (not really ideal, though).

[dstebila]

I need an email address for the primary_contact field.

Ideally you'd use [email protected] (listed at https://openquantumsafe.org/liboqs/security.html#reporting-security-bugs) that different people read. Please let us know if it must be a gmail account (not really ideal, though).

We can also set up a dedicated fuzz-related alias if it's helpful.

[silvergasp]

I need an email address for the primary_contact field.

Ideally you'd use [email protected] (listed at https://openquantumsafe.org/liboqs/security.html#reporting-security-bugs) that different people read. Please let us know if it must be a gmail account (not really ideal, though).

It doesn't need to be a gmail account you just won't have access to the dashboard and will only receive email updates. The dashboard has a bunch of useful features for analysing fuzzing performance and also automatically bisecting bugs to see when they where introduced. So I'd recommend adding at least one gmail account that someone on the core team has access to. I can add as many email accounts as you like, but only the gmail ones will have full access. I know some projects will setup a specific separate gmail account for this purpose so that it's separate from their personal accounts e.g. rhai.

We can also set up a dedicated fuzz-related alias if it's helpful.

This might be worthwhile as oss-fuzz will send off an email everytime a fuzzer crashes this includes both security and non-security related bugs. In some projects this can be a lot of emails and it's often hard to triage them all without the filtering tools on the dashboard. If you use a gmail account you can configure your notifications to only receive security updates rather than every crash report.

I'll leave that up to you to decide :)

[dstebila]

@ryjones Do you have a preferred way to handle setting up a Gmail account for project use? As you can see in the comment above there is apparently a benefit to using a Gmail account for this fuzzing dashboard rather than just a generic email address.

[ryjones]

@ryjones Do you have a preferred way to handle setting up a Gmail account for project use? As you can see in the comment above there is apparently a benefit to using a Gmail account for this fuzzing dashboard rather than just a generic email address.

Let me ask around. I don't think I can set up a gmail/google apps account for any of the domains PQCA controls.