OSS-fuzz integration

[Segwaz]

Hello,
I would like to integrate this project in OSS-fuzz. It's a free automated fuzzing service for open-source projects, maintained by Google, that led to the discovery of 10k+ vulnerabilities in open-source codebases (here is the repo: https://github.com/google/oss-fuzz). Are you interested ?

[erthink]

Are you interested ?

In short, yes.
I am interested and will provide all possible assistance as I can and as much as I have free time.

[Segwaz]

great, I'm on it then.

[Segwaz]

A 'primary contact' email address is required, used to receive issues found. I can put myself in the 'cc' fields but the primary one has to be from an active maintainer/contributor. It also has to be linked to a Google account. Please let me know which one I can use for this purpose (it will be specified in a publicly visible file similar to this one https://github.com/google/oss-fuzz/blob/master/projects/libarchive/project.yaml).

[erthink]

A 'primary contact' email address is required...
It also has to be linked to a Google account.

You should use the address (leo@yuriev.ru) specified in the commits to this repo and linked by the GPG-key used for signing ones.
AFAIK, the noted email will linked to someone Google account. However, it has not been used on any device for a long time. Therefore, if need to log in through this account, it may be difficult (I will need to (re)set a new password using only email, etc).

[erthink]

homepage: "https://libmdbx.dqdkfa.ru"
language: c++
primary_contact: "leo@yuriev.ru"
auto_ccs:
  - "segwaz@gmail.com"
  - "libmdbx@dqdkfa.ru"
sanitizers:
  - address
  - memory:
     experimental: True
  - undefined
main_repo: 'https://git@git.sourcecraft.dev/dqdkfa/libmdbx.git'

[Segwaz]

Thanks. I believe they will verify your email address so you will need to access it. I'm writing the fuzzing harnesses and will let them run for awhile prior to submitting the project so it is accepted more easily, so no hurry.

[erthink]

@Segwaz, among other things, there are three technical difficulties for automatic fuzzing testing:

1. Historically, the libmdbx API comes from LMDB (users still ask for compatibility), and some parts are dramatically unclean and confusing.
2. Most operations depend not only on the incoming data/argumenst, but also on the database structure (including all aspects of the b+tree), which in turn depends on:
   • the entire history of data insertion/deletion/modification operations;
   • actions of other processes interacting with the database, including high-concurrently using lockfree approach.
3. Currently, there is no API convenient for integration with fuzzing systems:
   • no complete binding to Python (using SWIG is in hte TODO-list);
   • no CLI front-end, it also still in the TODO-list (libmdbx CLI front-end improvement ideas #218).

[Segwaz]

None of it is a problem except for multi process interactions which I will probably not implement. It still leaves a lot of room for raw db format fuzzing (which I'm working on right now alongside oss-fuzz integration) and higher level C API fuzzing (point 2) using protobufs or a custom seed format. There is no need for CLI or python bindings at all.

[Segwaz]

The harnesses must be stored in libdbmx repo, not OSS-fuzz side. Could you open pull requests so I can put them here ?

[erthink]

Could you open pull requests so I can put them here ?

Done.
But I'll probably have to manually integrate your changes, since the github repository contains an amalgamated derived form of the source code only.
If it makes sense, then in the future I will be able to provide you with access to the primary repository on SourceCraft. But we are not ready for this yet and there are too many other tasks before it can be done.

[erthink]

@Segwaz, nonetheless of above, I'm waiting for a pull request from you to see a whole picture of the proposed/required changes, and then manually integrate ones into the main repository.

[Segwaz]

I was busier than expected, but here it is (the first part of it that is). I'll add a README with explanations on how to test/run it in a moment.

[Segwaz]

When using libfuzzer (what OSS-fuzz run by default), assertions should no be used against invalid inputs. Could assert at mdbx.c:19720 be replaced by a regular error condition ? I proposed such an alternative in a new pull request.The assertion is triggered by libfuzzer when a meta with high txnid is skipped due to invalid file size but later selected as recent.
Solving it let the fuzzer continue which revealed another bug that I will look into prior to continuing the integration with the C API harness.

[erthink]

As you probably already know from the mail notifications, the libmdbx-fuzzing repository has been created and you have been given all the rights.

Inside, I added the libmdbx as a submodule, and added the license with a couple of files - to make it clearer what I was talking about earlier.

You are welcome!