Resources requirement when building CodeQL database #311

ducphanduyagentp · 2021-03-13T03:10:38Z

ducphanduyagentp
Mar 13, 2021

Hi there,
Recently I am doing research on a large OSS project, chromium. Since I have limited resources, and building databases for such large projects requires a lot of it, is there a way to know or anticipate the amount of memory/storage needed? My database create process was stopped because I ran out of memory so I wonder if I can also continue the build.

Answered by ducphanduyagentp

Mar 15, 2021

So at the end of the target build process, I got errors from Java. I hope this can provide some more insight to the situation if it is relevant.

Finalizing database at /home/mac/chromium/src/chromium-database.
Oops! A fatal internal error occurred.
java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
	at com.semmle.util.exception.Exceptions.asUnchecked(Exceptions.java:103)
	at com.semmle.inmemory.trap.TrapImporter.run(TrapImporter.java:106)
	at com.semmle.cli2.ql.dataset.ImportCommand.executeSubcommand(ImportCommand.java:86)
	at com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110)
	at com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(Subcomman…

View full answer

intrigus-lgtm · 2021-03-13T12:35:25Z

intrigus-lgtm
Mar 13, 2021

OOI: What are the specs of your computer?

Well, building chromium itself requires 100+ GBs of free disk space and at least 8 GB of RAM.

As far as I know it's not possible to incrementally create a codeql database. So you can also not use ccache.

I have a codeql database for Mozilla Firefox that takes ~15 GB of disk space and a chrome database will likely take the same amount of space or even more. If you run a query, codeql will also take additional space as a cache.

Depending on the specs of your computer it might be easier to build the database on a beefy machine in the "cloud" and then do the codeql analysis also in the cloud or on your local machine.

3 replies

ducphanduyagentp Mar 13, 2021
Author

Thank you for the reply. It was a 12 core, 26GB memory with 200+GB of storage. I think the only issue is memory. I had no issue building chromium itself but I can imagine CodeQL needs to do more work than that.

intrigus-lgtm Mar 13, 2021

I don't know if it would help, but did you try limiting the parallelism?
I.e. building chrome using e.g. only 4 cores?
Also, can you post the output of your database creation command + the creation command itself?

ducphanduyagentp Mar 15, 2021
Author

Unfortunately the output was quite large but here is the command I've used: ./build/install-build-deps.sh && gclient runhooks && gn gen out/Default && codeql database create chromium-database --language=cpp --command="autoninja -C out/Default chrome"

criemen · 2021-03-15T10:00:25Z

criemen
Mar 15, 2021
Collaborator

Hi @ducphanduyagentp ,
I recently built a lot of chromium databases, some remarks:

If you're on Windows, we unfortunately have a bug that I fixed recently that affects Chromium extraction. We haven't shipped the fix yet (see Missing Source Files from Database codeql-cli-binaries#39).
I successfully built a codeql database on my laptop with 16 cores and 64GB RAM - at times, most of the RAM was used. However, the process is quite painful, and I switched to using D64ds_v4 Azure VMs (64 virtual cores, 256GB RAM). This process also used almost all RAM. Thus, that hints that a ratio of 4GB RAM per core building chromium is a good idea, so I'd recommend you limit your parallelism in the build process to 6 cores.
Unfortunately, there's no general rule about how much memory database creation will take.
If this is your first time playing with CodeQL, Chromium will be a rather painful experience - it's huge, so everything takes a long time. If you want to learn more how to use CodeQL, I recommand starting with a smaller project.
Here's a readily built database with chromium HEAD from ~1-2 weeks ago, built on Linux https://drive.google.com/file/d/1sN95zSPY3bq4sxcNyQhgZRiJZXGDmA_2/view?usp=sharing

2 replies

ducphanduyagentp Mar 15, 2021
Author

Thank you for the very detailed reply. I realized I've provided a small inaccurate detail on my response above. The machine was 6 core with 2 threads/core. I am not sure if that makes any difference, but I will try to adjust the build based on your suggestion and see how it goes. Also I am using a Linux VM inside a Macbook, if this detail helps.

18Fl Jun 28, 2023

Hey, I have meet same problem again... would u mind give me anytips? here is my original report : github/codeql#13562

I tried ubuntu 18.04, 20.04, 22.04. failed and failed again. And I use VMWare Pro with ubuntu 20.04 failed again... and I try to use ninja -C out/codeql chrome instead of autoninja -C out/codeql chrome . and failed again...

And I have 64GB RAM and 69GB available space .. and I have 12 core...

Any sugesstion, thank u very much!

ducphanduyagentp · 2021-03-15T11:08:18Z

ducphanduyagentp
Mar 15, 2021
Author

So at the end of the target build process, I got errors from Java. I hope this can provide some more insight to the situation if it is relevant.

Finalizing database at /home/mac/chromium/src/chromium-database.
Oops! A fatal internal error occurred.
java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
	at com.semmle.util.exception.Exceptions.asUnchecked(Exceptions.java:103)
	at com.semmle.inmemory.trap.TrapImporter.run(TrapImporter.java:106)
	at com.semmle.cli2.ql.dataset.ImportCommand.executeSubcommand(ImportCommand.java:86)
	at com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110)
	at com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:159)
	at com.semmle.cli2.database.FinalizeCommand.executeSubcommand(FinalizeCommand.java:110)
	at com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110)
	at com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:159)
	at com.semmle.cli2.database.CreateCommand.executeSubcommand(CreateCommand.java:126)
	at com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:430)
	at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:201)
	at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:209)
	at com.semmle.cli2.CodeQL.main(CodeQL.java:93)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.brotli.dec.Decode.maybeReallocateRingBuffer(Decode.java:542)
	at org.brotli.dec.Decode.readNextMetablockHeader(Decode.java:581)
	at org.brotli.dec.Decode.decompress(Decode.java:741)
	at org.brotli.dec.BrotliInputStream.read(BrotliInputStream.java:145)
	at java.base/sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
	at java.base/sun.nio.cs.StreamDecoder.implRead(Unknown Source)
	at java.base/sun.nio.cs.StreamDecoder.read(Unknown Source)
	at java.base/java.io.InputStreamReader.read(Unknown Source)
	at java.base/java.io.BufferedReader.fill(Unknown Source)
	at java.base/java.io.BufferedReader.readLine(Unknown Source)
	at java.base/java.io.BufferedReader.readLine(Unknown Source)
	at com.semmle.util.trap.dependencies.TextFile.load(TextFile.java:61)
	at com.semmle.util.trap.dependencies.TrapSet.<init>(TrapSet.java:56)
	at com.semmle.inmemory.trap.TRAPLinker.addTrapAndDependenciesToSet(TRAPLinker.java:408)
	at com.semmle.inmemory.trap.TRAPLinker.computeObjectToTrap(TRAPLinker.java:392)
	at com.semmle.inmemory.trap.TRAPLinker.getTasks(TRAPLinker.java:178)
	at com.semmle.inmemory.trap.TrapImporter.lambda$scanAndLink$2(TrapImporter.java:180)
	at com.semmle.inmemory.trap.TrapImporter$$Lambda$212/0x00000008002efc40.apply(Unknown Source)
	at com.semmle.util.concurrent.FutureUtils.lambda$null$4(FutureUtils.java:120)
	at com.semmle.util.concurrent.FutureUtils$$Lambda$214/0x00000008002ef440.get(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

3 replies

criemen Mar 15, 2021
Collaborator

That's unexpected (to me)! I'll talk to my colleagues and get back to you.

ducphanduyagentp Mar 15, 2021
Author

I forgot to note that this is from the initial problematic build I mentioned in the original post. The retry is still running fine.

criemen Mar 15, 2021
Collaborator

You're seeing an OOM failure that happens after the actual chrome build is completed, so it doesn't have anything to do with how parallel the build is.
This step is very memory-intense.
You can give it more memory by passing -J-Xmx1234M to codeql database create, replacing 1234 by 80% of your available machine memory (in megabytes).
Hopefully that is enough for the database creation to go through.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resources requirement when building CodeQL database #311

{{title}}

Replies: 3 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Resources requirement when building CodeQL database #311

ducphanduyagentp Mar 13, 2021

Replies: 3 comments · 8 replies

intrigus-lgtm Mar 13, 2021

ducphanduyagentp Mar 13, 2021 Author

intrigus-lgtm Mar 13, 2021

ducphanduyagentp Mar 15, 2021 Author

criemen Mar 15, 2021 Collaborator

ducphanduyagentp Mar 15, 2021 Author

18Fl Jun 28, 2023

ducphanduyagentp Mar 15, 2021 Author

criemen Mar 15, 2021 Collaborator

ducphanduyagentp Mar 15, 2021 Author

criemen Mar 15, 2021 Collaborator

ducphanduyagentp
Mar 13, 2021

Replies: 3 comments 8 replies

intrigus-lgtm
Mar 13, 2021

ducphanduyagentp Mar 13, 2021
Author

ducphanduyagentp Mar 15, 2021
Author

criemen
Mar 15, 2021
Collaborator

ducphanduyagentp Mar 15, 2021
Author

ducphanduyagentp
Mar 15, 2021
Author

criemen Mar 15, 2021
Collaborator

ducphanduyagentp Mar 15, 2021
Author

criemen Mar 15, 2021
Collaborator