Repository navigation
Report an astgen run that fails, and keep the files it finished - #227
Merged
Merged
Conversation
AstGenRunner dropped a failed astgen run without a word and handed on no files, so a project astgen ran out of memory on came out as an atom with one file and no methods. astgen writes each file's AST as it finishes it, so the files before the failure are sound: keep them, and say on stderr that astgen failed and how many files the atom holds, with astgen's last words. stderr because atom does not print chen's log. Signed-off-by: Team AppThreat <cloud@appthreat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When astgen fails, AstGenRunner throws the result away without a word and hands on no files. A project astgen ran out of memory on came out as an atom with one file and no methods, and nothing told the user why reachables found no flows.
astgen writes each file's AST as soon as it finishes it, so the files written before a failure are sound. The runner now keeps them, and prints on stderr that astgen failed, how many files the atom holds, and astgen's last few lines (the V8 stack frames are dropped). It uses stderr because atom ships slf4j-nop, so anything chen sends to its logger never appears. CdxPass and ChennaiTagsPass report their problems on stderr for the same reason. A failed run still isn't stored in the parse cache, as before.
I checked this against the astgen crash it came from. The globally installed astgen 4.5.1 runs out of memory on node-express-realworld-example-app. With this change atom reports:
astgen failed on …/node-express-realworld-example-app, so the atom holds only the 7 files it finished: … FATAL ERROR: Ineffective mark-compacts near heap limit …The atom then has 89 methods and 234 tags, instead of nothing. AppThreat/atom-parsetools#25 fixes that particular crash.
jssrc2cpg (673) and x2cpg (141) pass. There's no unit test, because the runner calls
astgenfrom PATH and the forked test JVM has no way to substitute a failing one.