Merge the default one-prompt method into the new agentic workflow (Analysis+Enhancement) #785
Conversation
|
/gcbrun exp -n dg |
1 similar comment
|
/gcbrun exp -n dg |
|
/gcbrun exp -n dg |
|
/gcbrun exp -n dg |
|
/gcbrun exp -n dg |
|
/gcbrun exp -n dg1 |
|
/gcbrun exp -n dg1 |
|
/gcbrun exp -n dg2 |
|
/gcbrun exp -n dg3 |
|
/gcbrun exp -n dg3 |
|
@DonggeLiu LGTM for the merge. |
|
/gcbrun exp -n dg4 |
|
/gcbrun exp -n dg5 |
DonggeLiu
requested review from
oliverchang,
mihaimaruseac and
DavidKorczynski
|
The report looks good! |
| @@ -0,0 +1,73 @@ | |||
| """An LLM agent to improve a fuzz target's runtime performance. | |||
There was a problem hiding this comment.
Fixed, thanks!
I will add a CI check later.
|
|
||
| MAX_ROUND = 100 | ||
| MAX_ROUND = 5 |
There was a problem hiding this comment.
is it intentional to set it to 5? Seems like a big reduction from 100?
There was a problem hiding this comment.
Good point—it's worth trying a higher value (e.g., 10).
The original one-prompt workflow was capped at 5 rounds because each round built a fuzz target on a new Cloud Build, which was expensive. Now that we're using a cheaper approach, we can afford more rounds. However, since the one-prompt method only gets build error messages, if the LLM can't generate a valid target within a few rounds, additional rounds likely won't help, so 100 would be excessive.
For context, 100 rounds were used in the agentic method where rounds are lightweight and can obtain new information (e.g., bash commands).
| generated_project, benchmark_target_name, | ||
| self.work_dirs.run_logs_target(benchmark_target_name, iteration)) | ||
| # run_log_path = self.work_dirs.run_logs_target(benchmark_target_name, | ||
| # iteration) |
There was a problem hiding this comment.
do we want to keep these comments?
| @@ -538,7 +548,8 @@ def build_and_run_local( | |||
| run_result.crashes, run_result.crash_info, \ | |||
| run_result.semantic_check = \ | |||
| self._parse_libfuzzer_logs(f, project_name, flag) | |||
| run_result.succeeded = not run_result.semantic_check.has_err | |||
| # run_result.succeeded = not run_result.semantic_check.has_err | |||
There was a problem hiding this comment.
intentional to leave this commented out?
| generated_project) | ||
| # # Overwrite the Dockerfile to be caching friendly | ||
| # oss_fuzz_checkout.rewrite_project_to_cached_project_chronos( | ||
| # generated_project) |
There was a problem hiding this comment.
could you leave a comment on why this is being commented out?
| # run_result.semantic_check.type, | ||
| # run_result.triage, | ||
| # compile_error=build_result.log_path, | ||
| # compile_log=build_result.log_path)) |
There was a problem hiding this comment.
could you leave a comment here?
| local_file.write(tmp_file.read()) | ||
|
|
||
| os.remove(tmp_path) | ||
| # blob.download_to_filename(local_path) |
There was a problem hiding this comment.
could you leave a comment/remove/uncomment?
| @@ -4,9 +4,13 @@ | |||
| import json | |||
There was a problem hiding this comment.
missing a license here as well?
| @@ -4,7 +4,7 @@ | |||
|
|
|||
| import logger | |||
| @@ -4,6 +4,7 @@ | |||
|
|
|||
| @@ -0,0 +1,73 @@ | |||
| """An LLM agent to improve a fuzz target's runtime performance. | |||
container_toolfrom agentic workflow instead of the oldtarget_experiment.pyscript.Build and collect results in 'OnePromptPrototyper'container_toolin OnePromptPrototyper/OnePromptEnhancer.Next:
JCC.cyclein results.cyclewhen saying logs.Run/BuildResults with the new one.