synthesis(question_gen/adversarial): massive code duplication in _parse_response (~150 lines nearly identical)

[himanshu231204]

Summary

The _parse_response() method in openagent_eval/synthesis/question_gen.py (~148 lines, lines 80–228) is nearly identical to _parse_response() in openagent_eval/synthesis/adversarial.py (~152 lines, lines 281–433). Both implement the same 5-tier fallback parsing strategy (single JSON object → JSON array → individual JSON objects → regex → fallback regex) with the exact same logic and cleanup steps.

The only functional difference is:

• question_gen.py: Validates if question and answer: (both required)
• adversarial.py: Validates if question: (only question required; answer can be empty for unanswerable cases)

Impact

Medium. Any bug encountered in the LLM response parsing must be fixed in two separate files with identical logic. This has already been demonstrated by the .format() crash bug (see related issue) — the fix needs to be applied in both files independently. Similarly, any improvement to the parsing strategy (support for new response formats, better error messages, etc.) requires changes in two places, creating a maintenance burden.

Relevant Code

# question_gen.py, lines 80-228
def _parse_response(self, raw_response, context, source_document, chunk_index):
    # 148 lines of parsing logic with 5 fallback strategies

# adversarial.py, lines 281-433  
def _parse_response(self, raw_response, context, test_type, source_document, chunk_index):
    # 152 lines of NEARLY IDENTICAL parsing logic with 5 fallback strategies

Suggested Fix

Extract the shared parsing logic into a common helper function, either in a new synthesis/parser.py utility module or as a static method on a shared base class. The helper should accept a callback or flag for the question_only vs question_and_answer validation difference.