[Feature] Implement Native LLM Structured Outputs for the Debugging Engine

[Anmol-Bhatnagar]

Problem Statement

Currently, the analyze_code_structured method in backend/app/services/llm_analysis.py relies heavily on prompt engineering (e.g., instructing the model to "respond ONLY JSON with this shape") and manual string parsing (_extract_json) to generate structured code analysis. This approach is inherently brittle. If the LLM hallucinates markdown blocks incorrectly, misses a bracket, or returns a string instead of an integer for a debugging line number, the manual parser throws an invalid_json_payload exception. This not only breaks the API response but can also crash downstream consumers like the VS Code extension that rely on strict types for rendering line markers.

Proposed Solution

Migrate the AI calls to use native LLM Structured Outputs (JSON Schema validation), which is supported by most modern AI providers.

1. Define a strict schema (e.g., via Pydantic) representing the exact required shape for the code analysis payload (including explanation, debugging, suggestions, complexity, and optimized_version).
2. Pass this schema directly into the LLM provider's structured output parameter within LLMAnalysisClient._chat_completion.
3. Remove the legacy _extract_json string-stripping fallback logic entirely, as the API will natively guarantee the JSON shape and data types.

Alternatives Considered

• Enhancing the Regex in _extract_json: We could write more robust regex to handle edge cases in the LLM's text output, but this does not solve the core issue of type enforcement (e.g., ensuring line is always an integer and severity is a strict enum).
• Using Third-Party Wrappers (e.g., Instructor or Guardrails): While these enforce schemas, they introduce unnecessary dependencies and latency. Using the native structured output parameters of the underlying models (like gpt-4o-mini) is cleaner and more performant.

Additional Context

• Target File: backend/app/services/llm_analysis.py
• Target Method: analyze_code_structured
• As a contributor participating in GSSoC 2026, I would love to be assigned to this issue so I can implement this architectural stability enhancement for the backend!

[rashi-gupta-08]

Hi @imDarshanGK — I'd like to take this on as part of GSSoC '26. Kindly assign it to me if available.

I've cloned the repo and studied the relevant files. Here's my exact implementation plan so you can evaluate it before assigning.

---

What I found in the codebase

The project uses FastAPI + Pydantic v2 (backend/app/schemas.py) and an LLM abstraction in backend/app/services/ai_provider.py. The current analyze_code_structured flow builds a prompt string that instructs the model to "respond ONLY in JSON", then runs the response through _extract_json to strip markdown fences and parse manually — a pattern that breaks on any deviation in the model's output format.

---

My implementation plan (3 surgical steps)

Step 1 — Define a strict Pydantic v2 schema in schemas.py

Rather than a new file, I'll co-locate the output schema with the existing Pydantic models already in schemas.py:

class DebuggingItem(BaseModel):
    line: int          # enforced as int, never a string
    severity: Literal["error", "warning", "info"]
    message: str

class CodeAnalysisOutput(BaseModel):
    explanation: str
    debugging: list[DebuggingItem]
    suggestions: list[str]
    complexity: Literal["O(1)", "O(n)", "O(n²)", "O(log n)", "O(n log n)", "unknown"]
    optimized_version: str

Step 2 — Pass the schema into _chat_completion via the provider's native structured output param

For OpenAI-compatible providers this is the response_format parameter:

# In LLMAnalysisClient._chat_completion (ai_provider.py)
response = client.beta.chat.completions.parse(
    model=self.model,
    messages=messages,
    response_format=CodeAnalysisOutput,   # pass the Pydantic class directly
)
return response.choices[0].message.parsed  # already a validated CodeAnalysisOutput

This guarantees: line is always an integer, severity is always a valid enum value, no JSON parsing step needed at all.

Step 3 — Delete _extract_json and update tests

_extract_json can be removed entirely. I'll update backend/tests/test_endpoints.py to:

• Assert the debugging[].line field is int, not str
• Assert severity rejects values outside the enum
• Add a test that previously-failing malformed LLM output no longer causes invalid_json_payload

What I won't do

I agree with the issue author — no Instructor, no Guardrails, no third-party schema wrappers. Native structured outputs only.

Backward compatibility

If the deployed provider doesn't support structured outputs (e.g. older self-hosted models), I'll add a graceful fallback flag in .env.example (USE_STRUCTURED_OUTPUT=true) so the legacy _extract_json path can be toggled on — but off by default.

Files touched

backend/app/schemas.py · backend/app/services/ai_provider.py · backend/tests/test_endpoints.py · .env.example · CONTRIBUTING.md (update dev notes)

Can open a draft PR within 2–3 days. Happy to discuss any part of the approach before you assign. 🙏

[virakshi07]

Hi @imDarshanGK,

I'd like to work on this issue as part of GSSoC 2026. Kindly assign it to me if it's available.

I've reviewed the issue description and inspected the relevant code paths. My understanding is that the current implementation relies on prompt instructions plus manual JSON extraction/parsing, which can fail when the model returns malformed output or incorrect data types.

Proposed implementation approach

Step 1 — Define a strict response schema

I'll create a dedicated Pydantic schema representing the expected structured analysis response, including:

• explanation
• debugging entries
• suggestions
• complexity
• optimized_version

This will ensure type validation happens automatically rather than through manual parsing.

Step 2 — Integrate native structured outputs

I'll update the LLM response handling to use provider-supported structured outputs / JSON schema validation wherever supported, allowing the model response to be validated directly against the schema.

Step 3 — Remove fragile parsing logic

After schema validation is integrated, I'll simplify the existing extraction/parsing flow and remove unnecessary manual JSON-cleanup logic where appropriate.

Testing Plan

• Add tests for valid structured responses.
• Add tests for malformed LLM outputs.
• Verify incorrect field types are rejected gracefully.
• Ensure existing API contracts remain backward compatible.

Files likely involved

• backend/app/services/llm_analysis.py
• backend/app/services/ai_provider.py
• backend/app/schemas.py
• Related backend tests

I can start working on this immediately after assignment and will submit a well-tested PR following the project guidelines.

Thank you for your consideration.