Handling OutputValidationError retry loops and workflow hangs with local LLMs

[seyeop03]

I am currently testing Shannon using a local LLM (Qwen 3.5 122B via an OpenAI-compatible endpoint). During my testing, I've encountered a few workflow stability issues and would appreciate some guidance on how to configure or handle them.

1. OutputValidationError and Temporal Retry Loops

The pre-recon agent successfully progresses through many turns but eventually fails with an OutputValidationError.

The main challenge is that Temporal treats this validation failure as a retryable error (nonRetryable: false). As a result, instead of failing fast, the system enters a retry loop. I observed a single pre-recon activity running for over 75 minutes (elapsedSeconds: 4484) across multiple attempts, consuming significant resources and appearing to hang.

Temporal Dashboard Log:

{
  "sdkComponent": "worker",
  "taskQueue": "shannon-pipeline",
  "attempt": 2,
  "activityType": "runPreReconAgent",
  "error": "ApplicationFailure: Agent pre-recon failed output validation",
  "cause": {
    "type": "OutputValidationError",
    "nonRetryable": false
  },
  "durationMs": 386443
}

[ezl-keygraph]

Output quality depends heavily on the model. Shannon is built on top of the Anthropic Agent SDK and is optimized and primarily tested with Anthropic Claude models. Alternative providers may produce inconsistent results (including failing early phases like Recon) depending on the model.

[ariapalmer1231-png]

Thanks for the clarification! Good to know that Shannon performs best with Anthropic Claude models and that results may vary with other providers.” Le sam. 28 mar. 2026, 01:14, Ezhil ***@***.***> a écrit :

…

*ezl-keygraph* left a comment (KeygraphHQ/shannon#260) <#260 (comment)> Output quality depends heavily on the model. Shannon is built on top of the Anthropic Agent SDK and is optimized and primarily tested with Anthropic Claude models. Alternative providers may produce inconsistent results (including failing early phases like Recon) depending on the model. — Reply to this email directly, view it on GitHub <#260?email_source=notifications&email_token=CAYBNVIWF7XVHYPNGSIEINT4S5UTDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJUG4ZTGNBZGE22M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#issuecomment-4147334915>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CAYBNVMMWXLURMVTHB6JS7T4S5UTDAVCNFSM6AAAAACXBT2K7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCNBXGMZTIOJRGU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[kevinward1231-ai]

Following up on this, do you have recommendations for specific Claude models that offer the best balance of performance and reliability for the Recon phase? Additionally, are there any plans to improve support for local LLMs to prevent these retry loops in the future?

…

On Mon, Mar 30, 2026, 9:51 AM ariapalmer1231-png ***@***.***> wrote: *ariapalmer1231-png* left a comment (KeygraphHQ/shannon#260) <#260 (comment)> Thanks for the clarification! Good to know that Shannon performs best with Anthropic Claude models and that results may vary with other providers.” Le sam. 28 mar. 2026, 01:14, Ezhil ***@***.***> a écrit : > *ezl-keygraph* left a comment (KeygraphHQ/shannon#260) > < #260 (comment)> > > Output quality depends heavily on the model. Shannon is built on top of > the Anthropic Agent SDK and is optimized and primarily tested with > Anthropic Claude models. Alternative providers may produce inconsistent > results (including failing early phases like Recon) depending on the model. > > — > Reply to this email directly, view it on GitHub > < #260?email_source=notifications&email_token=CAYBNVIWF7XVHYPNGSIEINT4S5UTDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJUG4ZTGNBZGE22M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#issuecomment-4147334915>, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/CAYBNVMMWXLURMVTHB6JS7T4S5UTDAVCNFSM6AAAAACXBT2K7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCNBXGMZTIOJRGU> > . > You are receiving this because you are subscribed to this thread.Message > ID: ***@***.***> > — Reply to this email directly, view it on GitHub <#260?email_source=notifications&email_token=CAYAHXPARUEPYOKWONKFMOT4TIYRNA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJVGMZTIMZXG4YKM4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#issuecomment-4153343770>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CAYAHXPEKVZVMRRHOWSMJUT4TIYRNAVCNFSM6AAAAACXBT2K7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCNJTGM2DGNZXGA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>