Conversation
The tool called a separate server-side LLM (FMSR_MODEL_ID, default WatsonX Llama-3.3-70B), so fmsr scenario results depended on that model rather than the agent under test, and silently degraded to 'LLM unavailable' without WatsonX credentials. Agents now derive failure modes from their own domain knowledge and persist them with add_failure_modes, matching how the failure-mode/sensor mapping is already handled. Removes the LLM backend, prompts, related tests/fixtures, the FMSR_MODEL_ID docs, and the preflight WatsonX check. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
- Look up records by comparing normalised asset_class values in Python instead of by _id. The loader keys docs as fm:<raw asset_class>, so classes like 'Hydraulic Pump' were unreachable by the lowercased key. - Normalise only case, punctuation and whitespace; stop stripping digits, which turned 'CO2 Compressor' into 'co compressor'. - Keep the stored asset_class name when updating a record. - Suggest close matches in the not-found error. Tests: FakeDatabase now returns copies from find() and enforces _rev conflicts like CouchDB; add coverage for write failures, concurrent updates, duplicate normalised classes and non-fm: _ids. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
5 tasks
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
fix(fmsr): match failure-mode records on asset_class only
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Simplifies the FMSR MCP server so it only stores and looks up failure-mode catalogs. It no longer calls an LLM of its own. This PR also fixes how asset classes are matched.
1. Drop
generate_failure_modes. The tool called a separate LLM inside the server (FMSR_MODEL_ID, default WatsonX Llama-3.3-70B). Because of that:{"error": "LLM unavailable"}, solite-profile fmsr scenarios scored badly and no error was shown;Agents now come up with failure modes from their own domain knowledge and save them with
add_failure_modes. The failure-mode/sensor mapping already works this way. MCP sampling was considered, but none of the three runners support it yet.2. Fix asset-class matching (merged in from #553).
fm:<raw asset_class>, while the server lowercased the input and looked upfm:<key>. Records such as"Hydraulic Pump"were never found, andadd_failure_modescreated duplicates of them.CO2 Compressorbecameco compressor, and3-Phase Motorbecamephase motor.Changes
Server (
src/servers/fmsr/main.py)generate_failure_modes, its result model, the prompts and the LLM setup (FMSR_MODEL_ID). The server now makes no LLM calls.asset_classonly, never_id. Input and stored names go through the same normalisation, which ignores case, punctuation,_and whitespace and leaves digits alone._idwins.add_failure_modeskeeps the storedasset_classname. New records are created asfm:<normalised name>.add_failure_modes.Tests (
src/servers/fmsr/tests/){get_failure_modes, add_failure_modes}.FakeDatabase.findnow returns copies. It used to return the stored dicts themselves, so write tests passed even ifsavenever ran. It also rejects conflicting writes (_rev) the way CouchDB does._ids not in thefm:form_idnever used for matchingDocs and scripts
docs/mcp-servers.md: tool list and requirements.docs/running_benchmark.md: removed theFMSR_MODEL_IDrow, the "WatsonX — needed more often than it looks" section and the related troubleshooting row.docs/opencode-agent.md: removed the generate smoke tests and renumbered the rest.scripts/preflight_run.sh: removed the WatsonX check.Impact on benchmarking
pump,Pumpandhydraulic_pumpgive the same results as before for the bundledpumpsample.Pump-1no longer map topump. They now get a not-found error that suggestspump.generate_failure_modes. Localreports/for 902–916 show answers taken from the catalog.Testing
uv run pytest src/servers/fmsr -qpasses (32 tests).uv run pytest src/ -k "not integration": 19 failures, the same set that fails onmain(iottest_invalid_site,test_static_json_scorer,test_file_exporter). None are in fmsr.ruff format: clean.ruff checkstill reports the file's existingList/Optionaltyping-style warnings.Not in this PR
notebook/kdd_tutorial/03_mcp_fmsr_fixed.ipynbstill referencesgenerate_failure_modes. It is untracked locally.Checklist