Add GitHub Action for JSON reporter

[dcramer]

Similar to some of the existing action reporters, we could just do a quick one off that takes the JSON payload from vitest and outputs in a semi-useful way.

This would let us easily expose Evals in CI in a way that makes them far more debuggable.

[eliatcodecov]

Codecov has this action for reporting test resutls already: https://github.com/codecov/basic-test-results

Which, under the hood, is powered by this: https://github.com/codecov/test-results-parser

We could likely just extend test-results-parser to accept the modified junit output from this project and report it in the action's PR comment.

This biggest drawback here is that basic-test-results is nowhere near as widely used as our cloud Test Analytics offering, but we can incorporate this into Test Analytics after experimenting with basic-test-results since extending that should be really quick.

@dcramer can you provide an example of the junit output from vitest-evals so we can work against it?

[dcramer]

{
    "numTotalTestSuites": 24,
    "numPassedTestSuites": 24,
    "numFailedTestSuites": 0,
    "numPendingTestSuites": 0,
    "numTotalTests": 26,
    "numPassedTests": 26,
    "numFailedTests": 0,
    "numPendingTests": 0,
    "numTodoTests": 0,
    "snapshot": {
        "added": 0,
        "failure": false,
        "filesAdded": 0,
        "filesRemoved": 0,
        "filesRemovedList": [],
        "filesUnmatched": 0,
        "filesUpdated": 0,
        "matched": 0,
        "total": 0,
        "unchecked": 0,
        "uncheckedKeysByFile": [],
        "unmatched": 0,
        "updated": 0,
        "didUpdate": false
    },
    "startTime": 1745854856920,
    "success": true,
    "testResults": [
        {
            "assertionResults": [
                {
                    "ancestorTitles": [
                        "begin-autofix"
                    ],
                    "fullName": "begin-autofix Whats the status on root causing this issue in Sentry?\nhttps://sentry-mcp-evals.sentry.io/issues/PEATED-A8/",
                    "status": "passed",
                    "title": "Whats the status on root causing this issue in Sentry?\nhttps://sentry-mcp-evals.sentry.io/issues/PEATED-A8/",
                    "duration": 26514.328667,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer is a superset of the expert answer and is fully consistent with it. The expert answer identifies the root cause as a batched TRPC request incorrectly passing bottle ID 3216 to `bottleById` instead of 16720, resulting in a 'Bottle not found' error. The submitted answer includes this information and expands on it by providing a detailed root cause analysis, proposed solutions, and solution implementation steps. It identifies the same issue of mismatched IDs and provides additional context and steps to resolve the issue, such as inspecting client-side code, correcting parameter assignments, adding logging, and testing the fix. Therefore, the submitted answer is consistent with the expert answer and provides more comprehensive information."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "begin-autofix"
                    ],
                    "fullName": "begin-autofix Can you root cause this issue and retrieve the analysis?\nhttps://sentry-mcp-evals.sentry.io/issues/PEATED-A8/",
                    "status": "passed",
                    "title": "Can you root cause this issue and retrieve the analysis?\nhttps://sentry-mcp-evals.sentry.io/issues/PEATED-A8/",
                    "duration": 13212.376792,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer is a superset of the expert answer and is fully consistent with it. The expert answer identifies the root cause as a mismatch in the bottle ID passed to the `bottleById` function, which results in a 'Bottle not found' error. The submitted answer includes this same root cause but provides additional details, such as the specific IDs involved (3216 and 16720), and offers a comprehensive proposed solution and implementation strategy. This includes steps to inspect and correct the client-side code, verify parameter mapping, and test the fix. Therefore, the submission expands upon the expert's analysis without contradicting it."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                }
            ],
            "startTime": 1745854857405,
            "endTime": 1745854897133.3767,
            "status": "passed",
            "message": "",
            "name": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/autofix.eval.ts"
        },
        {
            "assertionResults": [
                {
                    "ancestorTitles": [
                        "create-project"
                    ],
                    "fullName": "create-project Create a new SENTRY_DSN for 'sentry-mcp-evals/cloudflare-mcp'",
                    "status": "passed",
                    "title": "Create a new SENTRY_DSN for 'sentry-mcp-evals/cloudflare-mcp'",
                    "duration": 7217.116875,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer includes all the factual content of the expert answer, specifically the SENTRY_DSN URL, which is the key piece of information. Additionally, the submission provides extra context by mentioning the project name and suggesting how the DSN can be used to initialize Sentry's SDKs. This makes the submission a superset of the expert answer, as it contains all the information from the expert answer plus additional relevant details. There is no conflict between the two answers, and the additional information in the submission is consistent with the expert answer."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                }
            ],
            "startTime": 1745854863361,
            "endTime": 1745854870578.117,
            "status": "passed",
            "message": "",
            "name": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/create-dsn.eval.ts"
        },
        {
            "assertionResults": [
                {
                    "ancestorTitles": [
                        "create-project"
                    ],
                    "fullName": "create-project Create a new project in Sentry for 'sentry-mcp-evals' called 'cloudflare-mcp' with the 'the-goats' team. Output **only** the project slug and the SENTRY_DSN in the format of:\n<PROJECT_SLUG>\n<SENTRY_DSN>",
                    "status": "passed",
                    "title": "Create a new project in Sentry for 'sentry-mcp-evals' called 'cloudflare-mcp' with the 'the-goats' team. Output **only** the project slug and the SENTRY_DSN in the format of:\n<PROJECT_SLUG>\n<SENTRY_DSN>",
                    "duration": 8874.870458000001,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 1,
                                    "metadata": {
                                        "rationale": "The submitted answer contains exactly the same details as the expert answer. Both provide the same project slug 'cloudflare-mcp' and the same SENTRY_DSN URL. There are no additional details or discrepancies between the two answers, making the submission factually identical to the expert's response."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 1
                        }
                    }
                }
            ],
            "startTime": 1745854857410,
            "endTime": 1745854866284.8704,
            "status": "passed",
            "message": "",
            "name": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/create-project.eval.ts"
        },
        {
            "assertionResults": [
                {
                    "ancestorTitles": [
                        "create-team"
                    ],
                    "fullName": "create-team Create a new team in Sentry for 'sentry-mcp-evals' called 'the-goats' response with **only** the team slug and no other text.",
                    "status": "passed",
                    "title": "Create a new team in Sentry for 'sentry-mcp-evals' called 'the-goats' response with **only** the team slug and no other text.",
                    "duration": 5640.758209,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 1,
                                    "metadata": {
                                        "rationale": "The submitted answer 'the-goats' is exactly the same as the expert answer 'the-goats'. Both answers contain the same factual content, which is the team slug 'the-goats'. There are no additional details or discrepancies between the two answers, making them identical in terms of factual content."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 1
                        }
                    }
                }
            ],
            "startTime": 1745854857398,
            "endTime": 1745854863038.7583,
            "status": "passed",
            "message": "",
            "name": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/create-team.eval.ts"
        },
        {
            "assertionResults": [
                {
                    "ancestorTitles": [
                        "get-issue"
                    ],
                    "fullName": "get-issue Explain CLOUDFLARE-MCP-41 from Sentry.",
                    "status": "passed",
                    "title": "Explain CLOUDFLARE-MCP-41 from Sentry.",
                    "duration": 11208.615334,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer is a superset of the expert answer and is fully consistent with it. The expert answer provides the error description, issue ID, stack trace, and URL. The submitted answer includes all these details and adds additional information such as the culprit, first seen date, last seen date, occurrences, users impacted, status, platform, and project. All the additional information in the submission is consistent with the expert answer, making it a superset."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                }
            ],
            "startTime": 1745854857399,
            "endTime": 1745854868607.6152,
            "status": "passed",
            "message": "",
            "name": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/get-issue.eval.ts"
        },
        {
            "assertionResults": [
                {
                    "ancestorTitles": [
                        "list-dsns"
                    ],
                    "fullName": "list-dsns I need a SENTRY_DSN for 'sentry-mcp-evals/cloudflare-mcp'",
                    "status": "passed",
                    "title": "I need a SENTRY_DSN for 'sentry-mcp-evals/cloudflare-mcp'",
                    "duration": 10773.89825,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer includes the exact same SENTRY_DSN as the expert answer, which means it contains all the factual content of the expert answer. Additionally, the submission provides extra context by specifying the project and organization names ('cloudflare-mcp' and 'sentry-mcp-evals') and explains the use of the DSN for initializing Sentry's SDKs. Therefore, the submitted answer is a superset of the expert answer and is fully consistent with it."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                }
            ],
            "startTime": 1745854863474,
            "endTime": 1745854874247.8982,
            "status": "passed",
            "message": "",
            "name": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/list-dsns.eval.ts"
        },
        {
            "assertionResults": [
                {
                    "ancestorTitles": [
                        "list-issues"
                    ],
                    "fullName": "list-issues Can you you give me a list of common production errors, with their stacktrace and a url for more information?",
                    "status": "passed",
                    "title": "Can you you give me a list of common production errors, with their stacktrace and a url for more information?",
                    "duration": 17777.898583,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer is a superset of the expert answer. It includes all the details provided in the expert answer for the issue CLOUDFLARE-MCP-41, such as the error description, stacktrace, and URL. Additionally, the submission includes another issue, CLOUDFLARE-MCP-42, which is not present in the expert answer. This makes the submission a superset, as it contains more information while being fully consistent with the expert's details."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "list-issues"
                    ],
                    "fullName": "list-issues Give me a summary of my top issues in sentry-mcp-evals",
                    "status": "passed",
                    "title": "Give me a summary of my top issues in sentry-mcp-evals",
                    "duration": 12911.958625,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer is a superset of the expert answer and is fully consistent with it. The expert answer provides a brief summary of the top issues by listing their IDs and URLs. The submitted answer includes all the information from the expert answer (the issue IDs and URLs) and adds additional details such as descriptions, culprits, first seen and last seen dates, and a note on how to reference the issue IDs in commit messages. Therefore, the submission contains all the information from the expert answer plus more, making it a superset."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "list-issues"
                    ],
                    "fullName": "list-issues Find the most recent production issues in sentry-mcp-evals",
                    "status": "passed",
                    "title": "Find the most recent production issues in sentry-mcp-evals",
                    "duration": 6790.161375,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer is a superset of the expert answer and is fully consistent with it. The expert answer provides the issue IDs and URLs for two issues, CLOUDFLARE-MCP-41 and CLOUDFLARE-MCP-42. The submitted answer includes all of this information and adds additional details such as descriptions, culprits, first seen and last seen dates for each issue. These additional details do not contradict the expert answer but rather expand upon it, making the submission a superset of the expert answer."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "list-issues"
                    ],
                    "fullName": "list-issues Find the newest production issues in sentry-mcp-evals",
                    "status": "passed",
                    "title": "Find the newest production issues in sentry-mcp-evals",
                    "duration": 9234.857790999995,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer is a superset of the expert answer and is fully consistent with it. The expert answer lists two issues, CLOUDFLARE-MCP-42 and CLOUDFLARE-MCP-41, with their URLs. The submitted answer includes these same issues with additional details such as descriptions, culprits, first seen and last seen dates, and formatted URLs. All the information in the expert answer is present in the submission, along with extra details that do not contradict the expert answer."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "list-issues"
                    ],
                    "fullName": "list-issues What issues are affecting david@sentry.io in sentry-mcp-evals?",
                    "status": "passed",
                    "title": "What issues are affecting david@sentry.io in sentry-mcp-evals?",
                    "duration": 10944.217749999996,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer is a superset of the expert answer. It includes all the information provided by the expert, such as the issue ID and URL, but also adds additional details like the description, culprit, first seen date, and last seen date. These additional details do not contradict the expert answer; they simply provide more context and information about the issue. Therefore, the submitted answer is consistent with the expert answer and extends it with more information."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "list-issues"
                    ],
                    "fullName": "list-issues How many issues are affecting david@sentry.io in sentry-mcp-evals?",
                    "status": "passed",
                    "title": "How many issues are affecting david@sentry.io in sentry-mcp-evals?",
                    "duration": 14658.441250000003,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The expert answer states that there is 1 issue affecting david@sentry.io, which is consistent with the submitted answer. The submitted answer provides additional details about the issue, such as the issue ID, description, culprit, first seen date, last seen date, and a URL to view the issue. These additional details make the submitted answer a superset of the expert answer, but they do not contradict the expert's statement. Therefore, the submitted answer is a superset of the expert answer and is fully consistent with it."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "list-issues"
                    ],
                    "fullName": "list-issues How many issues are in sentry-mcp-evals/cloudflare-mcp?",
                    "status": "passed",
                    "title": "How many issues are in sentry-mcp-evals/cloudflare-mcp?",
                    "duration": 9013.777917,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer provides more detailed information than the expert answer, including specific issue identifiers (CLOUDFLARE-MCP-42 and CLOUDFLARE-MCP-41), descriptions, culprits, first and last seen dates, and links to view the issues. However, the core factual content, which is the number of issues (2), is consistent with the expert answer. Therefore, the submitted answer is a superset of the expert answer and is fully consistent with it."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "list-issues"
                    ],
                    "fullName": "list-issues What issues are affecting jane@sentry.io in sentry-mcp-evals?",
                    "status": "passed",
                    "title": "What issues are affecting jane@sentry.io in sentry-mcp-evals?",
                    "duration": 5530.544207999992,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 1,
                                    "metadata": {
                                        "rationale": "The submitted answer and the expert answer both convey the same factual content: that there are no issues affecting the user with the email jane@sentry.io in the sentry-mcp-evals organization. The submission provides the same information as the expert answer, just with slightly more detail in terms of specifying the organization and user email, but these details do not change the factual content. Therefore, the submitted answer contains all the same details as the expert answer."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 1
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "list-issues"
                    ],
                    "fullName": "list-issues How many issues are affecting jane@sentry.io in sentry-mcp-evals?",
                    "status": "passed",
                    "title": "How many issues are affecting jane@sentry.io in sentry-mcp-evals?",
                    "duration": 5302.750832999998,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 1,
                                    "metadata": {
                                        "rationale": "The expert answer states that there are 0 issues affecting jane@sentry.io in sentry-mcp-evals. The submitted answer conveys the same information by stating that there are no issues affecting jane@sentry.io in the 'sentry-mcp-evals' organization. Both answers communicate the same factual content, which is that there are no issues. Therefore, the submitted answer contains all the same details as the expert answer."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 1
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "list-issues"
                    ],
                    "fullName": "list-issues How many issues are in sentry-mcp-evals/foobar?",
                    "status": "passed",
                    "title": "How many issues are in sentry-mcp-evals/foobar?",
                    "duration": 4277.179375000007,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 1,
                                    "metadata": {
                                        "rationale": "The submitted answer states that there are no issues in the project 'sentry-mcp-evals/foobar,' which is factually the same as the expert answer indicating '0' issues. Both answers convey the same factual information, just in different wording. Therefore, the submitted answer contains all the same details as the expert answer."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 1
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "list-issues"
                    ],
                    "fullName": "list-issues How many issues are in project:remote-mcp?",
                    "status": "passed",
                    "title": "How many issues are in project:remote-mcp?",
                    "duration": 5900.013124999998,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The expert answer states 'Invalid project,' which implies that the project 'remote-mcp' does not exist or is not recognized. The submitted answer provides additional context by specifying that there is no project named 'remote-mcp' in the 'sentry-mcp-evals' organization and suggests checking the project name. This additional information does not contradict the expert answer but rather expands on it, making the submitted answer a superset of the expert answer while being fully consistent with it."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                }
            ],
            "startTime": 1745854857403,
            "endTime": 1745854959751.0132,
            "status": "passed",
            "message": "",
            "name": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/list-issues.eval.ts"
        },
        {
            "assertionResults": [
                {
                    "ancestorTitles": [
                        "list-organizations"
                    ],
                    "fullName": "list-organizations What organizations do I have access to in Sentry",
                    "status": "passed",
                    "title": "What organizations do I have access to in Sentry",
                    "duration": 3403.557042,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 1,
                                    "metadata": {
                                        "rationale": "The submitted answer contains the exact same organization name 'sentry-mcp-evals' as the expert answer. There are no additional organizations listed in the submission, nor are there any missing from the expert answer. Therefore, the submitted answer contains all the same details as the expert answer."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 1
                        }
                    }
                }
            ],
            "startTime": 1745854866594,
            "endTime": 1745854869997.5571,
            "status": "passed",
            "message": "",
            "name": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/list-organizations.eval.ts"
        },
        {
            "assertionResults": [
                {
                    "ancestorTitles": [
                        "list-projects"
                    ],
                    "fullName": "list-projects What projects do I have access to in Sentry for 'sentry-mcp-evals'",
                    "status": "passed",
                    "title": "What projects do I have access to in Sentry for 'sentry-mcp-evals'",
                    "duration": 5790.189833,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 1,
                                    "metadata": {
                                        "rationale": "The submitted answer contains the same factual information as the expert answer. Both indicate that the project 'cloudflare-mcp' is accessible in the 'sentry-mcp-evals' organization. There are no additional projects listed in the submission, nor are there any discrepancies between the two answers. Therefore, the submitted answer contains all the same details as the expert answer."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 1
                        }
                    }
                }
            ],
            "startTime": 1745854857400,
            "endTime": 1745854863190.19,
            "status": "passed",
            "message": "",
            "name": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/list-projects.eval.ts"
        },
        {
            "assertionResults": [
                {
                    "ancestorTitles": [
                        "list-releases"
                    ],
                    "fullName": "list-releases Show me the releases in sentry-mcp-evals",
                    "status": "passed",
                    "title": "Show me the releases in sentry-mcp-evals",
                    "duration": 15586.393833,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer is a superset of the expert answer. The expert answer only provides the release ID '8ce89484-0fec-4913-a2cd-e8e2d41dee36', while the submitted answer includes this release ID along with additional details such as the creation date, first and last event times, new issues count, and associated projects. All the information in the submitted answer is consistent with the expert answer, as it includes the same release ID and expands upon it with more details."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "list-releases"
                    ],
                    "fullName": "list-releases Show me a list of versions in sentry-mcp-evals/cloudflare-mcp",
                    "status": "passed",
                    "title": "Show me a list of versions in sentry-mcp-evals/cloudflare-mcp",
                    "duration": 9399.200416999998,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer provides additional details about the release version 8ce89484-0fec-4913-a2cd-e8e2d41dee36, such as the creation date, first and last event dates, new issues count, and associated projects. These details are not present in the expert answer, which only lists the version identifier. Therefore, the submitted answer is a superset of the expert answer and is fully consistent with it, as it includes all the information from the expert answer plus more."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                }
            ],
            "startTime": 1745854857409,
            "endTime": 1745854882395.2004,
            "status": "passed",
            "message": "",
            "name": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/list-releases.eval.ts"
        },
        {
            "assertionResults": [
                {
                    "ancestorTitles": [
                        "list-tags"
                    ],
                    "fullName": "list-tags What are common tags in sentry-mcp-evals",
                    "status": "passed",
                    "title": "What are common tags in sentry-mcp-evals",
                    "duration": 12655.78,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 1,
                                    "metadata": {
                                        "rationale": "The submitted answer contains all the same tags as the expert answer. Both lists of tags are identical, meaning the submission includes every tag mentioned by the expert. There are no additional tags or missing tags in the submission compared to the expert answer. Therefore, the submitted answer contains all the same details as the expert answer."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 1
                        }
                    }
                }
            ],
            "startTime": 1745854857399,
            "endTime": 1745854870054.78,
            "status": "passed",
            "message": "",
            "name": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/list-tags.eval.ts"
        },
        {
            "assertionResults": [
                {
                    "ancestorTitles": [
                        "list-teams"
                    ],
                    "fullName": "list-teams What teams do I have access to in Sentry for 'sentry-mcp-evals'",
                    "status": "passed",
                    "title": "What teams do I have access to in Sentry for 'sentry-mcp-evals'",
                    "duration": 6005.649875,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 1,
                                    "metadata": {
                                        "rationale": "The submitted answer states that the user has access to the team 'the-goats' in the 'sentry-mcp-evals' organization, which is exactly the same information provided by the expert answer. Both answers contain the same factual content, with no additional or missing details. Therefore, the submitted answer contains all the same details as the expert answer."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 1
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "list-teams"
                    ],
                    "fullName": "list-teams Do I have access to the team 'the-goats' for 'sentry-mcp-evals'",
                    "status": "passed",
                    "title": "Do I have access to the team 'the-goats' for 'sentry-mcp-evals'",
                    "duration": 4921.93875,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The submitted answer is a superset of the expert answer. The expert answer simply states 'Yes,' indicating access to the team 'the-goats' for 'sentry-mcp-evals.' The submitted answer provides additional context by specifying that the access is to the team 'the-goats' in the organization 'sentry-mcp-evals.' This additional information does not contradict the expert answer but rather expands on it, making the submitted answer a superset that is fully consistent with the expert answer."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                },
                {
                    "ancestorTitles": [
                        "list-teams"
                    ],
                    "fullName": "list-teams Do I have access to the team 'an-imaginary-team' for 'sentry-mcp-evals'",
                    "status": "passed",
                    "title": "Do I have access to the team 'an-imaginary-team' for 'sentry-mcp-evals'",
                    "duration": 4965.565791999999,
                    "failureMessages": [],
                    "meta": {
                        "eval": {
                            "scores": [
                                {
                                    "score": 0.6,
                                    "metadata": {
                                        "rationale": "The expert answer simply states 'No,' indicating that there is no access to the team 'an-imaginary-team' for 'sentry-mcp-evals.' The submitted answer provides additional information by stating that the user does not have access to 'an-imaginary-team' and specifies that the only available team is 'the-goats.' This additional detail does not contradict the expert answer but rather expands on it, making the submitted answer a superset of the expert answer while remaining fully consistent with it."
                                    },
                                    "name": "Factuality2"
                                }
                            ],
                            "avgScore": 0.6
                        }
                    }
                }
            ],
            "startTime": 1745854857405,
            "endTime": 1745854873298.5657,
            "status": "passed",
            "message": "",
            "name": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/list-teams.eval.ts"
        }
    ],
    "coverageMap": {
        "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/utils.ts": {
            "path": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/evals/utils.ts",
            "all": false,
            "statementMap": {
                "0": {
                    "start": {
                        "line": 1,
                        "column": 0
                    },
                    "end": {
                        "line": 1,
                        "column": 40
                    }
                },
                "1": {
                    "start": {
                        "line": 2,
                        "column": 0
                    },
                    "end": {
                        "line": 2,
                        "column": 8
                    }
                },
                "7": {
                    "start": {
                        "line": 8,
                        "column": 0
                    },
                    "end": {
                        "line": 8,
                        "column": 62
                    }
                },
                "8": {
                    "start": {
                        "line": 9,
                        "column": 0
                    },
                    "end": {
                        "line": 9,
                        "column": 24
                    }
                },
                "10": {
                    "start": {
                        "line": 11,
                        "column": 0
                    },
                    "end": {
                        "line": 11,
                        "column": 25
                    }
                },
                "11": {
                    "start": {
                        "line": 12,
                        "column": 0
                    },
                    "end": {
                        "line": 12,
                        "column": 39
                    }
                },
                "12": {
                    "start": {
                        "line": 13,
                        "column": 0
                    },
                    "end": {
                        "line": 13,
                        "column": 24
                    }
                },
                "13": {
                    "start": {
                        "line": 14,
                        "column": 0
                    },
                    "end": {
                        "line": 14,
                        "column": 32
                    }
                },
                "14": {
                    "start": {
                        "line": 15,
                        "column": 0
                    },
                    "end": {
                        "line": 15,
                        "column": 31
                    }
                },
                "15": {
                    "start": {
                        "line": 16,
                        "column": 0
                    },
                    "end": {
                        "line": 16,
                        "column": 75
                    }
                },
                "16": {
                    "start": {
                        "line": 17,
                        "column": 0
                    },
                    "end": {
                        "line": 17,
                        "column": 74
                    }
                },
                "17": {
                    "start": {
                        "line": 18,
                        "column": 0
                    },
                    "end": {
                        "line": 18,
                        "column": 105
                    }
                },
                "18": {
                    "start": {
                        "line": 19,
                        "column": 0
                    },
                    "end": {
                        "line": 19,
                        "column": 2
                    }
                },
                "20": {
                    "start": {
                        "line": 21,
                        "column": 0
                    },
                    "end": {
                        "line": 21,
                        "column": 38
                    }
                },
                "22": {
                    "start": {
                        "line": 23,
                        "column": 0
                    },
                    "end": {
                        "line": 23,
                        "column": 65
                    }
                },
                "23": {
                    "start": {
                        "line": 24,
                        "column": 0
                    },
                    "end": {
                        "line": 24,
                        "column": 51
                    }
                },
                "24": {
                    "start": {
                        "line": 25,
                        "column": 0
                    },
                    "end": {
                        "line": 25,
                        "column": 58
                    }
                },
                "25": {
                    "start": {
                        "line": 26,
                        "column": 0
                    },
                    "end": {
                        "line": 26,
                        "column": 21
                    }
                },
                "26": {
                    "start": {
                        "line": 27,
                        "column": 0
                    },
                    "end": {
                        "line": 27,
                        "column": 29
                    }
                },
                "27": {
                    "start": {
                        "line": 28,
                        "column": 0
                    },
                    "end": {
                        "line": 28,
                        "column": 12
                    }
                },
                "28": {
                    "start": {
                        "line": 29,
                        "column": 0
                    },
                    "end": {
                        "line": 29,
                        "column": 58
                    }
                },
                "29": {
                    "start": {
                        "line": 30,
                        "column": 0
                    },
                    "end": {
                        "line": 30,
                        "column": 8
                    }
                },
                "30": {
                    "start": {
                        "line": 31,
                        "column": 0
                    },
                    "end": {
                        "line": 31,
                        "column": 7
                    }
                },
                "31": {
                    "start": {
                        "line": 32,
                        "column": 0
                    },
                    "end": {
                        "line": 32,
                        "column": 58
                    }
                },
                "32": {
                    "start": {
                        "line": 33,
                        "column": 0
                    },
                    "end": {
                        "line": 33,
                        "column": 16
                    }
                },
                "33": {
                    "start": {
                        "line": 34,
                        "column": 0
                    },
                    "end": {
                        "line": 34,
                        "column": 7
                    }
                },
                "35": {
                    "start": {
                        "line": 36,
                        "column": 0
                    },
                    "end": {
                        "line": 36,
                        "column": 42
                    }
                },
                "37": {
                    "start": {
                        "line": 38,
                        "column": 0
                    },
                    "end": {
                        "line": 38,
                        "column": 9
                    }
                },
                "38": {
                    "start": {
                        "line": 39,
                        "column": 0
                    },
                    "end": {
                        "line": 39,
                        "column": 33
                    }
                },
                "39": {
                    "start": {
                        "line": 40,
                        "column": 0
                    },
                    "end": {
                        "line": 40,
                        "column": 14
                    }
                },
                "40": {
                    "start": {
                        "line": 41,
                        "column": 0
                    },
                    "end": {
                        "line": 41,
                        "column": 14
                    }
                },
                "41": {
                    "start": {
                        "line": 42,
                        "column": 0
                    },
                    "end": {
                        "line": 42,
                        "column": 15
                    }
                },
                "42": {
                    "start": {
                        "line": 43,
                        "column": 0
                    },
                    "end": {
                        "line": 43,
                        "column": 177
                    }
                },
                "43": {
                    "start": {
                        "line": 44,
                        "column": 0
                    },
                    "end": {
                        "line": 44,
                        "column": 22
                    }
                },
                "44": {
                    "start": {
                        "line": 45,
                        "column": 0
                    },
                    "end": {
                        "line": 45,
                        "column": 22
                    }
                },
                "45": {
                    "start": {
                        "line": 46,
                        "column": 0
                    },
                    "end": {
                        "line": 46,
                        "column": 21
                    }
                },
                "46": {
                    "start": {
                        "line": 47,
                        "column": 0
                    },
                    "end": {
                        "line": 47,
                        "column": 33
                    }
                },
                "47": {
                    "start": {
                        "line": 48,
                        "column": 0
                    },
                    "end": {
                        "line": 48,
                        "column": 26
                    }
                },
                "48": {
                    "start": {
                        "line": 49,
                        "column": 0
                    },
                    "end": {
                        "line": 49,
                        "column": 10
                    }
                },
                "49": {
                    "start": {
                        "line": 50,
                        "column": 0
                    },
                    "end": {
                        "line": 50,
                        "column": 29
                    }
                },
                "50": {
                    "start": {
                        "line": 51,
                        "column": 0
                    },
                    "end": {
                        "line": 51,
                        "column": 31
                    }
                },
                "51": {
                    "start": {
                        "line": 52,
                        "column": 0
                    },
                    "end": {
                        "line": 52,
                        "column": 10
                    }
                },
                "52": {
                    "start": {
                        "line": 53,
                        "column": 0
                    },
                    "end": {
                        "line": 53,
                        "column": 9
                    }
                },
                "54": {
                    "start": {
                        "line": 55,
                        "column": 0
                    },
                    "end": {
                        "line": 55,
                        "column": 48
                    }
                },
                "55": {
                    "start": {
                        "line": 56,
                        "column": 0
                    },
                    "end": {
                        "line": 56,
                        "column": 7
                    }
                },
                "57": {
                    "start": {
                        "line": 58,
                        "column": 0
                    },
                    "end": {
                        "line": 58,
                        "column": 31
                    }
                },
                "58": {
                    "start": {
                        "line": 59,
                        "column": 0
                    },
                    "end": {
                        "line": 59,
                        "column": 21
                    }
                },
                "59": {
                    "start": {
                        "line": 60,
                        "column": 0
                    },
                    "end": {
                        "line": 60,
                        "column": 27
                    }
                },
                "60": {
                    "start": {
                        "line": 61,
                        "column": 0
                    },
                    "end": {
                        "line": 61,
                        "column": 18
                    }
                },
                "61": {
                    "start": {
                        "line": 62,
                        "column": 0
                    },
                    "end": {
                        "line": 62,
                        "column": 15
                    }
                },
                "62": {
                    "start": {
                        "line": 63,
                        "column": 0
                    },
                    "end": {
                        "line": 63,
                        "column": 30
                    }
                },
                "63": {
                    "start": {
                        "line": 64,
                        "column": 0
                    },
                    "end": {
                        "line": 64,
                        "column": 5
                    }
                },
                "64": {
                    "start": {
                        "line": 65,
                        "column": 0
                    },
                    "end": {
                        "line": 65,
                        "column": 4
                    }
                },
                "65": {
                    "start": {
                        "line": 66,
                        "column": 0
                    },
                    "end": {
                        "line": 66,
                        "column": 1
                    }
                },
                "76": {
                    "start": {
                        "line": 77,
                        "column": 0
                    },
                    "end": {
                        "line": 77,
                        "column": 65
                    }
                },
                "77": {
                    "start": {
                        "line": 78,
                        "column": 0
                    },
                    "end": {
                        "line": 78,
                        "column": 42
                    }
                },
                "81": {
                    "start": {
                        "line": 82,
                        "column": 0
                    },
                    "end": {
                        "line": 82,
                        "column": 6
                    }
                },
                "82": {
                    "start": {
                        "line": 83,
                        "column": 0
                    },
                    "end": {
                        "line": 83,
                        "column": 45
                    }
                },
                "83": {
                    "start": {
                        "line": 84,
                        "column": 0
                    },
                    "end": {
                        "line": 84,
                        "column": 12
                    }
                },
                "89": {
                    "start": {
                        "line": 90,
                        "column": 0
                    },
                    "end": {
                        "line": 90,
                        "column": 15
                    }
                },
                "94": {
                    "start": {
                        "line": 95,
                        "column": 0
                    },
                    "end": {
                        "line": 95,
                        "column": 33
                    }
                },
                "96": {
                    "start": {
                        "line": 97,
                        "column": 0
                    },
                    "end": {
                        "line": 97,
                        "column": 34
                    }
                },
                "98": {
                    "start": {
                        "line": 99,
                        "column": 0
                    },
                    "end": {
                        "line": 99,
                        "column": 36
                    }
                },
                "112": {
                    "start": {
                        "line": 113,
                        "column": 0
                    },
                    "end": {
                        "line": 113,
                        "column": 24
                    }
                },
                "113": {
                    "start": {
                        "line": 114,
                        "column": 0
                    },
                    "end": {
                        "line": 114,
                        "column": 78
                    }
                },
                "114": {
                    "start": {
                        "line": 115,
                        "column": 0
                    },
                    "end": {
                        "line": 115,
                        "column": 20
                    }
                },
                "115": {
                    "start": {
                        "line": 116,
                        "column": 0
                    },
                    "end": {
                        "line": 116,
                        "column": 19
                    }
                },
                "116": {
                    "start": {
                        "line": 117,
                        "column": 0
                    },
                    "end": {
                        "line": 117,
                        "column": 68
                    }
                },
                "117": {
                    "start": {
                        "line": 118,
                        "column": 0
                    },
                    "end": {
                        "line": 118,
                        "column": 9
                    }
                },
                "118": {
                    "start": {
                        "line": 119,
                        "column": 0
                    },
                    "end": {
                        "line": 119,
                        "column": 7
                    }
                },
                "120": {
                    "start": {
                        "line": 121,
                        "column": 0
                    },
                    "end": {
                        "line": 121,
                        "column": 20
                    }
                },
                "121": {
                    "start": {
                        "line": 122,
                        "column": 0
                    },
                    "end": {
                        "line": 122,
                        "column": 13
                    }
                },
                "122": {
                    "start": {
                        "line": 123,
                        "column": 0
                    },
                    "end": {
                        "line": 123,
                        "column": 13
                    }
                },
                "123": {
                    "start": {
                        "line": 124,
                        "column": 0
                    },
                    "end": {
                        "line": 124,
                        "column": 11
                    }
                },
                "124": {
                    "start": {
                        "line": 125,
                        "column": 0
                    },
                    "end": {
                        "line": 125,
                        "column": 11
                    }
                },
                "125": {
                    "start": {
                        "line": 126,
                        "column": 0
                    },
                    "end": {
                        "line": 126,
                        "column": 11
                    }
                },
                "126": {
                    "start": {
                        "line": 127,
                        "column": 0
                    },
                    "end": {
                        "line": 127,
                        "column": 6
                    }
                },
                "128": {
                    "start": {
                        "line": 129,
                        "column": 0
                    },
                    "end": {
                        "line": 129,
                        "column": 12
                    }
                },
                "129": {
                    "start": {
                        "line": 130,
                        "column": 0
                    },
                    "end": {
                        "line": 130,
                        "column": 35
                    }
                },
                "130": {
                    "start": {
                        "line": 131,
                        "column": 0
                    },
                    "end": {
                        "line": 131,
                        "column": 17
                    }
                },
                "131": {
                    "start": {
                        "line": 132,
                        "column": 0
                    },
                    "end": {
                        "line": 132,
                        "column": 36
                    }
                },
                "132": {
                    "start": {
                        "line": 133,
                        "column": 0
                    },
                    "end": {
                        "line": 133,
                        "column": 8
                    }
                },
                "133": {
                    "start": {
                        "line": 134,
                        "column": 0
                    },
                    "end": {
                        "line": 134,
                        "column": 6
                    }
                },
                "134": {
                    "start": {
                        "line": 135,
                        "column": 0
                    },
                    "end": {
                        "line": 135,
                        "column": 4
                    }
                },
                "135": {
                    "start": {
                        "line": 136,
                        "column": 0
                    },
                    "end": {
                        "line": 136,
                        "column": 1
                    }
                }
            },
            "s": {
                "0": 1,
                "1": 1,
                "7": 1,
                "8": 1,
                "10": 1,
                "11": 1,
                "12": 1,
                "13": 1,
                "14": 1,
                "15": 1,
                "16": 1,
                "17": 1,
                "18": 1,
                "20": 1,
                "22": 1,
                "23": 12,
                "24": 26,
                "25": 26,
                "26": 26,
                "27": 26,
                "28": 26,
                "29": 26,
                "30": 26,
                "31": 26,
                "32": 26,
                "33": 26,
                "35": 26,
                "37": 26,
                "38": 26,
                "39": 26,
                "40": 26,
                "41": 26,
                "42": 26,
                "43": 26,
                "44": 26,
                "45": 26,
                "46": 26,
                "47": 26,
                "48": 26,
                "49": 26,
                "50": 0,
                "51": 0,
                "52": 26,
                "54": 26,
                "55": 3946,
                "57": 26,
                "58": 26,
                "59": 0,
                "60": 0,
                "61": 26,
                "62": 26,
                "63": 26,
                "64": 26,
                "65": 12,
                "76": 1,
                "77": 12,
                "81": 26,
                "82": 26,
                "83": 26,
                "89": 26,
                "94": 26,
                "96": 26,
                "98": 26,
                "112": 26,
                "113": 26,
                "114": 26,
                "115": 26,
                "116": 26,
                "117": 26,
                "118": 26,
                "120": 26,
                "121": 26,
                "122": 26,
                "123": 26,
                "124": 26,
                "125": 26,
                "126": 26,
                "128": 26,
                "129": 26,
                "130": 26,
                "131": 26,
                "132": 26,
                "133": 26,
                "134": 26,
                "135": 12
            },
            "branchMap": {
                "0": {
                    "type": "branch",
                    "line": 23,
                    "loc": {
                        "start": {
                            "line": 23,
                            "column": 7
                        },
                        "end": {
                            "line": 66,
                            "column": 1
                        }
                    },
                    "locations": [
                        {
                            "start": {
                                "line": 23,
                                "column": 7
                            },
                            "end": {
                                "line": 66,
                                "column": 1
                            }
                        }
                    ]
                },
                "1": {
                    "type": "branch",
                    "line": 24,
                    "loc": {
                        "start": {
                            "line": 24,
                            "column": 9
                        },
                        "end": {
                            "line": 65,
                            "column": 4
                        }
                    },
                    "locations": [
                        {
                            "start": {
                                "line": 24,
                                "column": 9
                            },
                            "end": {
                                "line": 65,
                                "column": 4
                            }
                        }
                    ]
                },
                "2": {
                    "type": "branch",
                    "line": 55,
                    "loc": {
                        "start": {
                            "line": 55,
                            "column": 47
                        },
                        "end": {
                            "line": 56,
                            "column": 7
                        }
                    },
                    "locations": [
                        {
                            "start": {
                                "line": 55,
                                "column": 47
                            },
                            "end": {
                                "line": 56,
                                "column": 7
                            }
                        }
                    ]
                },
                "3": {
                    "type": "branch",
                    "line": 59,
                    "loc": {
                        "start": {
                            "line": 59,
                            "column": 4
                        },
                        "end": {
                            "line": 62,
                            "column": 6
                        }
                    },
                    "locations": [
                        {
                            "start": {
                                "line": 59,
                                "column": 4
                            },
                            "end": {
                                "line": 62,
                                "column": 6
                            }
                        }
                    ]
                },
                "4": {
                    "type": "branch",
                    "line": 77,
                    "loc": {
                        "start": {
                            "line": 77,
                            "column": 7
                        },
                        "end": {
                            "line": 136,
                            "column": 1
                        }
                    },
                    "locations": [
                        {
                            "start": {
                                "line": 77,
                                "column": 7
                            },
                            "end": {
                                "line": 136,
                                "column": 1
                            }
                        }
                    ]
                },
                "5": {
                    "type": "branch",
                    "line": 78,
                    "loc": {
                        "start": {
                            "line": 78,
                            "column": 9
                        },
                        "end": {
                            "line": 135,
                            "column": 4
                        }
                    },
                    "locations": [
                        {
                            "start": {
                                "line": 78,
                                "column": 9
                            },
                            "end": {
                                "line": 135,
                                "column": 4
                            }
                        }
                    ]
                }
            },
            "b": {
                "0": [
                    12
                ],
                "1": [
                    26
                ],
                "2": [
                    3946
                ],
                "3": [
                    0
                ],
                "4": [
                    12
                ],
                "5": [
                    26
                ]
            },
            "fnMap": {
                "0": {
                    "name": "TaskRunner",
                    "decl": {
                        "start": {
                            "line": 23,
                            "column": 7
                        },
                        "end": {
                            "line": 66,
                            "column": 1
                        }
                    },
                    "loc": {
                        "start": {
                            "line": 23,
                            "column": 7
                        },
                        "end": {
                            "line": 66,
                            "column": 1
                        }
                    },
                    "line": 23
                },
                "1": {
                    "name": "TaskRunner2",
                    "decl": {
                        "start": {
                            "line": 24,
                            "column": 9
                        },
                        "end": {
                            "line": 65,
                            "column": 4
                        }
                    },
                    "loc": {
                        "start": {
                            "line": 24,
                            "column": 9
                        },
                        "end": {
                            "line": 65,
                            "column": 4
                        }
                    },
                    "line": 24
                },
                "2": {
                    "name": "onError",
                    "decl": {
                        "start": {
                            "line": 50,
                            "column": 17
                        },
                        "end": {
                            "line": 52,
                            "column": 10
                        }
                    },
                    "loc": {
                        "start": {
                            "line": 50,
                            "column": 17
                        },
                        "end": {
                            "line": 52,
                            "column": 10
                        }
                    },
                    "line": 50
                },
                "3": {
                    "name": "Factuality",
                    "decl": {
                        "start": {
                            "line": 77,
                            "column": 7
                        },
                        "end": {
                            "line": 136,
                            "column": 1
                        }
                    },
                    "loc": {
                        "start": {
                            "line": 77,
                            "column": 7
                        },
                        "end": {
                            "line": 136,
                            "column": 1
                        }
                    },
                    "line": 77
                },
                "4": {
                    "name": "Factuality2",
                    "decl": {
                        "start": {
                            "line": 78,
                            "column": 9
                        },
                        "end": {
                            "line": 135,
                            "column": 4
                        }
                    },
                    "loc": {
                        "start": {
                            "line": 78,
                            "column": 9
                        },
                        "end": {
                            "line": 135,
                            "column": 4
                        }
                    },
                    "line": 78
                }
            },
            "f": {
                "0": 12,
                "1": 26,
                "2": 0,
                "3": 12,
                "4": 26
            }
        },
        "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/bin/start-mock-stdio.ts": {
            "path": "/Users/dcramer/src/sentry-mcp/packages/mcp-server-evals/src/bin/start-mock-stdio.ts",
            "all": true,
            "statementMap": {
                "2": {
                    "start": {
                        "line": 3,
                        "column": 0
                    },
                    "end": {
                        "line": 3,
                        "column": 68
                    }
                },
                "3": {
                    "start": {
                        "line": 4,
                        "column": 0
                    },
                    "end": {
                        "line": 4,
                        "column": 65
                    }
                },
                "4": {
                    "start": {
                        "line": 5,
                        "column": 0
                    },
                    "end": {
                        "line": 5,
                        "column": 53
                    }
                },
                "6": {
                    "start": {
                        "line": 7,
                        "column": 0
                    },
                    "end": {
                        "line": 7,
                        "column": 18
                    }
                },
                "7": {
                    "start": {
                        "line": 8,
                        "column": 0
                    },
                    "end": {
                        "line": 8,
                        "column": 39
                    }
                },
                "8": {
                    "start": {
                        "line": 9,
                        "column": 0
                    },
                    "end": {
                        "line": 9,
                        "column": 56
                    }
                },
                "9": {
                    "start": {
                        "line": 10,
                        "column": 0
                    },
                    "end": {
                        "line": 10,
                        "column": 13
                    }
                },
                "10": {
                    "start": {
                        "line": 11,
                        "column": 0
                    },
                    "end": {
                        "line": 11,
                        "column": 5
                    }
                },
                "12": {
                    "start": {
                        "line": 13,
                        "column": 0
                    },
                    "end": {
                        "line": 13,
                        "column": 20
                    }
                },
                "13": {
                    "start": {
                        "line": 14,
                        "column": 0
                    },
                    "end": {
                        "line": 14,
                        "column": 41
                    }
                },
                "14": {
                    "start": {
                        "line": 15,
                        "column": 0
                    },
                    "end": {
                        "line": 15,
                        "column": 4
                    }
                },
                "16": {
                    "start": {
                        "line": 17,
                        "column": 0
                    },
                    "end": {
                        "line": 17,
                        "column": 3
                    }
                },
                "18": {
                    "start": {
                        "line": 19,
                        "column": 0
                    },
                    "end": {
                        "line": 19,
                        "column": 42
                    }
                },
                "20": {
                    "start": {
                        "line": 21,
                        "column": 0
                    },
                    "end": {
                        "line": 21,
                        "column": 30
                    }
                },
                "21": {
                    "start": {
                        "line": 22,
                        "column": 0
                    },
                    "end": {
                        "line": 22,
                        "column": 21
                    }
                },
                "22": {
                    "start": {
                        "line": 23,
                        "column": 0
                    },
                    "end": {
                        "line": 23,
                        "column": 19
                    }
                },
                "23": {
                    "start": {
                        "line": 24,
                        "column": 0
                    },
                    "end": {
                        "line": 24,
                        "column": 3
                    }
                },
                "27": {
                    "start": {
                        "line": 28,
                        "column": 0
                    },
                    "end": {
                        "line": 28,
                        "column": 20
                    }
                },
                "28": {
                    "start": {
                        "line": 29,
                        "column": 0
                    },
                    "end": {
                        "line": 29,
                        "column": 14
                    }
                },
                "29": {
                    "start": {
                        "line": 30,
                        "column": 0
                    },
                    "end": {
                        "line": 30,
                        "column": 25
                    }
                },
                "30": {
                    "start": {
                        "line": 31,
                        "column": 0
                    },
                    "end": {
                        "line": 31,
                        "column": 28
                    }
                },
                "31": {
                    "start": {
                        "line": 32,
                        "column": 0
                    },
                    "end": {
                        "line": 32,
                        "column": 38
                    }
                },
                "32": {
                    "start": {
                        "line": 33,
                        "column": 0
                    },
                    "end": {
                        "line": 33,
                        "column": 18
                    }
                },
                "33": {
                    "start": {
                        "line": 34,
                        "column": 0
                    },
                    "end": {
                        "line": 34,
                        "column": 3
                    }
                }
            },
            "s": {
                "2": 0,
                "3": 0,
                "4": 0,
                "6": 0,
                "7": 0,
                "8": 0,
                "9": 0,
                "10": 0,
                "12": 0,
                "13": 0,
                "14": 0,
                "16": 0,
                "18": 0,
                "20": 0,
                "21": 0,
                "22": 0,
                "23": 0,
                "27": 0,
                "28": 0,
                "29": 0,
                "30": 0,
                "31": 0,
                "32": 0,
                "33": 0
            },
            "branchMap": {
                "0": {
                    "type": "branch",
                    "line": 1,
                    "loc": {
                        "start": {
                            "line": 1,
                            "column": 869
                        },
                        "end": {
                            "line": 34,
                            "column": 3
                        }
                    },
                    "locations": [
                        {
                            "start": {
                                "line": 1,
                                "column": 869
                            },
                            "end": {
                                "line": 34,
                                "column": 3
                            }
                        }
                    ]
                }
            },
            "b": {
                "0": [
                    1
                ]
            },
            "fnMap": {
                "0": {
                    "name": "(empty-report)",
                    "decl": {
                        "start": {
                            "line": 1,
                            "column": 869
                        },
                        "end": {
                            "line": 34,
                            "column": 3
                        }
                    },
                    "loc": {
                        "start": {
                            "line": 1,
                            "column": 869
                        },
                        "end": {
                            "line": 34,
                            "column": 3
                        }
                    },
                    "line": 1
                }
            },
            "f": {
                "0": 1
            }
        }
    }
}

[rohan-at-sentry]

@dcramer @eliatcodecov would we only surface on a PR comment if score is below a certain threshold (i.e. less than 0.4 for example) ? My thinking is that (like a test ) you only wanna surface "failures".

What would be a good threshold?

[eliatcodecov]

thresholds are configured in the actual test, like so:

describeEval("capitals", {

  // ...

  // The threshold required for the average score for this eval to pass. This will be
  // based on the scorers you've provided, and in the case of Factuality, we might be
  // ok with a 60% score (see the implementation for why).
  threshold: 0.6, 
})

So if the junit output is reporting a failure, you already know you're below whatever threshold was specified as acceptable.

[dcramer]

In the context of this repo, yes there are threshold failures, but the default is 0.

For remote reporting I'd say two things:

1. You always want to know what the actual scores were
2. If you're doing historical reporting, you generally care about it the same was as coverage (did it regress)

The reason for my threshold behavior is because I'm not doing stateful reporting, and I'm trying to isolate it a bit more like normal unit tests. I dont know that a hardcoded score is desirable in most situations though.