feat(analyzer): Add flag to `dart analyze` to display effective analysis options and active lint rules

[gspencergoog]

Problem

Currently, it's difficult for developers to determine the final, resolved set of lint rules that are in effect for a given package or directory. The configuration is often spread across multiple analysis_options.yaml files (via include: statements), package presets (like package:lints/core.yaml), and potentially custom lint rules.

Developers cannot easily verify exactly how their options are being parsed, which rules are active, which directories are excluded, or what severity levels are applied.

Use Cases

1. Tooling Integration (LLMs): A primary motivation is to provide a definitive list of active lint rules (and optionally their descriptions) to external tools, such as LLMs. This allows the AI to generate or refactor code that adheres strictly to the project's specific coding standards.

2. Configuration Debugging: Developers (especially plugin authors or those new to a complex codebase) need a way to debug their analysis setup. A command to show the final "computed" configuration would confirm if:

   • include: statements are working as expected.
   • Files/directories are being correctly excluded.
   • Custom severities are being applied.
   • Experimental features are enabled.
   • Custom lint rules are being discovered and applied.

Proposed Solution

Add a new flag to dart analyze, such as --show-options or --show-config.

When run, this command would parse the analysis options for the target directory (just as regular analysis does) but instead of running the analysis, it would print a human-readable summary of the effective configuration.

Example usage:

dart analyze --show-options .
dart analyze --show-options test/

Output:

• A definitive list of all active lint rules (including any discovered custom lint rules).

Optional Output:

Additionally, a secondary flag (e.g., --show-descriptions) should be added. When present, the output should also include the short (two-ish-line) description for each active lint rule, pulling this information from existing metadata (like that found in pkg/linter/messages.yaml).

[scheglov]

Looks related to how we print analysis context collection state:

sdk/pkg/analyzer/test/src/dart/analysis/analysis_context_collection_test.dart

Line 980 in 9ed1efe

test_packageConfigWorkspace_singleAnalysisOptions_exclude() async {

We also have configuration to print more details.

class _AnalysisContextCollectionPrinterConfiguration {
  bool withAnalysisOptionsWithoutFiles = false;
  bool withEmptyContextRoots = false;
  bool withEnabledFeatures = false;
  bool withLintRules = false;
  bool withOptionFilesForContext = false;
}

[bwilkerson]

Tooling Integration (LLMs)

For this case, would it make more sense to have a tool exposed via the MCP server? I don't know whether the same general rules of discoverability apply to LLMs as they do for humans, but if I were thinking about this for human developers then I would say that this functionality is likely too hard to discover and that we might want a different approach.

Configuration Debugging

For this case, does it need to be accessible via a command-line tool?

The week before the summit we decided to redesign the analysis server's 'diagnostic' pages to be easier for developers to discover and use, and as part of that we're changing the name to the 'insights' pages. The idea is that these pages should be a way for developers in general, including plugin developers and analysis server developers, to get insights into how the analysis server is working.

Getting insights into the analysis options is definitely something that fits within that vision. We already have some of the data on the 'diagnostic' pages, but it's hard to find and not as well presented as it could be. I think we should definitely make it easier for developers to find these pages and to get the information they offer. I'm just wondering whether we also need a command-line tool to provide similar data, or whether IDE support is sufficient.

If we do need a command-line tool, then I can think of two possible directions:

• a flag that prints extra data, as described above,
• a separate tool that opens the insight pages in a browser without requiring a running IDE.

[gspencergoog]

For this case, would it make more sense to have a tool exposed via the MCP server?

This is fine, and maybe that's how it would need to be exposed, but there would still need to be some function call that was part of the analysis server that could get the information for the MCP server.

The main thing is that I'd like to be able to enable the LLM to discover a priori what output is expected of it when it writes code in a project. For instance if "always_specify_types" is turned on, then that has a significant impact on the "correct" output, and if it knows that ahead of time, it can avoid a lot of churn trying to eliminate all of the lint violations after the it writes the wrong code.

For both of the cases (integration and debugging), I don't think static pages are going to give me what I want: The presented data should be reflective of the current setup in the current project, either in the case of the "active" lints, or in the diagnostic case, the parsed options, etc. If the "insights" pages are dynamic, then that would satisfy that desire.

For this case, does it need to be accessible via a command-line tool?

That seems like the easiest way to integrate it into tooling (like CI that wants to verify the lints haven't changed, or if I just want to grep for something from a script). For the MCP server case, it probably just needs to be a function, although if people don't have our MCP server configured, it would be convenient for the LLM to be able to invoke a command line equivalent, as it can with most other dart commands.

[bwilkerson]

This is fine, and maybe that's how it would need to be exposed, but there would still need to be some function call that was part of the analysis server that could get the information for the MCP server.

Agreed. The amount of work in the analysis server is the same either way. It's "just" a case of finding the optimal way(s) to expose the functionality. I didn't explicitly say so, but I agree that this would likely be useful information for an LLM to have.

If the "insights" pages are dynamic, then that would satisfy that desire.

The insights pages reflect the state of the server at the time each page was requested. They're only dynamic in the sense that you can refresh the page to update the content. I don't know whether that would be sufficient.

But I'm not really suggesting that the LLMs use the insights pages. My understanding is that they do better with markdown, so having a dedicated protocol to access this information in a markdown format would likely be the better approach. What I am suggesting is that the insight pages might be a better way for developers to get this information because I think we can present the information in a more useful form by using that medium.

That seems like the easiest way to integrate it into tooling (like CI that wants to verify the lints haven't changed, or if I just want to grep for something from a script).

I understand the use case of a developer wanting to understand how analysis is configured in some package, and to understand where that configuration comes from. I'm not sure I understand the use case of a CI that wants to prevent the configuration from changing or that wants to grep through some description of the configuration (that is, what goal motivates that behavior), so I'd need more information in order to decide whether those are use cases we want to support.

If there are important use cases for having a CLI tool then we should add that; if not then I'd rather use the insights pages.

[gspencergoog]

My understanding is that they do better with markdown, so having a dedicated protocol to access this information in a markdown format would likely be the better approach.

Yes, that's definitely true. Better to provide it with Markdown than HTML from the insights pages. I think providing this on the insight pages is also useful for developers, though, and is probably better UX for them.

Markdown is more understandable by LLMs as-is, but JSON output is easier to do pre-processing on (e.g. filtering) before generating the context, although they can take either. If I had to pick one, I think I'd actually pick JSON, since it's easy to interpolate that into a Markdown string, but hard to parse Markdown to go the other way.

If there are important use cases for having a CLI tool then we should add that; if not then I'd rather use the insights pages.

The only CI use cases I can think of are: CI wants to enforce that particular flags are/are not turned on, or to detect that they have changed to make sure unintentional changes haven't been made. Mostly I can see this as part of enforcing coding standards for a repo. I'm not saying that this is necessarily an important use case. Just one I can see someone wanting.

As a user, I can see wanting to grep through an output list of lints to see if one is enabled. If I want to write a tool that generates code (algorithmically, not via LLM), I may want to conform to specific lints enabled for a project. Or, if I'm generating a prompt for an LLM, I may want to make parts of the prompt dependent upon which lints are enabled.

So, I guess these three cases seem interesting to me:

• Markdown or JSON output for an LLM to include in its context.
• HTML on the insights pages for developers who just want to browse it and confirm something.
• Some form of stdio output that can be processed by a tool (a tool like grep, but could be JSON too).

Part of the motivation here is that I'm imagining an enhancement to the Dart MCP server that will deliver not only the analyzer issues that were found, but a way to get the descriptions of the types of issues found so it can have clear instructions about what to do about the issues, but limit it to the ones that are found, to reduce context tokens. The analyzer doesn't need to support that itself, but access to the information would let the MCP server implement that.

[bwilkerson]

Part of the motivation here is that I'm imagining an enhancement to the Dart MCP server that will deliver not only the analyzer issues that were found, but a way to get the descriptions of the types of issues found so it can have clear instructions about what to do about the issues, but limit it to the ones that are found, to reduce context tokens.

That's very interesting; I've had similar thoughts.

The analysis server has URLs for most of the diagnostics that point to pages containing descriptions of why the diagnostic was produced and common ways of fixing the problem (see https://dart.dev/tools/diagnostics/ambiguous_set_or_map_literal_both for an example). That information is generated from markdown encoded descriptions in the analyzer's code base, so it wouldn't be hard to send that information back with the list of diagnostics (in one form or another: markdown, a URL, a resource, etc.).