HDDS-14816. Add Recon AI Assistant backend foundation with Multi LLM integration - #9915
Conversation
fc5c38b to
287c739
Compare
devmadhuu
left a comment
There was a problem hiding this comment.
@ArafatKhan2198 thanks for the patch, few comments at a high level in initial review. Please check.
Also one observation as per output you mentioned in PR description, various models output for same question: "Give me all the information there is about the current state of the cluster", I can see some wrong answers even. E.g. for gemini-3-flash-preview, it says that "All 12 internal Recon tasks are functioning correctly". Not sure from where the model got 12 tasks. This is surely it is hallucinating.
priyeshkaratha
left a comment
There was a problem hiding this comment.
Thanks @ArafatKhan2198 for the patch. Please check the comments.
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
priyeshkaratha
left a comment
There was a problem hiding this comment.
Thanks @ArafatKhan2198 for updating the patch. Please check the inline comment. Apart from the code overall LGTM
devmadhuu
left a comment
There was a problem hiding this comment.
@ArafatKhan2198 , thanks for the patch. Still some issues are not fixed. Kindly check. And current PR design needs to make flexible and extensible for future MCP perspective as current PR code is not extensible, but can be doable in next PR, but to merge this PR, needs to handle bare minimum comments.
And we can think working direction to make it more extensible, following points are major, but lets not handle in current PR.
Step 1: Extract Agent interface from ChatbotAgent. ChatbotAgent becomes ReconApiAgent implements Agent. The Guice module binds Set via Multibindings. ChatbotEndpoint talks to a new AgentOrchestrator (or just the Agent directly for now with a trivial single-agent router).
Step 2: Extract DataSourceExecutor interface from ToolExecutor. ToolExecutor becomes ReconRestExecutor implements DataSourceExecutor. Each Agent gets injected with the appropriate executor(s). A future JmxMetricsAgent would be injected with JmxHttpExecutor which simply wraps the existing JmxServiceProviderImpl.
Step 3: Make apiSchema a per-agent concern, not a singleton field. Each Agent loads its own tool schema. The AgentOrchestrator's tool-selection prompt is composed dynamically from all registered agents' schemas — a simple String.join across agent.getSchema().
| String fullEndpoint = endpoint; | ||
|
|
||
| // Ensure the path always starts with "/api/v1/" | ||
| if (!fullEndpoint.startsWith("/api/v1/")) { |
There was a problem hiding this comment.
@ArafatKhan2198 this is still not resolved. Basically ToolExecutor.normalizeEndpoint() only ensures the path starts with /api/v1/, but a compromised prompt or a hallucinating LLM could inject paths like /api/v1/../actuator/shutdown or /api/v1/%2F..%2F. You should maintain an allowlist of accepted path prefixes (e.g., /api/v1/containers, /api/v1/datanodes, etc.) and reject anything not on the list. The current requireSafeScope guard only checks for listKeys — it provides no protection for other endpoints.
fabbd8b to
fc45b8b
Compare
devmadhuu
left a comment
There was a problem hiding this comment.
One critical and few nits.
adoroszlai
left a comment
There was a problem hiding this comment.
Please do take a look at CI results in fork.
| <dependency> | ||
| <groupId>dev.langchain4j</groupId> | ||
| <artifactId>langchain4j-anthropic</artifactId> | ||
| <version>0.36.2</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>dev.langchain4j</groupId> | ||
| <artifactId>langchain4j-core</artifactId> | ||
| <version>0.36.2</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>dev.langchain4j</groupId> | ||
| <artifactId>langchain4j-google-ai-gemini</artifactId> | ||
| <version>0.36.2</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>dev.langchain4j</groupId> | ||
| <artifactId>langchain4j-open-ai</artifactId> | ||
| <version>0.36.2</version> | ||
| </dependency> |
There was a problem hiding this comment.
every other third-party library in the project follows this same pattern of individual version entries. Introducing a BOM import for just this one library would be inconsistent with how the rest of the project is structured.
| <dependency> | ||
| <groupId>dev.langchain4j</groupId> | ||
| <artifactId>langchain4j-core</artifactId> | ||
| </dependency> |
There was a problem hiding this comment.
Jar files under share/ozone/lib in the build have changed.
Please update:
* hadoop-ozone/dist/src/main/license/bin/LICENSE.txt
(add new dependencies with the appropriate license, delete any removed dependencies)
* hadoop-ozone/dist/src/main/license/jar-report.txt
(based on the diff shown below)
If you notice unexpected differences (can happen when the check is run in a
fork), please first update your branch from upstream master to get any other
recent dependency changes.
If you are running this locally after build with -DskipShade, please ignore any
ozone-filesystem jars reported to be missing.
Changes detected:
--- /dev/fd/63 2026-05-27 10:02:54.566056827 +0000
+++ /dev/fd/62 2026-05-27 10:02:54.566056827 +0000
@@ -35,6 +35,7 @@
share/ozone/lib/commons-pool2.jar
share/ozone/lib/commons-text.jar
share/ozone/lib/commons-validator.jar
+share/ozone/lib/converter-jackson.jar
share/ozone/lib/curator-client.jar
share/ozone/lib/curator-framework.jar
share/ozone/lib/derby.jar
@@ -158,6 +159,7 @@
share/ozone/lib/jsp-api.jar
share/ozone/lib/jspecify.jar
share/ozone/lib/jsr311-api.jar
+share/ozone/lib/jtokkit.jar
share/ozone/lib/kerb-core.jar
share/ozone/lib/kerb-crypto.jar
share/ozone/lib/kerb-util.jar
@@ -165,7 +167,14 @@
share/ozone/lib/kerby-config.jar
share/ozone/lib/kerby-pkix.jar
share/ozone/lib/kerby-util.jar
+share/ozone/lib/kotlin-stdlib-common.jar
+share/ozone/lib/kotlin-stdlib-jdk7.jar
+share/ozone/lib/kotlin-stdlib-jdk8.jar
share/ozone/lib/kotlin-stdlib.jar
+share/ozone/lib/langchain4j-anthropic.jar
+share/ozone/lib/langchain4j-core.jar
+share/ozone/lib/langchain4j-google-ai-gemini.jar
+share/ozone/lib/langchain4j-open-ai.jar
share/ozone/lib/listenablefuture-empty-to-avoid-conflict-with-guava.jar
share/ozone/lib/log4j-api.jar
share/ozone/lib/log4j-core.jar
@@ -193,7 +202,11 @@
share/ozone/lib/netty-transport.Final.jar
share/ozone/lib/nimbus-jose-jwt.jar
share/ozone/lib/okhttp-jvm.jar
+share/ozone/lib/okhttp-sse.jar
+share/ozone/lib/okhttp.jar
share/ozone/lib/okio-jvm.jar
+share/ozone/lib/okio.jar
+share/ozone/lib/openai4j.jar
share/ozone/lib/opentelemetry-api.jar
share/ozone/lib/opentelemetry-common.jar
share/ozone/lib/opentelemetry-context.jar
@@ -259,6 +272,7 @@
share/ozone/lib/re2j.jar
share/ozone/lib/reflections.jar
share/ozone/lib/reload4j.jar
+share/ozone/lib/retrofit.jar
share/ozone/lib/rocksdb-checkpoint-differ.jar
share/ozone/lib/rocksdbjni.jar
share/ozone/lib/simpleclient.jar
https://github.com/ArafatKhan2198/ozone/actions/runs/26503992485/job/78053259617#step:13:18
There was a problem hiding this comment.
Done. I've updated both jar-report.txt and LICENSE.txt to include the new transitive dependencies introduced by LangChain4j (such as jtokkit, okhttp, retrofit, and kotlin-stdlib).
| <artifactId>langchain4j-core</artifactId> | ||
| <version>0.36.2</version> |
There was a problem hiding this comment.
This does not work with Java 8.
Error: bad class file: /home/runner/.m2/repository/dev/langchain4j/langchain4j-core/0.36.2/langchain4j-core-0.36.2.jar(dev/langchain4j/data/message/AiMessage.class)
Error: class file has wrong version 61.0, should be 52.0
There was a problem hiding this comment.
Thanks for pointing this out. LangChain4j officially dropped Java 8 support starting with 0.36.0.
I have downgraded the langchain4j.version to 0.35.0. This is the latest release that should compiles to Java 8.
There was a problem hiding this comment.
Files with unapproved licenses:
/home/runner/work/ozone/ozone/hadoop-ozone/recon/src/main/resources/chatbot/recon-summarization-prompt.txt
/home/runner/work/ozone/ozone/hadoop-ozone/recon/src/main/resources/chatbot/recon-fallback-prompt-template.txt
/home/runner/work/ozone/ozone/hadoop-ozone/recon/src/main/resources/chatbot/recon-api-guide.md
/home/runner/work/ozone/ozone/hadoop-ozone/recon/src/main/resources/chatbot/recon-tool-selection-prompt-preamble.txt
There was a problem hiding this comment.
I tried to avoid adding license headers to these files, as the raw text is fed directly to the LLM and the header might interfere with the prompts. Instead, I have added it to rat-exclusions.txt to exclude the chatbot resources directory from the RAT check.
devmadhuu
left a comment
There was a problem hiding this comment.
Thanks @ArafatKhan2198 , LGTM +1. Just a nit for adding log. PR approved. Add the log.
devmadhuu
left a comment
There was a problem hiding this comment.
Please check few minor comments.
|
|
||
| // ── Execution policy ──────────────────────────────────────── | ||
| public static final String OZONE_RECON_CHATBOT_EXEC_MAX_RECORDS = OZONE_RECON_CHATBOT_PREFIX | ||
| + "exec.max.records"; |
There was a problem hiding this comment.
If this is really needed.
| ChatbotConfigKeys.OZONE_RECON_CHATBOT_EXEC_PAGE_SIZE, | ||
| ChatbotConfigKeys.OZONE_RECON_CHATBOT_EXEC_PAGE_SIZE_DEFAULT); | ||
|
|
||
| LOG.info("ToolExecutor initialized with Recon URL: {}, connectTimeoutMs={}, " + |
There was a problem hiding this comment.
Pls check if configs are needed in this class to read.
adoroszlai
left a comment
There was a problem hiding this comment.
@ArafatKhan2198 There are still some minor problems with the patch already merged (note that I only checked the dependency sections, not the actual implementation). BTW, next time please mention that AI assistance was used for implementing the changes (see policy).
| OZONE-SITE.XML_ozone.om.http-address=om:9874 | ||
| OZONE-SITE.XML_ozone.scm.http-address=scm:9876 | ||
| OZONE-SITE.XML_ozone.scm.container.size=100MB | ||
| OZONE-SITE.XML_ozone.scm.container.size=1GB |
There was a problem hiding this comment.
Why was this changed? We use a small value to reduce chances of disk out of space error in acceptance tests.
| Apache License 2.0 | ||
| ===================== | ||
| com.squareup.okhttp3:okhttp | ||
| com.squareup.okhttp3:okhttp-sse | ||
| com.squareup.okio:okio | ||
| com.squareup.retrofit2:converter-jackson | ||
| com.squareup.retrofit2:retrofit | ||
| dev.ai4j:openai4j | ||
| dev.langchain4j:langchain4j-anthropic | ||
| dev.langchain4j:langchain4j-core | ||
| dev.langchain4j:langchain4j-open-ai | ||
| org.jetbrains.kotlin:kotlin-stdlib-common | ||
| org.jetbrains.kotlin:kotlin-stdlib-jdk7 | ||
| org.jetbrains.kotlin:kotlin-stdlib-jdk8 | ||
|
|
||
| Apache License 2.0 | ||
| ===================== |
There was a problem hiding this comment.
Please don't duplicate License sections, add the dependencies in existing sections in alphabetical order.
| share/ozone/lib/apache-log4j-extras.jar | ||
| share/ozone/lib/aopalliance.jar | ||
| share/ozone/lib/aopalliance-repackaged.jar | ||
| share/ozone/lib/aopalliance.jar | ||
| share/ozone/lib/apache-log4j-extras.jar |
There was a problem hiding this comment.
Please don't move around lines unnecessarily.
| <dependency> | ||
| <!-- langchain4j-bom pins JUnit 5.10.0; override to keep Ozone on ${junit5.version}. --> | ||
| <groupId>org.junit.jupiter</groupId> | ||
| <artifactId>junit-jupiter-api</artifactId> | ||
| <version>${junit5.version}</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.junit.jupiter</groupId> | ||
| <artifactId>junit-jupiter-params</artifactId> | ||
| <version>${junit5.version}</version> | ||
| </dependency> |
| <dependency> | ||
| <!-- langchain4j-bom pins AWS SDK 2.21.44; override to keep Ozone on ${aws-java-sdk2.version}. --> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>apache-client</artifactId> | ||
| <version>${aws-java-sdk2.version}</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>auth</artifactId> | ||
| <version>${aws-java-sdk2.version}</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>aws-core</artifactId> | ||
| <version>${aws-java-sdk2.version}</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>http-client-spi</artifactId> | ||
| <version>${aws-java-sdk2.version}</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>identity-spi</artifactId> | ||
| <version>${aws-java-sdk2.version}</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>regions</artifactId> | ||
| <version>${aws-java-sdk2.version}</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>s3</artifactId> | ||
| <version>${aws-java-sdk2.version}</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>s3-transfer-manager</artifactId> | ||
| <version>${aws-java-sdk2.version}</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>sdk-core</artifactId> | ||
| <version>${aws-java-sdk2.version}</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>utils</artifactId> | ||
| <version>${aws-java-sdk2.version}</version> | ||
| </dependency> |
There was a problem hiding this comment.
These are unnecessary. AWS SDK v2 is only used in tests and that module uses the BOM:
ozone/hadoop-ozone/integration-test-s3/pom.xml
Lines 29 to 30 in 2020d86
|
@ArafatKhan2198 though PR is merged by you, but as discussed, I still don't see any test if chatbot is functional in secure cluster which is bare minimum requirement. |
What changes were proposed in this pull request?
What the Recon AI Chatbot Does (For Context)
The Recon AI Assistant is an intelligent query interface that bridges the gap between natural language and Recon's complex REST APIs. Instead of requiring administrators to memorize API endpoints and manually parse JSON payloads, the chatbot acts as an autonomous agent.
Example Scenario:
User Question: > "How many datanodes are unhealthy?"
The Traditional Workflow:
/containers/unhealthyor/datanodes).curlcommand.The New Chatbot Workflow:
Recon AI Chatbot: Backend Architecture Guide - https://docs.google.com/document/d/1jyYZz0llKwN7lexJzVRjxfzl1Iwdbx8dABwLXGzAajQ/edit?tab=t.0
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-14816
How was this patch tested?
Here is what you can add to the "How was this patch tested?" section of your PR description. I have formatted it to clearly show the testing methodology, the script you used to verify the multi-model routing, and the configuration instructions needed for reviewers to reproduce it!
How was this patch tested?
This patch was tested by configuring all three supported AI providers (Google Gemini, OpenAI, and Anthropic) via Docker configuration, and programmatically looping through 9 different AI models to verify that the LLMDispatcher correctly routes natural language queries to the appropriate provider without requiring the user to specify it.
1. Configuration Setup
To test this locally, reviewers can configure their API keys by adding the following properties to their
docker-configfile:2. API Verification Script
A shell script was used to hit the Chatbot REST API (
/api/v1/chatbot/chat), iterating through a predefined list of models to ensure proxying worked seamlessly across all client implementations.Attached are the test results after evaluating each provider model by asking a question and recording its response - testingChatBot.txt
3. Test Results
gpt-4.1to the OpenAIClient, andgemini-2.5-flashto the GeminiClient)./clusterState), executed the API call internally, and summarized the JSON payload back into natural English.claude-opus-4-6,claude-sonnet-4-6) without an API key present, returning expected JSON errors:{"error": "No API key configured for provider 'anthropic'."}