Add a Kaggle MCP usage example (#1209)

Kaggle protected MCP server has additional auth requirements that are addressed in this PR. Changes include:
1. A Kaggle MCP example to show usage with an api_key authentication provider
2. Fix to not pass None fields in the tool_args. This change is limited to MCP functions only (although it may work universally). The fix is optional but without it tool calls to the Kaggle MCP server fail.
3. Fix pydantic secret_str handling in the mcp client implementation.
4. Fix validations done in APIKeyAuthProviderConfig, custom header config is only required if auth_scheme is `HeaderAuthScheme.CUSTOM`

Note:
The tool descriptions on the kaggle MCP server are not very verbose which makes it challenging for agents to call the tool with the right format. To workaround you can either supply the tool schema to the agent or update tool description via mcp_client function group tool overrides. See `search_dataset` override in the sample config file:
<img width="582" height="734" alt="image" src="https://github.com/user-attachments/assets/e1920d75-55f9-4b03-8bc7-a97da388b390" />

Closes #1207 

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/develop/docs/source/resources/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.



## Summary by CodeRabbit

* **New Features**
  * CLI: add bearer-token auth via flag or environment variable.
  * Example: add Kaggle MCP example with README, workflow config, and packaging.

* **Bug Fixes / Improvements**
  * Client calls validate inputs and omit null fields before sending.
  * Secret values are unwrapped where required.
  * Validation enforces custom-scheme requirements and prevents conflicting auth/direct usage.

* **Tests**
  * Add tests covering bearer-token flows and error cases.

* **Documentation**
  * Update tutorial text and add Kaggle README with usage and troubleshooting.

<sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub>

Authors:
  - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)

Approvers:
  - Eric Evans II (https://github.com/ericevans-nv)

URL: #1209

Author: AnuradhaKaruppiah

ci/markdown-link-check-config.json

@@ -17,6 +17,9 @@
         },
         {
             "pattern": "^https://arize\\.com"
+        },
+        {
+            "pattern": "^https://milvus\\.io"
         }
     ]
 }

ci/vale/styles/config/vocabularies/nat/accept.txt

@@ -78,6 +78,7 @@ isort
 Jama
 Jira
 jsonlines
+[Kk]aggle
 Langfuse
 LangChain
 LangGraph

docs/source/_static/nat-maas-jira-ssa.png

@@ -115,7 +115,7 @@ Examining the `webquery_tool` function (`examples/getting_started/simple_web_que
     docs = [document async for document in loader.alazy_load()]
 ```
 
-For the new tool, instead of the `WebBaseLoader` class, use the [`langchain_community.document_loaders.DirectoryLoader`](https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.directory.DirectoryLoader.html) and [`langchain_community.document_loaders.TextLoader`](https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.text.TextLoader.html) classes.
+For the new tool, instead of the `WebBaseLoader` class, use the `langchain_community.document_loaders.DirectoryLoader` and `langchain_community.document_loaders.TextLoader` classes.
 
 ```python
     (ingest_dir, ingest_glob) = os.path.split(config.ingest_glob)

docs/source/_static/nat-maas-jira.png

@@ -0,0 +1,173 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Kaggle MCP Example
+
+This example demonstrates how to use the Kaggle MCP server with NVIDIA NeMo Agent Toolkit to interact with Kaggle's datasets, notebooks, models, and competitions.
+
+## Prerequisites
+
+- NeMo Agent Toolkit installed with MCP support (`nvidia-nat-mcp` package)
+- A Kaggle account and API token
+
+### Getting Your Kaggle Bearer Token
+
+The Kaggle MCP server uses bearer token authentication. Obtain your Kaggle bearer token from [Kaggle Account Settings](https://www.kaggle.com/settings/account).
+
+## Configuration
+
+The `config.yml` file uses the built-in `api_key` authentication provider with Bearer token scheme:
+
+```yaml
+authentication:
+  kaggle:
+    _type: api_key
+    raw_key: ${KAGGLE_BEARER_TOKEN}
+    auth_scheme: Bearer
+```
+
+### Environment Variables
+
+Set the following environment variable:
+
+```bash
+export KAGGLE_BEARER_TOKEN="your_kaggle_api_key_here"
+```
+
+## Usage
+
+Run the workflow with a query:
+
+```bash
+nat run --config_file examples/MCP/kaggle_mcp/configs/config.yml \
+  --input "Find the most popular datasets about natural language processing"
+```
+
+Example queries:
+- "What is the titanic dataset about?"
+- "What competitions are currently active?"
+
+## Configuration Details
+
+### MCP Client Setup
+
+The configuration connects to Kaggle's MCP server using:
+- **Transport**: `streamable-http` (recommended for HTTP-based MCP servers)
+- **URL**: `https://www.kaggle.com/mcp`
+- **Authentication**: Bearer token via the built-in `api_key` authentication provider
+
+## CLI Commands
+
+You can use the following CLI commands to interact with the Kaggle MCP server. This is useful for prototyping and debugging.
+
+### Discover Tools (No Authentication Required)
+
+To list available tools from the Kaggle MCP server:
+
+```bash
+nat mcp client tool list --url https://www.kaggle.com/mcp
+```
+
+### Get Tool Schema (No Authentication Required)
+
+To validate the tool schema:
+
+```bash
+nat mcp client tool list --url https://www.kaggle.com/mcp --tool search_datasets
+```
+
+### Authenticated Tool Calls
+
+The Kaggle MCP server requires bearer token authentication for some tool calls.
+
+#### Using Environment Variable (Recommended)
+
+```bash
+# Set your Kaggle bearer token
+export KAGGLE_BEARER_TOKEN="your_kaggle_api_key_here"
+
+# Search for Titanic datasets
+nat mcp client tool call search_datasets \
+  --url https://www.kaggle.com/mcp \
+  --bearer-token-env KAGGLE_BEARER_TOKEN \
+  --json-args '{"request": {"search": "titanic"}}'
+```
+
+#### Using Direct Token
+
+```bash
+# Search for Titanic datasets with direct token (less secure)
+nat mcp client tool call search_datasets \
+  --url https://www.kaggle.com/mcp \
+  --bearer-token "your_kaggle_api_key_here" \
+  --json-args '{"request": {"search": "titanic"}}'
+```
+
+**Note**: The `--bearer-token-env` approach is more secure because it doesn't expose the token in command history or process lists.
+
+## Troubleshooting
+
+### Agent Uses Wrong Parameter Names
+
+**Problem**: The agent generates tool calls with incorrect parameter names, such as using `query` instead of `search` for `search_datasets`.
+
+**Cause**: The default tool descriptions from Kaggle MCP are generic and don't specify parameter names, causing the LLM to infer incorrect names.
+
+**Solution**: Check the tool schema and add tool overrides in your `config.yml` to provide explicit parameter guidance:
+
+```bash
+nat mcp client tool list --url https://www.kaggle.com/mcp --tool search_datasets
+```
+
+After getting the tool schema, add the following tool overrides to your `config.yml`:
+
+```yaml
+function_groups:
+  kaggle_mcp_tools:
+    tool_overrides:
+      search_datasets:
+        description: >
+          Search for datasets on Kaggle. Use the 'search' parameter (not 'query')
+          to search by keywords. Example: {"request": {"search": "titanic"}}
+```
+
+### Permission Denied Errors
+
+**Problem**: Tool calls fail with "Permission 'datasets.get' was denied" or similar errors.
+
+**Cause**: Your Kaggle API token lacks the required permissions for certain operations.
+
+**Solution**:
+- Ensure you're using a valid Kaggle API key from https://www.kaggle.com/settings/account
+- Some operations require dataset ownership or special permissions
+- Use `search_datasets` for browsing (requires minimal permissions)
+- Use `list_dataset_files` only for datasets you own or have access to
+
+### CLI Tool Calls Work but Workflow Fails
+
+**Problem**: `nat mcp client tool call` succeeds but `nat run` with a workflow fails with the same tool.
+
+**Possible causes**:
+1. **Parameter validation**: CLI bypasses some validation that workflows enforce
+2. **Agent parameter inference**: Agent might use wrong parameter names (see "Agent Uses Wrong Parameter Names" above)
+
+**Solution**: Use `--direct` mode to test the raw MCP server behavior, then add tool overrides to guide the agent.
+
+## References
+
+- [Kaggle MCP Documentation](https://www.kaggle.com/docs/mcp)
+- [NeMo Agent Toolkit MCP Documentation](../../../docs/source/workflows/mcp/index.md)

docs/source/tutorials/create-a-new-workflow.md

@@ -0,0 +1,52 @@
+# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+llms:
+   # Tell NeMo Agent Toolkit which LLM to use for the agent
+   nim_llm:
+      _type: nim
+      model_name: meta/llama-3.1-70b-instruct
+      temperature: 0.0
+function_groups:
+  kaggle_mcp_tools:
+    _type: mcp_client
+    server:
+      transport: streamable-http
+      url: https://www.kaggle.com/mcp
+      auth_provider: kaggle
+    tool_overrides:
+      search_datasets:
+        description: >
+          Search for datasets on Kaggle. Use the 'search' parameter to search by keywords.
+          Returns a list of datasets with metadata including
+          title, owner, download count, and URL. Example: {"request": {"search": "titanic"}}
+
+authentication:
+  kaggle:
+    _type: api_key
+    raw_key: ${KAGGLE_BEARER_TOKEN}
+    auth_scheme: Bearer
+
+workflow:
+   # Use an agent that 'reasons' and 'acts'
+   _type: react_agent
+   # Give it access to our kaggle MCP tools
+   tool_names: [kaggle_mcp_tools]
+   # Tell it which LLM to use
+   llm_name: nim_llm
+   # Make it verbose
+   verbose: true
+   # Retry up to 3 times
+   parse_agent_response_max_retries: 3

examples/MCP/kaggle_mcp/README.md

@@ -0,0 +1,24 @@
+[build-system]
+build-backend = "setuptools.build_meta"
+requires = ["setuptools >= 64", "setuptools-scm>=8"]
+
+[tool.setuptools]
+packages = []
+
+[tool.setuptools_scm]
+git_describe_command = "git describe --long --first-parent"
+root = "../../.."
+
+[project]
+name = "nat_kaggle_mcp"
+dynamic = ["version"]
+dependencies = [
+    "nvidia-nat[mcp]~=1.4",
+]
+requires-python = ">=3.11,<3.14"
+description = "Kaggle MCP integration example with bearer token authentication"
+keywords = ["ai", "mcp", "protocol", "agents", "kaggle", "datasets"]
+classifiers = ["Programming Language :: Python"]
+
+[tool.uv.sources]
+nvidia-nat = { path = "../../..", editable = true }

examples/MCP/kaggle_mcp/configs/config.yml

@@ -34,6 +34,7 @@
 from nat.authentication.interfaces import AuthProviderBase
 from nat.authentication.oauth2.oauth2_auth_code_flow_provider_config import OAuth2AuthCodeFlowProviderConfig
 from nat.data_models.authentication import AuthResult
+from nat.data_models.common import get_secret_value
 from nat.plugins.mcp.auth.auth_flow_handler import MCPAuthenticationFlowHandler
 from nat.plugins.mcp.auth.auth_provider_config import MCPOAuth2ProviderConfig
 
@@ -371,7 +372,7 @@ async def _discover_and_register(self, response: httpx.Response | None = None):
                 # Manual registration mode
                 self._cached_credentials = OAuth2Credentials(
                     client_id=self.config.client_id,
-                    client_secret=self.config.client_secret,
+                    client_secret=get_secret_value(self.config.client_secret),
                 )
                 logger.info("Using manual client_id: %s", self._cached_credentials.client_id)
             else:

examples/MCP/kaggle_mcp/pyproject.toml

@@ -450,11 +450,19 @@ async def _response_fn(tool_input: BaseModel | None = None, **kwargs) -> str:
 
             # Preserve original calling convention
             if tool_input:
-                args = tool_input.model_dump()
+                args = tool_input.model_dump(exclude_none=True, mode='json')
                 return await session_tool.acall(args)
 
-            _ = session_tool.input_schema.model_validate(kwargs)
-            return await session_tool.acall(kwargs)
+            # kwargs arrives with all optional fields set to None because NAT's framework
+            # converts the input dict to a Pydantic model (filling in all Field(default=None)),
+            # then dumps it back to a dict. We need to strip out these None values because
+            # many MCP servers (e.g., Kaggle) reject requests with excessive null fields.
+            # We re-validate here (yes, redundant) to leverage Pydantic's exclude_none with
+            # mode='json' for recursive None removal in nested models.
+            # Reference: function_info.py:_convert_input_pydantic
+            validated_input = session_tool.input_schema.model_validate(kwargs)
+            args = validated_input.model_dump(exclude_none=True, mode='json')
+            return await session_tool.acall(args)
         except Exception as e:
             logger.warning("Error calling tool %s", tool.name, exc_info=True)
             return str(e)