feat(docs-server): dynamic sitemap discovery for automatic content adaptation

qiskit-docs-mcp-server/README.md

@@ -12,12 +12,13 @@ The Qiskit Documentation MCP Server provides AI assistants and agents with seaml
 
 ### Key Features
 
-- **📚 Complete Documentation Access**: Query all 16 Qiskit SDK modules, 6 addon packages, and 30+ implementation guides
+- **📚 Complete Documentation Access**: Query all Qiskit SDK modules, addon packages, API references, guides, and tutorials
+- **🔄 Dynamic Content Discovery**: Automatically discovers available documentation from the live sitemap — no manual updates needed when new content is published
 - **📖 Implementation Guides**: Access best practices for optimization, error mitigation, dynamic circuits, and more
 - **🔍 Smart Search**: Search across the entire Qiskit documentation with fuzzy matching
 - **🎯 No Authentication Required**: Public documentation access without API tokens
 - **📝 Markdown Output**: Clean, formatted documentation ready for AI consumption
-- **⚡ Fast Retrieval**: Efficient HTTP-based documentation fetching with configurable timeouts
+- **⚡ Fast Retrieval**: Efficient HTTP-based documentation fetching with TTL caching and configurable timeouts
 
 ## Components
 
@@ -33,15 +34,25 @@ The server implements three tools for documentation access:
 
 ### Resources
 
-The server provides four resources for listing available documentation:
+The server provides six resources for listing available documentation. Content lists for modules, addons, guides, tutorials, and API packages are **dynamically discovered** from the documentation sitemap and cached, with hardcoded fallback values used when the sitemap is unreachable.
 
 | Resource URI | Description |
 |--------------|-------------|
-| `qiskit-docs://modules` | List of all Qiskit SDK modules with descriptions |
-| `qiskit-docs://addons` | List of Qiskit addon modules and tutorials |
+| `qiskit-docs://modules` | List of all Qiskit SDK modules with URL paths |
+| `qiskit-docs://addons` | List of Qiskit addon packages with URL paths |
 | `qiskit-docs://guides` | List of implementation guides and best practices |
+| `qiskit-docs://tutorials` | List of Qiskit tutorials with URL paths |
+| `qiskit-docs://api-packages` | List of API packages (runtime, transpiler, REST APIs, etc.) |
 | `qiskit-docs://error-codes` | List of Qiskit error code categories |
 
+### Resource Templates
+
+| Resource URI | Description |
+|--------------|-------------|
+| `qiskit-docs://modules/{module_name}` | Documentation for a specific SDK module |
+| `qiskit-docs://guides/{guide_name}` | A specific implementation guide |
+| `qiskit-docs://addons/{addon_name}` | Documentation for a specific addon package |
+
 ## Prerequisites
 
 - Python 3.10 or higher
@@ -302,12 +313,40 @@ All responses include rich metadata:
 }
 ```
 
+### Dynamic Sitemap Discovery
+
+Resource lists (modules, addons, guides, tutorials, API packages) are automatically discovered from the live documentation sitemap at startup. This means the server adapts to new content without code changes. If the sitemap is unreachable, the server falls back to hardcoded values in `constants.py`.
+
+To update the hardcoded fallback values from the live sitemap:
+
+```bash
+cd qiskit-docs-mcp-server
+uv run python scripts/update_fallback_constants.py
+```
+
+This prints updated constant lists that can be copied into `constants.py`.
+
 ### HTML to Markdown Conversion
 
 Documentation is automatically converted from HTML to clean Markdown format, optimized for AI consumption and human readability.
 
 ## Development
 
+### Project Structure
+
+```
+src/qiskit_docs_mcp_server/
+├── server.py           # MCP server definition (tools, resources, prompts)
+├── data_fetcher.py     # Business logic for fetching and processing documentation
+├── http.py             # HTTP infrastructure: client management, caching, retries
+├── sitemap.py          # Dynamic sitemap discovery and page classification
+├── html_processing.py  # HTML content extraction and markdown conversion
+└── constants.py        # Configuration constants and hardcoded fallback values
+
+scripts/
+└── update_fallback_constants.py  # Regenerate fallback values from live sitemap
+```
+
 ### Running Tests
 
 ```bash

qiskit-docs-mcp-server/pyproject.toml

@@ -6,6 +6,7 @@ readme = "README.md"
 requires-python = ">=3.10,<3.15"
 dependencies = [
     "beautifulsoup4>=4.12.0",
+    "defusedxml>=0.7.1",
     "fastmcp>=2.8.1,<3",
     "html2text>=2020.1.16",
     "httpx>=0.28.1",
@@ -54,6 +55,7 @@ dev = [
     "mypy>=1.15.0",
     "pre-commit>=4.1.0",
     "ruff>=0.9.4",
+    "types-defusedxml>=0.7.0",
 ]
 test = [
     "pytest>=7.4.0",

qiskit-docs-mcp-server/scripts/update_fallback_constants.py

@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+# This code is part of Qiskit.
+#
+# (C) Copyright IBM 2026.
+#
+# This code is licensed under the Apache License, Version 2.0. You may
+# obtain a copy of this license in the LICENSE.txt file in the root directory
+# of this source tree or at http://www.apache.org/licenses/LICENSE-2.0.
+#
+# Any modifications or derivative works of this code must retain this
+# copyright notice, and modified files need to carry a notice indicating
+# that they have been altered from the originals.
+
+"""Fetch the live sitemap and print updated fallback constant lists.
+
+Run this script periodically to keep the hardcoded fallback values in
+``constants.py`` in sync with the live Qiskit documentation sitemap.
+
+Usage:
+    cd qiskit-docs-mcp-server
+    uv run python scripts/update_fallback_constants.py
+"""
+
+from __future__ import annotations
+
+import sys
+
+import httpx
+
+
+sys.path.insert(0, "src")
+from qiskit_docs_mcp_server.constants import SITEMAP_URL
+from qiskit_docs_mcp_server.sitemap import _parse_sitemap_xml
+
+
+def _format_list(name: str, values: list[str]) -> str:
+    """Format a Python list constant."""
+    items = ",\n".join(f'    "{v}"' for v in values)
+    return f"{name}: list[str] = [\n{items},\n]"
+
+
+def main() -> None:
+    print(f"Fetching sitemap from {SITEMAP_URL} ...")
+    response = httpx.get(SITEMAP_URL, follow_redirects=True, timeout=30.0)
+    response.raise_for_status()
+    xml_text = response.text
+
+    pages = _parse_sitemap_xml(xml_text)
+
+    print("\nDiscovered:")
+    for category, items in pages.items():
+        print(f"  {category}: {len(items)} entries")
+
+    print("\n" + "=" * 72)
+    print("Copy the following into constants.py (fallback lists section):")
+    print("=" * 72 + "\n")
+
+    mapping = {
+        "AVAILABLE_MODULES": "modules",
+        "AVAILABLE_ADDONS": "addons",
+        "AVAILABLE_API_PACKAGES": "api_packages",
+        "AVAILABLE_GUIDES": "guides",
+        "AVAILABLE_TUTORIALS": "tutorials",
+    }
+
+    for const_name, key in mapping.items():
+        print(_format_list(const_name, pages[key]))
+        print()
+
+
+if __name__ == "__main__":
+    main()