Auto-discover connector documentation links via docs sitemap

[daneshk]

Description

Fixes: wso2/product-integrator#1731

Summary

Previously, every new connector required a manual change to CONNECTOR_DOCS in connector-utils.ts to add a Documentation button to its detail page. This PR removes that requirement by auto-discovering published doc pages from the docs site's sitemap.

How it works

• Hardcoded entries (CONNECTOR_DOCS) remain unchanged — existing connectors with non-standard URL slugs continue to resolve instantly with no sitemap lookup.
• New connectors are auto-discovered — for connectors not in the hardcoded map, a documentation URL is derived from the connector's Area/ keyword metadata using the standard slug convention: {base}/{category}/{packageName}/connector-overview/. The derived URL is then validated against the published docs sitemap before the Documentation button is shown.
• Sitemap is fetched once and cached — parsed on first visit to a connector detail page, with the result (just the set of package names) stored in localStorage for 6 hours. Concurrent calls within a session are deduplicated via a module-level
  promise.
• Non-blocking — the sitemap fetch runs in parallel with fetchPackageDetails at page mount. While the check is pending, the button renders in a disabled loading state rather than immediately falling back to Ballerina Central.
• Graceful fallback — if docs aren't published yet, the network fails, or CORS blocks the request, the button falls back to "View on Ballerina Central" automatically.

Also fixed

• Added sap.jco and microsoft.sharepoint.pages to CONNECTOR_DOCS with their correct non-standard URLs.

Test plan

• A hardcoded connector (e.g. kafka) shows the Documentation button immediately with no spinner
• A new connector with docs at the standard path shows the spinner briefly, then resolves to Documentation
• A connector whose docs aren't published yet shows the spinner briefly, then falls back to Ballerina Central
• Navigating between connector detail pages does not re-fetch the sitemap (check Network tab — only one sitemap request per 6h)
• Hard-refreshing the page with cleared localStorage triggers a fresh sitemap fetch
• sap.jco detail page links to erp-business/sap.jco/overview
• microsoft.sharepoint.pages detail page links to storage-file/microsoft.sharepoint.pages/connector-overview

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR replaces the per-URL checkDerivedDocsUrl function (which issued individual HEAD requests with sessionStorage caching) with getDocumentedConnectors, which fetches the published docs sitemap.xml once, extracts all documented connector package names via regex, caches them in localStorage for 6 hours, and deduplicates concurrent calls using a module-level promise. A hardcoded entry for microsoft.sharepoint.pages is also added to CONNECTOR_DOCS. In ConnectorDetailPage, docsUrl state gains a third value (null) to represent the resolving state, a prefetch effect runs on mount, derived docs validation is switched to a sitemap membership check, and the sidebar renders a disabled button with a spinner while the check is pending.

Suggested Reviewers

• aashikam
• sm1990

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: auto-discovering connector documentation links via the docs sitemap instead of requiring manual configuration.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The pull request description comprehensively documents the changes, approach, and test plan with clear before/after behavior and implementation details.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/connector-utils/connector-utils.ts`:
- Around line 666-669: The catch handler attempting to reset sitemapPromise to
null is unreachable because the async IIFE wrapping the sitemap fetch logic
catches all errors internally and always resolves successfully, so the promise
never rejects. To fix this, modify the inner catch block within the async IIFE
to rethrow the error after handling it (or throw a new error), which will cause
the outer promise to reject and allow the .catch() handler on sitemapPromise to
execute and properly reset the variable for retry on the next call.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 839c775f-9d45-4b7d-90cd-f9612ca7da13

📥 Commits

Reviewing files that changed from the base of the PR and between 67bb235 and bbcd0b7.

📒 Files selected for processing (3)

• src/lib/connector-utils/connector-utils.ts
• src/lib/connector-utils/index.ts
• src/pages/ConnectorDetailPage.tsx

[ThisaruGuruge]

Don't we need to add the microsoft.sharepoint.sites connector as well?

[daneshk]

No need. This will be discovered automatically. Going forward, we don't need to add it.

[ThisaruGuruge]

LGTM