Fall back to secret for SQS queue URL when SourceCrawler CR doesn't set it

[PuneetPunamiya]

• SourceCrawler now reads SOURCE_S3_SQS_QUEUE_URL from its secretRef as a fallback, so pipeline authors don't have to hardcode the queue URL by hand.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 98ba771e-fee6-42a7-a13d-33e4a9ed8fc8

📥 Commits

Reviewing files that changed from the base of the PR and between 5a86b26 and 9e86f8a.

📒 Files selected for processing (2)

• internal/controller/controllerutils/secret_helpers.go
• internal/controller/sourcecrawler_controller.go

Included review availability: Your plan includes up to 12 reviews per rolling hour; 11 remain after this review.

---

📝 Walkthrough

Summary by CodeRabbit

• New Features

  • S3 crawlers can now retrieve their SQS queue URL from a referenced Kubernetes Secret when no queue URL is configured directly.
  • Secret-based queue configuration supports the SOURCE_S3_ prefix.

• Bug Fixes

  • Errors encountered while retrieving the queue URL are now reported during reconciliation.
  • Existing explicitly configured queue URLs continue to be used unchanged.

Walkthrough

The controller now resolves an SQS queue URL from a referenced Kubernetes Secret when the S3 crawler has no queue URL configured. The new helper reads the prefixed Secret key and propagates lookup errors.

Changes

SQS secret fallback

Layer / File(s)	Summary
Secret lookup and crawler reconciliation internal/controller/controllerutils/secret_helpers.go, internal/controller/sourcecrawler_controller.go	SQSQueueURLFromSecret reads <prefix>SQS_QUEUE_URL from a namespaced Secret. S3 crawler reconciliation uses the SOURCE_S3_ fallback when S3Config.SQSQueueURL is empty and reports lookup errors.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 9e86f

This change lets SourceCrawler obtain the SQS queue URL from its secret when the custom resource omits it; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: concaf, gshikhar2021

Sequence Diagram(s)

sequenceDiagram
  participant SourceCrawlerController
  participant SQSQueueURLFromSecret
  participant KubernetesSecretAPI
  SourceCrawlerController->>SQSQueueURLFromSecret: request SOURCE_S3_ queue URL
  SQSQueueURLFromSecret->>KubernetesSecretAPI: fetch namespaced Secret
  KubernetesSecretAPI-->>SQSQueueURLFromSecret: return queue URL or error
  SQSQueueURLFromSecret-->>SourceCrawlerController: apply fallback or report error

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the fallback to a Secret when the SourceCrawler custom resource lacks an SQS queue URL.
Description check	✅ Passed	The description directly explains the added Secret-based fallback for the SQS queue URL.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}