[components] Scrapeless - add new actions #17086
Conversation
- Introduced a new README.md for Scrapeless, outlining its features and getting started guide. - Implemented multiple Scrapeless actions inclues `Crawler`, `Scraping API`, and `Universal Scraping API`.
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
|
@joy-chanboop is attempting to deploy a commit to the Pipedreamers Team on Vercel. A member of the Team first needs to authorize it. |
|
Important Review skippedMore than 25% of the files skipped due to max files limit. The review is being skipped to prevent a low-quality review. 117 files out of 224 files are above the max files limit of 100. Please upgrade to Pro plan to get higher limits. You can disable this status message by setting the WalkthroughThis update adds comprehensive Scrapeless platform support, including new action modules for crawling, scraping via API, and universal scraping with dynamic input properties. It updates the core app with new API endpoints and methods, adds extensive country constants and utility functions, introduces a detailed README, and increments versions and package dependencies for stability and clarity. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant ActionModule
participant ScrapelessApp
participant API
User->>ActionModule: Provide input (e.g., URL, query, options)
ActionModule->>ScrapelessApp: Call appropriate method (scrapingApi/universalScrapingApi/crawlerCrawl/crawlerScrape)
ScrapelessApp->>API: Send request (POST to endpoint)
API-->>ScrapelessApp: Respond with data or job/task ID
alt Immediate data
ScrapelessApp-->>ActionModule: Return result
else Job/task ID
loop Poll until complete
ScrapelessApp->>API: Check job/task status
API-->>ScrapelessApp: Return status/data
end
ScrapelessApp-->>ActionModule: Return aggregated result
end
ActionModule-->>User: Output result/summary
Poem
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
Thank you so much for submitting this! We've added it to our backlog to review, and our team has been notified. |
|
Thanks for submitting this PR! When we review PRs, we follow the Pipedream component guidelines. If you're not familiar, here's a quick checklist:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
🧹 Nitpick comments (8)
components/scrapeless/common/constants.mjs (2)
67-73: Correct the typo in “Google FLights” labels“FLights” is miss-cased. The canonical brand spelling is “Flights”.
- label: "Google FLights", + label: "Google Flights",and
- label: "Google FLights Chart", + label: "Google Flights Chart",Apply the same change in both entries to keep the UX professional and searchable.
139-1260: Consider externalising the 1 200-linecountryOptionsarrayEmbedding such a large literal bloats the bundle, hurts diff-reviewability and invites merge conflicts. Two lightweight alternatives:
- Move the JSON into
countryOptions.jsonandimportit.- Host it on a small CDN / gist and fetch lazily; cache in memory.
Either preserves type-safety while restoring readability to this module.
components/scrapeless/actions/crawler/crawler.mjs (2)
13-24: Grammar: “an API server”, not “a API server”- label: "Please select a API server", + label: "Please select an API server",
60-80: Label mis-match for shared prop
label: "URL to Crawl"is misleading whenapiServer === 'scrape'. Rename to neutral “URL”.- label: "URL to Crawl", + label: "URL",components/scrapeless/actions/scraping-api/scraping-api.mjs (1)
13-21: Grammar tweak- label: "Please select a API server", + label: "Please select an API server",components/scrapeless/README.md (1)
1-17: Add a quick code example for instant onboardingReaders benefit from a minimal “Hello World” snippet showing how to call one of the new actions programmatically. Consider adding a short example under “Getting Started”.
components/scrapeless/actions/universal-scraping-api/universal-scraping-api.mjs (1)
13-21: Grammar + UX nitLabel should read “Please select an API server” (missing article) to keep UI professional.
components/scrapeless/scrapeless.app.mjs (1)
90-118: Input sanitisation & unused variable
crawlerCrawl()ignores extra fields insubmitData(e.g.,browserOptions) yet still spreads them in the POST body later. Consider validating/whitelisting expected keys to avoid accidental leakage of large objects.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
components/scrapeless/README.md(1 hunks)components/scrapeless/actions/crawler/crawler.mjs(1 hunks)components/scrapeless/actions/scraping-api/scraping-api.mjs(1 hunks)components/scrapeless/actions/universal-scraping-api/universal-scraping-api.mjs(1 hunks)components/scrapeless/common/constants.mjs(1 hunks)components/scrapeless/common/utils.mjs(1 hunks)components/scrapeless/scrapeless.app.mjs(2 hunks)
components/scrapeless/actions/universal-scraping-api/universal-scraping-api.mjs
Outdated
Show resolved
Hide resolved
components/scrapeless/actions/universal-scraping-api/universal-scraping-api.mjs
Outdated
Show resolved
Hide resolved
|
Hi, could you please review and approve the GitHub Actions workflows for this PR when you have a moment? Thank you! |
dannyroosevelt
moved this to Ready for PR Review
in Component (Source and Action) Backlog
- fix lint code error - fix Component check error
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (3)
components/scrapeless/scrapeless.app.mjs (3)
58-73:⚠️ Potential issueNo timeout / max-attempts – risk of endless polling
while (true)continuously polls every second with no exit strategy. If the API never returns a result, the step hangs forever and burns credits.- while (true) { - await new Promise((resolve) => setTimeout(resolve, 1000)); + const MAX_ATTEMPTS = 120; // 2 min @ 1 s + let attempts = 0; + while (attempts++ < MAX_ATTEMPTS) { + await new Promise((resolve) => setTimeout(resolve, 1000)); const result = await this.getScrapeResult({ scrapeJobId: res.taskId, }); @@ return result; } } + throw new Error(`Timed out after ${MAX_ATTEMPTS}s waiting for scrape result`);
138-145: 🛠️ Refactor suggestionPagination loop clobbers
statusResponseand misusesnextURLRe-assigning
statusResponseinside the pagination loop overwrites the object you still rely on, and passingstatusResponse.nextto_makeRequest({ path })prependsbaseUrleven whennextis an absolute URL.- let data = statusResponse.data; - while (typeof statusResponse === "object" && "next" in statusResponse) { + let data = statusResponse.data; + let page = statusResponse; + while (page && typeof page === "object" && page.next) { if (data.length === 0) break; - statusResponse = await this._makeRequest({ - method: "GET", - path: statusResponse.next, + const isAbsolute = /^https?:\/\//.test(page.next); + page = await this._makeRequest({ + method: "GET", + ...(isAbsolute ? { url: page.next } : { path: page.next }), }); - data = data.concat(statusResponse.data); + data = data.concat(page.data ?? []); } - statusResponse.data = data; - return statusResponse; + statusResponse.data = data; + return statusResponse;
196-207:⚠️ Potential issueEndless polling in
crawlerScrapeSame issue as above: the
while (true)loop has no termination condition. Add a max-attempt / timeout or propagate the promise returned bymonitorJobStatusto reuse that logic.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (9)
components/scrapeless/actions/crawler/crawler.mjs(1 hunks)components/scrapeless/actions/get-scrape-result/get-scrape-result.mjs(1 hunks)components/scrapeless/actions/scraping-api/scraping-api.mjs(1 hunks)components/scrapeless/actions/submit-scrape-job/submit-scrape-job.mjs(1 hunks)components/scrapeless/actions/universal-scraping-api/universal-scraping-api.mjs(1 hunks)components/scrapeless/common/constants.mjs(1 hunks)components/scrapeless/common/utils.mjs(1 hunks)components/scrapeless/package.json(1 hunks)components/scrapeless/scrapeless.app.mjs(2 hunks)
✅ Files skipped from review due to trivial changes (3)
- components/scrapeless/package.json
- components/scrapeless/actions/submit-scrape-job/submit-scrape-job.mjs
- components/scrapeless/actions/get-scrape-result/get-scrape-result.mjs
🚧 Files skipped from review as they are similar to previous changes (5)
- components/scrapeless/common/constants.mjs
- components/scrapeless/actions/crawler/crawler.mjs
- components/scrapeless/actions/scraping-api/scraping-api.mjs
- components/scrapeless/actions/universal-scraping-api/universal-scraping-api.mjs
- components/scrapeless/common/utils.mjs
🔇 Additional comments (1)
components/scrapeless/scrapeless.app.mjs (1)
183-188: Duplicate POST removed – 👍Only a single POST is sent to start the scrape job now, preventing unintentional duplicate jobs.
No further issues spotted here.
|
Hi team, I've fixed all the issues pointed out in the review. The component is now ready for re-review. Any chance we could move this forward? Thank you! |
Hi team, I've fixed all the issues pointed out in the review. The component is now ready for re-review. Any chance we could move this forward? Thank you! |
There was a problem hiding this comment.
Hi @joy-chanboop thanks for this great contribution. I noticed there is a lot of polling request strategy to get the status of the service until it is done. Do you think you can make a refactor so it can look similar to this approach in contactout/actions/verify-email-bulk/verify-email-bulk.mjs I'm making this suggestion because we need to support $.flow.rerun step feature as much as we can.
jcortes
moved this from Ready for PR Review
to Changes Required
in Component (Source and Action) Backlog
- use nodes integration server to simple request logic
There was a problem hiding this comment.
Hi @joy-chanboop Just wondering why you are changing the base url of the api! just in case the official docs are here https://apidocs.scrapeless.com/
…de documentation links
|
Hi team, I've fixed all the issues pointed out in the review. The component is now ready for re-review. Any chance we could move this forward? Thank you! |
|
Hi @joy-chanboop let me know if this is what you are looking for. If not it's fine you can work also win Pipedream axios lib! |
There was a problem hiding this comment.
Hi @joy-chanboop pending minor change in crawler component!
…nality - Added @scrapeless-ai/sdk as a dependency. - Updated API endpoints for scraping and crawling functionalities. - Implemented error handling and job management for scraping tasks. - Refactored existing methods to utilize the new SDK for improved performance and reliability.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (1)
components/scrapeless/actions/scraping-api/scraping-api.mjs (1)
114-119: Reuse the existingcountryOptionsforgl
glshould be a dropdown fed bycountryOptions, ensuring valid ISO country codes and consistent UX.Import countryOptions at the top of the file:
import scrapeless from "../../scrapeless.app.mjs"; import { log } from "../../common/utils.mjs"; +import { countryOptions } from "../../common/constants.mjs";Then update the gl prop definition:
props.gl = { type: "string", label: "Country", description: "Parameter defines the country to use for the Google search. It's a two-letter country code. (e.g., us for the United States, uk for United Kingdom, or fr for France).", default: "us", + options: countryOptions, };
🧹 Nitpick comments (1)
components/scrapeless/actions/scraping-api/scraping-api.mjs (1)
79-79: Simplify the conditional check using optional chaining.The current condition
job && job?.data?.taskIdis redundant since optional chaining already handles null/undefined checks.- } else if (job && job?.data?.taskId) { + } else if (job?.data?.taskId) {
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (5)
components/scrapeless/actions/crawler/crawler.mjs(1 hunks)components/scrapeless/actions/scraping-api/scraping-api.mjs(1 hunks)components/scrapeless/actions/universal-scraping-api/universal-scraping-api.mjs(1 hunks)components/scrapeless/package.json(2 hunks)components/scrapeless/scrapeless.app.mjs(3 hunks)
🚧 Files skipped from review as they are similar to previous changes (4)
- components/scrapeless/package.json
- components/scrapeless/scrapeless.app.mjs
- components/scrapeless/actions/crawler/crawler.mjs
- components/scrapeless/actions/universal-scraping-api/universal-scraping-api.mjs
🧰 Additional context used
🪛 Biome (1.9.4)
components/scrapeless/actions/scraping-api/scraping-api.mjs
[error] 79-79: Change to an optional chain.
Unsafe fix: Change to an optional chain.
(lint/complexity/useOptionalChain)
Hi Jorge Cortes, I’ve refactored the action calls using the scrapeless-sdk, and set the official API endpoint as the base URL. I’ve also updated the crawler action key. Let me know if you have any suggestions or feedback! |
There was a problem hiding this comment.
Hi @joy-chanboop lgtm! Ready for QA! Can you please rebase with master so we don't have conflicts in pnpm-lock.yaml
jcortes
moved this from Changes Required
to Ready for QA
in Component (Source and Action) Backlog
There was a problem hiding this comment.
Actionable comments posted: 11
🧹 Nitpick comments (9)
components/habitica/habitica.app.mjs (1)
101-101: Consider making the client ID configurable.The hardcoded client ID might need to be configurable or moved to constants for maintainability.
-"x-client": "3a326108-1895-4c23-874e-37668c75f2ad-Pipedream", +"x-client": `${constants.CLIENT_ID}`,Then add to constants.mjs:
export default { CLIENT_ID: "3a326108-1895-4c23-874e-37668c75f2ad-Pipedream", // ... other constants };components/leaddyno/leaddyno.app.mjs (2)
4-4: Empty propDefinitions suggests incomplete implementation.The
propDefinitionsobject is empty, which typically indicates an incomplete component. LeadDyno components would likely need authentication and configuration properties defined.Would you like me to help generate common propDefinitions for API-based integrations like API keys, base URLs, etc.?
7-9: Replace console.log with appropriate logging.Using
console.login production code is not recommended. Consider using Pipedream's logging mechanisms or remove this debug code if it's not needed.- authKeys() { - console.log(Object.keys(this.$auth)); - }, + authKeys() { + // Return auth keys for debugging/validation purposes + return Object.keys(this.$auth); + },components/quickbooks/actions/send-invoice/send-invoice.mjs (1)
37-37: Inconsistent export summary key.This action uses
$.export("$summary", ...)while the void-invoice action uses$.export("summary", ...). Consider standardizing the export key across all actions for consistency.-$.export("$summary", `Successfully sent invoice with ID ${this.invoiceId}`); +$.export("summary", `Successfully sent invoice with ID ${this.invoiceId}`);components/quickbooks/actions/send-estimate/send-estimate.mjs (1)
37-37: Inconsistent export summary key.This action uses
$.export("$summary", ...)while the void-invoice action uses$.export("summary", ...). Consider standardizing the export key across all actions for consistency.-$.export("$summary", `Successfully sent estimate with ID ${this.estimateId}`); +$.export("summary", `Successfully sent estimate with ID ${this.estimateId}`);components/quickbooks/actions/create-estimate/create-estimate.mjs (1)
239-242: Consider validating date formats and the logical relationship between acceptance fields.The
ExpirationDateandAcceptedDatefields are expected in YYYY-MM-DD format but are not validated. Additionally,AcceptedByandAcceptedDateappear logically related - consider validating that both are provided together if one is specified.+ // Validate date formats + const dateRegex = /^\d{4}-\d{2}-\d{2}$/; + if (this.expirationDate && !dateRegex.test(this.expirationDate)) { + throw new ConfigurationError("Expiration Date must be in YYYY-MM-DD format"); + } + if (this.acceptedDate && !dateRegex.test(this.acceptedDate)) { + throw new ConfigurationError("Accepted Date must be in YYYY-MM-DD format"); + } + + // Validate logical relationship + if ((this.acceptedBy && !this.acceptedDate) || (!this.acceptedBy && this.acceptedDate)) { + throw new ConfigurationError("Both Accepted By and Accepted Date must be provided together"); + } + const data = { Line: lines, CustomerRef: { value: this.customerRefValue, }, ExpirationDate: this.expirationDate, AcceptedBy: this.acceptedBy, AcceptedDate: this.acceptedDate,components/quickbooks/actions/create-purchase-order/create-purchase-order.mjs (1)
201-201: Validate date format for Due Date.The
dueDatefield expects YYYY-MM-DD format but is not validated.+ // Validate date format + if (this.dueDate && !/^\d{4}-\d{2}-\d{2}$/.test(this.dueDate)) { + throw new ConfigurationError("Due Date must be in YYYY-MM-DD format"); + } const data = { Line: lines, VendorRef: { value: this.vendorRefValue, }, DueDate: this.dueDate,components/quickbooks/actions/update-estimate/update-estimate.mjs (1)
34-38: Validate date formats before API call.Similar to create-estimate, the date fields should be validated for correct format.
Add date validation before building the data object:
// Validate date formats const dateRegex = /^\d{4}-\d{2}-\d{2}$/; if (this.expirationDate && !dateRegex.test(this.expirationDate)) { throw new ConfigurationError("Expiration Date must be in YYYY-MM-DD format"); } if (this.acceptedDate && !dateRegex.test(this.acceptedDate)) { throw new ConfigurationError("Accepted Date must be in YYYY-MM-DD format"); }Also applies to: 48-50, 296-303
components/quickbooks/actions/update-invoice/update-invoice.mjs (1)
34-38: Add date format validation for consistency.The
dueDatefield should be validated for YYYY-MM-DD format before sending to the API.Add validation before building the data object:
// Validate date format if (this.dueDate && !/^\d{4}-\d{2}-\d{2}$/.test(this.dueDate)) { throw new ConfigurationError("Due Date must be in YYYY-MM-DD format"); }Also applies to: 318-318
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (84)
components/alttextify/alttextify.app.mjs(1 hunks)components/amazon_redshift/amazon_redshift.app.mjs(1 hunks)components/capgo/capgo.app.mjs(1 hunks)components/codeberg/codeberg.app.mjs(1 hunks)components/codeqr/codeqr.app.mjs(1 hunks)components/deepsouce/deepsouce.app.mjs(1 hunks)components/elmah_io/elmah_io.app.mjs(2 hunks)components/elmah_io/package.json(2 hunks)components/elmah_io/sources/new-error/new-error.mjs(4 hunks)components/finalscout/finalscout.app.mjs(1 hunks)components/habitica/.gitignore(0 hunks)components/habitica/actions/create-challenge/create-challenge.mjs(1 hunks)components/habitica/actions/delete-challenge/delete-challenge.mjs(1 hunks)components/habitica/actions/get-challenge/get-challenge.mjs(1 hunks)components/habitica/common/constants.mjs(1 hunks)components/habitica/habitica.app.mjs(1 hunks)components/habitica/package.json(1 hunks)components/keboola/keboola.app.mjs(1 hunks)components/leaddyno/leaddyno.app.mjs(1 hunks)components/leaddyno/package.json(1 hunks)components/picqer/picqer.app.mjs(1 hunks)components/predictleads/predictleads.app.mjs(1 hunks)components/quickbooks/actions/create-ap-aging-report/create-ap-aging-report.mjs(1 hunks)components/quickbooks/actions/create-bill/create-bill.mjs(2 hunks)components/quickbooks/actions/create-customer/create-customer.mjs(1 hunks)components/quickbooks/actions/create-estimate/create-estimate.mjs(1 hunks)components/quickbooks/actions/create-invoice/create-invoice.mjs(2 hunks)components/quickbooks/actions/create-payment/create-payment.mjs(1 hunks)components/quickbooks/actions/create-pl-report/create-pl-report.mjs(1 hunks)components/quickbooks/actions/create-purchase-order/create-purchase-order.mjs(1 hunks)components/quickbooks/actions/create-purchase/create-purchase.mjs(2 hunks)components/quickbooks/actions/create-sales-receipt/create-sales-receipt.mjs(2 hunks)components/quickbooks/actions/delete-purchase/delete-purchase.mjs(1 hunks)components/quickbooks/actions/get-bill/get-bill.mjs(1 hunks)components/quickbooks/actions/get-customer/get-customer.mjs(1 hunks)components/quickbooks/actions/get-invoice/get-invoice.mjs(1 hunks)components/quickbooks/actions/get-my-company/get-my-company.mjs(1 hunks)components/quickbooks/actions/get-payment/get-payment.mjs(1 hunks)components/quickbooks/actions/get-purchase-order/get-purchase-order.mjs(1 hunks)components/quickbooks/actions/get-purchase/get-purchase.mjs(1 hunks)components/quickbooks/actions/get-sales-receipt/get-sales-receipt.mjs(1 hunks)components/quickbooks/actions/get-time-activity/get-time-activity.mjs(1 hunks)components/quickbooks/actions/search-accounts/search-accounts.mjs(1 hunks)components/quickbooks/actions/search-customers/search-customers.mjs(1 hunks)components/quickbooks/actions/search-invoices/search-invoices.mjs(1 hunks)components/quickbooks/actions/search-items/search-items.mjs(1 hunks)components/quickbooks/actions/search-products/search-products.mjs(1 hunks)components/quickbooks/actions/search-purchases/search-purchases.mjs(1 hunks)components/quickbooks/actions/search-query/search-query.mjs(1 hunks)components/quickbooks/actions/search-services/search-services.mjs(1 hunks)components/quickbooks/actions/search-time-activities/search-time-activities.mjs(1 hunks)components/quickbooks/actions/search-vendors/search-vendors.mjs(1 hunks)components/quickbooks/actions/send-estimate/send-estimate.mjs(1 hunks)components/quickbooks/actions/send-invoice/send-invoice.mjs(1 hunks)components/quickbooks/actions/sparse-update-invoice/sparse-update-invoice.mjs(1 hunks)components/quickbooks/actions/update-customer/update-customer.mjs(1 hunks)components/quickbooks/actions/update-estimate/update-estimate.mjs(1 hunks)components/quickbooks/actions/update-invoice/update-invoice.mjs(1 hunks)components/quickbooks/actions/update-item/update-item.mjs(1 hunks)components/quickbooks/actions/void-invoice/void-invoice.mjs(1 hunks)components/quickbooks/common/utils.mjs(2 hunks)components/quickbooks/package.json(1 hunks)components/quickbooks/quickbooks.app.mjs(4 hunks)components/quickbooks/sources/new-customer-created/new-customer-created.mjs(1 hunks)components/quickbooks/sources/new-customer-updated/new-customer-updated.mjs(1 hunks)components/quickbooks/sources/new-employee-created/new-employee-created.mjs(1 hunks)components/quickbooks/sources/new-employee-updated/new-employee-updated.mjs(1 hunks)components/quickbooks/sources/new-invoice-created/new-invoice-created.mjs(1 hunks)components/quickbooks/sources/new-invoice-updated/new-invoice-updated.mjs(1 hunks)components/quickbooks/sources/new-item-created/new-item-created.mjs(1 hunks)components/quickbooks/sources/new-item-updated/new-item-updated.mjs(1 hunks)components/v1ce/v1ce.app.mjs(1 hunks)components/yay_com/yay_com.app.mjs(1 hunks)docs-v2/.tool-versions(1 hunks)docs-v2/pages/connect/api.mdx(1 hunks)docs-v2/pages/connect/components/index.mdx(1 hunks)docs-v2/pages/connect/mcp/openai.mdx(1 hunks)packages/connect-react/CHANGELOG.md(1 hunks)packages/connect-react/package.json(1 hunks)packages/connect-react/src/components/ComponentForm.tsx(2 hunks)packages/connect-react/src/components/ControlApp.tsx(3 hunks)packages/connect-react/src/components/RemoteOptionsContainer.tsx(2 hunks)packages/connect-react/src/hooks/form-context.tsx(6 hunks)packages/connect-react/src/utils/resolve-user-id.ts(1 hunks)
💤 Files with no reviewable changes (1)
- components/habitica/.gitignore
✅ Files skipped from review due to trivial changes (61)
- components/keboola/keboola.app.mjs
- components/amazon_redshift/amazon_redshift.app.mjs
- components/finalscout/finalscout.app.mjs
- components/picqer/picqer.app.mjs
- components/codeqr/codeqr.app.mjs
- components/codeberg/codeberg.app.mjs
- components/deepsouce/deepsouce.app.mjs
- components/v1ce/v1ce.app.mjs
- components/capgo/capgo.app.mjs
- components/alttextify/alttextify.app.mjs
- components/predictleads/predictleads.app.mjs
- components/yay_com/yay_com.app.mjs
- docs-v2/.tool-versions
- components/quickbooks/actions/delete-purchase/delete-purchase.mjs
- components/quickbooks/actions/get-time-activity/get-time-activity.mjs
- components/quickbooks/actions/create-payment/create-payment.mjs
- components/quickbooks/sources/new-invoice-updated/new-invoice-updated.mjs
- components/elmah_io/package.json
- components/quickbooks/actions/get-payment/get-payment.mjs
- components/quickbooks/actions/create-purchase/create-purchase.mjs
- components/habitica/common/constants.mjs
- components/quickbooks/actions/get-purchase-order/get-purchase-order.mjs
- components/quickbooks/actions/search-query/search-query.mjs
- components/quickbooks/actions/create-pl-report/create-pl-report.mjs
- components/quickbooks/actions/get-purchase/get-purchase.mjs
- components/quickbooks/package.json
- components/leaddyno/package.json
- components/quickbooks/actions/create-invoice/create-invoice.mjs
- components/quickbooks/actions/search-customers/search-customers.mjs
- components/quickbooks/actions/create-sales-receipt/create-sales-receipt.mjs
- components/quickbooks/sources/new-item-updated/new-item-updated.mjs
- components/quickbooks/actions/get-bill/get-bill.mjs
- components/quickbooks/actions/get-customer/get-customer.mjs
- components/quickbooks/actions/search-invoices/search-invoices.mjs
- components/quickbooks/actions/search-accounts/search-accounts.mjs
- packages/connect-react/package.json
- components/quickbooks/actions/search-items/search-items.mjs
- components/quickbooks/actions/get-invoice/get-invoice.mjs
- components/quickbooks/actions/get-sales-receipt/get-sales-receipt.mjs
- components/quickbooks/sources/new-customer-updated/new-customer-updated.mjs
- components/quickbooks/actions/update-item/update-item.mjs
- components/quickbooks/actions/get-my-company/get-my-company.mjs
- components/quickbooks/actions/create-ap-aging-report/create-ap-aging-report.mjs
- components/quickbooks/actions/create-bill/create-bill.mjs
- components/quickbooks/sources/new-invoice-created/new-invoice-created.mjs
- packages/connect-react/src/components/RemoteOptionsContainer.tsx
- components/quickbooks/sources/new-customer-created/new-customer-created.mjs
- components/quickbooks/actions/search-vendors/search-vendors.mjs
- components/quickbooks/actions/search-time-activities/search-time-activities.mjs
- components/quickbooks/actions/search-purchases/search-purchases.mjs
- components/quickbooks/sources/new-item-created/new-item-created.mjs
- components/quickbooks/actions/create-customer/create-customer.mjs
- components/quickbooks/actions/sparse-update-invoice/sparse-update-invoice.mjs
- components/quickbooks/sources/new-employee-created/new-employee-created.mjs
- packages/connect-react/CHANGELOG.md
- components/quickbooks/actions/search-services/search-services.mjs
- components/quickbooks/sources/new-employee-updated/new-employee-updated.mjs
- components/habitica/actions/get-challenge/get-challenge.mjs
- components/quickbooks/actions/search-products/search-products.mjs
- components/quickbooks/actions/update-customer/update-customer.mjs
- docs-v2/pages/connect/api.mdx
🧰 Additional context used
🧬 Code Graph Analysis (1)
packages/connect-react/src/components/ControlApp.tsx (1)
packages/connect-react/src/hooks/form-context.tsx (1)
useFormContext(58-67)
🔇 Additional comments (46)
docs-v2/pages/connect/mcp/openai.mdx (2)
156-156: Update cURL endpoint to/v1/responses
The request now targets the new/v1/responsesendpoint, aligning with the Node.jsclient.responses.createexample.
161-161: Switch to theinputproperty in the payload
Replacing the"messages"array with a single"input"string correctly matches the/v1/responsesAPI specification.packages/connect-react/src/components/ControlApp.tsx (1)
5-5: LGTM! Clean migration from userId to externalUserId.The changes correctly implement the user ID migration by:
- Importing and using
useFormContextto accessexternalUserId- Passing the resolved
externalUserIdto theuseAccountshookThe implementation maintains the same functionality while adopting the new user identifier pattern.
Also applies to: 32-32, 78-78
docs-v2/pages/connect/components/index.mdx (2)
19-19: Good terminology improvement.Changing "public demo app" to "SDK playground" is more descriptive and accurate.
22-29: Excellent documentation reorganization.The restructured implementation section improves logical flow by:
- First introducing server SDK/REST API options
- Then covering the frontend React SDK as a separate subsection
- Providing clearer section headers and descriptions
This makes it easier for developers to understand their integration options.
packages/connect-react/src/utils/resolve-user-id.ts (1)
1-29: Excellent utility function for user ID migration.The implementation correctly handles all migration scenarios:
- ✅ Prioritizes
externalUserIdwhen both are provided- ✅ Falls back to deprecated
userIdwith appropriate warning- ✅ Handles empty string edge cases
- ✅ Provides clear error when neither ID is supplied
- ✅ Well-documented with JSDoc
- ✅ Type-safe return with warning information
This is a solid foundation for the user ID migration across the package.
packages/connect-react/src/components/ComponentForm.tsx (2)
13-20: Good deprecation documentation.The JSDoc clearly marks
userIdas deprecated and directs users to useexternalUserIdinstead.
32-35: Excellent use of TypeScript union types for migration.The union type correctly enforces that exactly one of
externalUserIdoruserIdmust be provided, preventing invalid combinations while maintaining backward compatibility. Thenevertype effectively prevents simultaneous usage.packages/connect-react/src/hooks/form-context.tsx (3)
43-45: Good type structure for migration.Adding
externalUserIdwhile markinguserIdas deprecated with proper JSDoc documentation helps guide users through the migration.
87-105: Excellent deprecation warning implementation.The warning logic is well-designed:
- ✅ Uses
useMemofor efficient ID resolution- ✅ Handles warnings in
useEffectto avoid render side effects- ✅ Only shows warnings in development mode
- ✅ Provides clear, actionable warning messages
- ✅ Distinguishes between "both provided" and "deprecated usage" scenarios
This approach provides excellent developer experience during migration.
577-578: Smart backward compatibility approach.Exposing
userIdin the context set to the resolved external user ID maintains backward compatibility while ensuring all internal logic uses the new identifier. This allows existing consumers to continue working without breaking changes.components/habitica/package.json (3)
3-3: LGTM: Appropriate version bump for new functionality.The minor version increment from 0.0.2 to 0.1.0 is appropriate for adding new challenge management actions.
5-5: LGTM: Main entry point updated correctly.The main entry point change from the dist directory to direct source file aligns with the new file structure.
15-16: LGTM: Required dependency added.The
@pipedream/platformdependency is necessary for the axios import used in the app module.components/habitica/actions/create-challenge/create-challenge.mjs (4)
1-1: LGTM: Correct import path.The relative import path correctly references the app module.
4-8: LGTM: Standard Pipedream action structure.The action follows the correct structure with appropriate key naming convention, descriptive name, and proper version.
17-25: LGTM: Dynamic prop definition with context.The group prop correctly uses dynamic options based on the type selection, enhancing user experience.
63-78: LGTM: Proper async implementation with error handling.The run method correctly passes the $ parameter for error handling and provides meaningful feedback via export summary.
components/habitica/actions/delete-challenge/delete-challenge.mjs (3)
1-1: LGTM: Correct import path.The relative import path correctly references the app module.
4-8: LGTM: Standard Pipedream action structure.The action follows the correct structure with appropriate key naming convention and descriptive documentation link.
18-25: LGTM: Proper delete implementation with feedback.The delete operation correctly passes the challengeId and provides meaningful success feedback.
components/habitica/habitica.app.mjs (3)
1-2: LGTM: Correct imports.The imports correctly reference the Pipedream platform and local constants.
89-105: LGTM: Well-structured HTTP client method.The _makeRequest method properly handles authentication headers and provides a clean interface for API calls.
107-147: LGTM: Comprehensive CRUD methods.All the API methods follow consistent patterns and properly destructure parameters. The REST endpoints are correctly structured.
components/elmah_io/elmah_io.app.mjs (3)
1-3: LGTM: Appropriate import addition for error handling.Adding
ConfigurationErrorto the imports supports the enhanced error handling implemented in the_makeRequestmethod.
11-11: LGTM: Helpful clarification of permission requirements.Adding the specific permission requirement (
logs_read) improves user experience by clearly communicating API key requirements.
16-19: LGTM: Robust defensive programming with optional chaining.The optional chaining with fallback to an empty array prevents runtime errors when
getLogs()returns undefined or null.components/elmah_io/sources/new-error/new-error.mjs (8)
5-5: LGTM: Simplified query for better component control.Removing the "isNew:true" condition allows the component to manage deduplication internally through its state tracking mechanism.
9-9: LGTM: Standard version increment.Version bump from 0.0.3 to 0.0.4 follows standard practices for component updates.
11-11: LGTM: Improved documentation with API reference.Adding the documentation link enhances developer experience by providing direct access to API specifications.
29-33: LGTM: Excellent UX improvement with permission alert.The informational alert clearly communicates required API key permissions, preventing configuration issues and user confusion.
39-39: LGTM: Minor clarity improvement.The summary text provides clearer event identification.
47-47: LGTM: Explicit return for clarity.Adding an explicit return statement improves code readability and makes the intent clear.
60-62: LGTM: Proper defensive programming with early returns.The early returns when no messages are found prevent unnecessary processing and potential errors with empty arrays.
Also applies to: 85-87
65-65: LGTM: Improved state management and efficiency.The changes enhance the component by:
- Storing
lastEventDatetimein a variable for efficiency- Setting the last event datetime after processing to ensure state consistency
- Using proper conditional logic for the
fromparameterAlso applies to: 70-70, 79-81, 90-90
components/leaddyno/leaddyno.app.mjs (3)
1-11: Verify PR context - file doesn't match PR objectives.This file appears to be for LeadDyno component, but the PR objectives indicate this should be about "[components] Scrapeless - add new actions." Please confirm this is the correct file for review.
Likely an incorrect or invalid review comment.
3-3: ```shell
#!/bin/bashSearch for any remaining references to "habitica" across .js, .mjs, and .json files
rg "habitica" -g ".js" -g ".mjs" -g "*.json" -A 2 -B 2
--- `1-1`: ```bash #!/usr/bin/env bash set -e echo "🔍 Searching for 'defineApp' references in .js/.mjs files..." grep -R -n --include \*.js --include \*.mjs "defineApp" . || echo "✅ No 'defineApp' occurrences found." echo echo "🔍 Searching for plain object default exports ('export default {') in .js/.mjs files..." grep -R -n --include \*.js --include \*.mjs -E "export default[[:space:]]*\\{" . || echo "✅ No default object exports found."components/quickbooks/common/utils.mjs (1)
17-46: LGTM! Well-implemented recursive parsing function.The
parseObjectfunction correctly handles recursive parsing of nested structures including arrays and objects, with proper error handling for invalid JSON.components/quickbooks/quickbooks.app.mjs (4)
12-12: LGTM! Good correction to the description.The description fix improves clarity for the
invoiceIdproperty.
119-154: LGTM! Well-implemented property definitions.The new
estimateIdandvendorproperty definitions follow the established patterns and include appropriate async options with proper mapping functions.
476-547: LGTM! Comprehensive address property definitions.The billing and shipping address properties are well-structured and follow consistent naming conventions. All properties are appropriately marked as optional.
630-699: LGTM! Consistent method implementations.The new methods follow the established pattern of using
_makeRequestwith appropriate paths and HTTP methods. The implementation is consistent with existing methods in the codebase.components/quickbooks/actions/void-invoice/void-invoice.mjs (1)
1-55: LGTM! Well-implemented void invoice action.The implementation correctly:
- Retrieves the invoice first to get the required SyncToken
- Handles errors appropriately for missing invoice or SyncToken
- Follows the QuickBooks API requirements for voiding invoices
- Provides clear success feedback
The logic flow and error handling are solid.
components/quickbooks/actions/update-estimate/update-estimate.mjs (1)
212-228: Well-designed helper method for conditional property addition.The
addIfDefinedmethod effectively reduces code duplication and provides flexibility with transformation support.components/quickbooks/actions/update-invoice/update-invoice.mjs (1)
236-256: Excellent validation and error handling for invoice retrieval.This validation pattern with clear error messages is exemplary and should be adopted by other update actions in this codebase.
components/quickbooks/actions/create-estimate/create-estimate.mjs
Outdated
Show resolved
Hide resolved
components/quickbooks/actions/create-purchase-order/create-purchase-order.mjs
Outdated
Show resolved
Hide resolved
components/quickbooks/actions/create-purchase-order/create-purchase-order.mjs
Outdated
Show resolved
Hide resolved
components/quickbooks/actions/update-estimate/update-estimate.mjs
Outdated
Show resolved
Hide resolved
components/quickbooks/actions/update-invoice/update-invoice.mjs
Outdated
Show resolved
Hide resolved
joy-chanboop
force-pushed
the
feature/scrapeless-new-actions
branch
3 times, most recently
from
71cd30a to
fa1e7c7
Compare
Hi Jorge Cortes, I’ve rebased my branch onto the latest master, but it seems that my PR history has been polluted with unrelated commits from other contributors. I’m concerned this might make the PR harder to review or merge cleanly. Would you mind helping with the merge directly on GitHub? I really appreciate your support—thank you! Best regards |
vunguyenhung
moved this from Ready for QA
to Changes Required
in Component (Source and Action) Backlog
|
Hello everyone, I have tested this PR and there're some test cases failed or needed improvement. Please check the test report below for more information |
WHY
Crawler,Scraping API, andUniversal Scraping API.Summary by CodeRabbit