|
1 | | - |
| 1 | +# n8n-nodes-scraperapi |
2 | 2 |
|
3 | | -# n8n-nodes-starter |
| 3 | +This is an n8n community node. It lets you use _ScraperAPI_ in your n8n workflows. |
4 | 4 |
|
5 | | -This starter repository helps you build custom integrations for [n8n](https://n8n.io). It includes example nodes, credentials, the node linter, and all the tooling you need to get started. |
| 5 | +_ScraperAPI_ is a solution to help you unlock and scrape any website, no matter the scale or difficulty. |
6 | 6 |
|
7 | | -## Quick Start |
| 7 | +[n8n](https://n8n.io/) is a [fair-code licensed](https://docs.n8n.io/reference/license/) workflow automation platform. |
8 | 8 |
|
9 | | -> [!TIP] |
10 | | -> **New to building n8n nodes?** The fastest way to get started is with `npm create @n8n/node`. This command scaffolds a complete node package for you using the [@n8n/node-cli](https://www.npmjs.com/package/@n8n/node-cli). |
| 9 | +[Installation](#installation) |
| 10 | +[Operations](#operations) |
| 11 | +[Credentials](#credentials) <!-- delete if no auth needed --> |
| 12 | +[Compatibility](#compatibility) |
| 13 | +[Usage](#usage) <!-- delete if not using this section --> |
| 14 | +[Resources](#resources) |
| 15 | +[Version history](#version-history) <!-- delete if not using this section --> |
11 | 16 |
|
12 | | -**To create a new node package from scratch:** |
| 17 | +## Installation |
13 | 18 |
|
14 | | -```bash |
15 | | -npm create @n8n/node |
16 | | -``` |
| 19 | +Follow the [installation guide](https://docs.n8n.io/integrations/community-nodes/installation/) in the n8n community nodes documentation. |
17 | 20 |
|
18 | | -**Already using this starter? Start developing with:** |
19 | | - |
20 | | -```bash |
21 | | -npm run dev |
22 | | -``` |
23 | | - |
24 | | -This starts n8n with your nodes loaded and hot reload enabled. |
25 | | - |
26 | | -## What's Included |
27 | | - |
28 | | -This starter repository includes two example nodes to learn from: |
29 | | - |
30 | | -- **[Example Node](nodes/Example/)** - A simple starter node that shows the basic structure with a custom `execute` method |
31 | | -- **[GitHub Issues Node](nodes/GithubIssues/)** - A complete, production-ready example built using the **declarative style**: |
32 | | - - **Low-code approach** - Define operations declaratively without writing request logic |
33 | | - - Multiple resources (Issues, Comments) |
34 | | - - Multiple operations (Get, Get All, Create) |
35 | | - - Two authentication methods (OAuth2 and Personal Access Token) |
36 | | - - List search functionality for dynamic dropdowns |
37 | | - - Proper error handling and typing |
38 | | - - Ideal for HTTP API-based integrations |
39 | | - |
40 | | -> [!TIP] |
41 | | -> The declarative/low-code style (used in GitHub Issues) is the recommended approach for building nodes that interact with HTTP APIs. It significantly reduces boilerplate code and handles requests automatically. |
42 | | -
|
43 | | -Browse these examples to understand both approaches, then modify them or create your own. |
44 | | - |
45 | | -## Finding Inspiration |
46 | | - |
47 | | -Looking for more examples? Check out these resources: |
48 | | - |
49 | | -- **[npm Community Nodes](https://www.npmjs.com/search?q=keywords:n8n-community-node-package)** - Browse thousands of community-built nodes on npm using the `n8n-community-node-package` tag |
50 | | -- **[n8n Built-in Nodes](https://github.com/n8n-io/n8n/tree/master/packages/nodes-base/nodes)** - Study the source code of n8n's official nodes for production-ready patterns and best practices |
51 | | -- **[n8n Credentials](https://github.com/n8n-io/n8n/tree/master/packages/nodes-base/credentials)** - See how authentication is implemented for various services |
52 | | - |
53 | | -These are excellent resources to understand how to structure your nodes, handle different API patterns, and implement advanced features. |
54 | | - |
55 | | -## Prerequisites |
56 | | - |
57 | | -Before you begin, install the following on your development machine: |
58 | | - |
59 | | -### Required |
60 | | - |
61 | | -- **[Node.js](https://nodejs.org/)** (v22 or higher) and npm |
62 | | - - Linux/Mac/WSL: Install via [nvm](https://github.com/nvm-sh/nvm) |
63 | | - - Windows: Follow [Microsoft's NodeJS guide](https://learn.microsoft.com/en-us/windows/dev-environment/javascript/nodejs-on-windows) |
64 | | -- **[git](https://git-scm.com/downloads)** |
65 | | - |
66 | | -### Recommended |
67 | | - |
68 | | -- Follow n8n's [development environment setup guide](https://docs.n8n.io/integrations/creating-nodes/build/node-development-environment/) |
69 | | - |
70 | | -> [!NOTE] |
71 | | -> The `@n8n/node-cli` is included as a dev dependency and will be installed automatically when you run `npm install`. The CLI includes n8n for local development, so you don't need to install n8n globally. |
72 | | -
|
73 | | -## Getting Started with this Starter |
74 | | - |
75 | | -Follow these steps to create your own n8n community node package: |
76 | | - |
77 | | -### 1. Create Your Repository |
78 | | - |
79 | | -[Generate a new repository](https://github.com/n8n-io/n8n-nodes-starter/generate) from this template, then clone it: |
80 | | - |
81 | | -```bash |
82 | | -git clone https://github.com/<your-organization>/<your-repo-name>.git |
83 | | -cd <your-repo-name> |
84 | | -``` |
85 | | - |
86 | | -### 2. Install Dependencies |
87 | | - |
88 | | -```bash |
89 | | -npm install |
90 | | -``` |
91 | | - |
92 | | -This installs all required dependencies including the `@n8n/node-cli`. |
93 | | - |
94 | | -### 3. Explore the Examples |
95 | | - |
96 | | -Browse the example nodes in [nodes/](nodes/) and [credentials/](credentials/) to understand the structure: |
97 | | - |
98 | | -- Start with [nodes/Example/](nodes/Example/) for a basic node |
99 | | -- Study [nodes/GithubIssues/](nodes/GithubIssues/) for a real-world implementation |
100 | | - |
101 | | -### 4. Build Your Node |
102 | | - |
103 | | -Edit the example nodes to fit your use case, or create new node files by copying the structure from [nodes/Example/](nodes/Example/). |
104 | | - |
105 | | -> [!TIP] |
106 | | -> If you want to scaffold a completely new node package, use `npm create @n8n/node` to start fresh with the CLI's interactive generator. |
107 | | -
|
108 | | -### 5. Configure Your Package |
109 | | - |
110 | | -Update `package.json` with your details: |
111 | | - |
112 | | -- `name` - Your package name (must start with `n8n-nodes-`) |
113 | | -- `author` - Your name and email |
114 | | -- `repository` - Your repository URL |
115 | | -- `description` - What your node does |
116 | | - |
117 | | -Make sure your node is registered in the `n8n.nodes` array. |
118 | | - |
119 | | -### 6. Develop and Test Locally |
120 | | - |
121 | | -Start n8n with your node loaded: |
122 | | - |
123 | | -```bash |
124 | | -npm run dev |
125 | | -``` |
126 | | - |
127 | | -This command runs `n8n-node dev` which: |
128 | | - |
129 | | -- Builds your node with watch mode |
130 | | -- Starts n8n with your node available |
131 | | -- Automatically rebuilds when you make changes |
132 | | -- Opens n8n in your browser (usually http://localhost:5678) |
133 | | - |
134 | | -You can now test your node in n8n workflows! |
135 | | - |
136 | | -> [!NOTE] |
137 | | -> Learn more about CLI commands in the [@n8n/node-cli documentation](https://www.npmjs.com/package/@n8n/node-cli). |
138 | | -
|
139 | | -### 7. Lint Your Code |
140 | | - |
141 | | -Check for errors: |
142 | | - |
143 | | -```bash |
144 | | -npm run lint |
145 | | -``` |
146 | | - |
147 | | -Auto-fix issues when possible: |
148 | | - |
149 | | -```bash |
150 | | -npm run lint:fix |
151 | | -``` |
152 | | - |
153 | | -### 8. Build for Production |
154 | | - |
155 | | -When ready to publish: |
156 | | - |
157 | | -```bash |
158 | | -npm run build |
159 | | -``` |
160 | | - |
161 | | -This compiles your TypeScript code to the `dist/` folder. |
162 | | - |
163 | | -### 9. Prepare for Publishing |
164 | | - |
165 | | -Before publishing: |
166 | | - |
167 | | -1. **Update documentation**: Replace this README with your node's documentation. Use [README_TEMPLATE.md](README_TEMPLATE.md) as a starting point. |
168 | | -2. **Update the LICENSE**: Add your details to the [LICENSE](LICENSE.md) file. |
169 | | -3. **Test thoroughly**: Ensure your node works in different scenarios. |
170 | | - |
171 | | -### 10. Publish to npm |
172 | | - |
173 | | -Publish your package to make it available to the n8n community: |
174 | | - |
175 | | -```bash |
176 | | -npm publish |
177 | | -``` |
178 | | - |
179 | | -Learn more about [publishing to npm](https://docs.npmjs.com/packages-and-modules/contributing-packages-to-the-registry). |
180 | | - |
181 | | -### 11. Submit for Verification (Optional) |
182 | | - |
183 | | -Get your node verified for n8n Cloud: |
184 | | - |
185 | | -1. Ensure your node meets the [requirements](https://docs.n8n.io/integrations/creating-nodes/deploy/submit-community-nodes/): |
186 | | - - Uses MIT license ✅ (included in this starter) |
187 | | - - No external package dependencies |
188 | | - - Follows n8n's design guidelines |
189 | | - - Passes quality and security review |
190 | | - |
191 | | -2. Submit through the [n8n Creator Portal](https://creators.n8n.io/nodes) |
192 | | - |
193 | | -**Benefits of verification:** |
194 | | - |
195 | | -- Available directly in n8n Cloud |
196 | | -- Discoverable in the n8n nodes panel |
197 | | -- Verified badge for quality assurance |
198 | | -- Increased visibility in the n8n community |
199 | | - |
200 | | -## Available Scripts |
201 | | - |
202 | | -This starter includes several npm scripts to streamline development: |
203 | | - |
204 | | -| Script | Description | |
205 | | -| --------------------- | ---------------------------------------------------------------- | |
206 | | -| `npm run dev` | Start n8n with your node and watch for changes (runs `n8n-node dev`) | |
207 | | -| `npm run build` | Compile TypeScript to JavaScript for production (runs `n8n-node build`) | |
208 | | -| `npm run build:watch` | Build in watch mode (auto-rebuild on changes) | |
209 | | -| `npm run lint` | Check your code for errors and style issues (runs `n8n-node lint`) | |
210 | | -| `npm run lint:fix` | Automatically fix linting issues when possible (runs `n8n-node lint --fix`) | |
211 | | -| `npm run release` | Create a new release (runs `n8n-node release`) | |
212 | | - |
213 | | -> [!TIP] |
214 | | -> These scripts use the [@n8n/node-cli](https://www.npmjs.com/package/@n8n/node-cli) under the hood. You can also run CLI commands directly, e.g., `npx n8n-node dev`. |
215 | | -
|
216 | | -## Troubleshooting |
217 | | - |
218 | | -### My node doesn't appear in n8n |
219 | | - |
220 | | -1. Make sure you ran `npm install` to install dependencies |
221 | | -2. Check that your node is listed in `package.json` under `n8n.nodes` |
222 | | -3. Restart the dev server with `npm run dev` |
223 | | -4. Check the console for any error messages |
224 | | - |
225 | | -### Linting errors |
226 | | - |
227 | | -Run `npm run lint:fix` to automatically fix most common issues. For remaining errors, check the [n8n node development guidelines](https://docs.n8n.io/integrations/creating-nodes/). |
| 21 | +## Resources |
228 | 22 |
|
229 | | -### TypeScript errors |
| 23 | +- API Endpoint: use ScraperAPI's endpoint for web scraping. Configure parameters and send GET requests enabling JS rendering, setting geotargeting, etc. |
230 | 24 |
|
231 | | -Make sure you're using Node.js v22 or higher and have run `npm install` to get all type definitions. |
232 | 25 |
|
233 | | -## Resources |
| 26 | +## Credentials |
234 | 27 |
|
235 | | -- **[n8n Node Documentation](https://docs.n8n.io/integrations/creating-nodes/)** - Complete guide to building nodes |
236 | | -- **[n8n Community Forum](https://community.n8n.io/)** - Get help and share your nodes |
237 | | -- **[@n8n/node-cli Documentation](https://www.npmjs.com/package/@n8n/node-cli)** - CLI tool reference |
238 | | -- **[n8n Creator Portal](https://creators.n8n.io/nodes)** - Submit your node for verification |
239 | | -- **[Submit Community Nodes Guide](https://docs.n8n.io/integrations/creating-nodes/deploy/submit-community-nodes/)** - Verification requirements and process |
| 28 | +Sign up and get your API key [here](https://dashboard.scraperapi.com/signup) |
240 | 29 |
|
241 | | -## Contributing |
| 30 | +## Version history |
242 | 31 |
|
243 | | -Have suggestions for improving this starter? [Open an issue](https://github.com/n8n-io/n8n-nodes-starter/issues) or submit a pull request! |
| 32 | +- "0.1.0": API resource. |
244 | 33 |
|
245 | | -## License |
246 | 34 |
|
247 | | -[MIT](https://github.com/n8n-io/n8n-nodes-starter/blob/master/LICENSE.md) |
0 commit comments