Skip to content

Files

Latest commit

DPendeDPende
doc: update readme
Feb 4, 2025
e72de2e · Feb 4, 2025

History

History
281 lines (207 loc) · 6.76 KB

README.md

File metadata and controls

281 lines (207 loc) · 6.76 KB

🌐 ScrapeGraph JavaScript SDK

npm version License Documentation Status

ScrapeGraph API Banner

Official JavaScript/TypeScript SDK for the ScrapeGraph AI API - Smart web scraping powered by AI.

🚀 Features

  • ✨ Smart web scraping with AI
  • 🔄 Fully asynchronous design
  • 🔍 Detailed error handling
  • ⚡ Automatic retries and logging
  • 🔐 Secure API authentication

📦 Installation

Install the package using npm or yarn:

# Using npm
npm i scrapegraph-js

# Using yarn
yarn add scrapegraph-js

🔧 Quick Start

Note: Store your API keys securely in environment variables. Use .env files and libraries like dotenv to load them into your app.

Basic Example

import { smartScraper } from 'scrapegraph-js';
import 'dotenv/config';

// Initialize variables
const apiKey = process.env.SGAI_APIKEY; // Set your API key as an environment variable
const websiteUrl = 'https://example.com';
const prompt = 'What does the company do?';

(async () => {
  try {
    const response = await smartScraper(apiKey, websiteUrl, prompt);
    console.log(response.result);
  } catch (error) {
    console.error('Error:', error);
  }
})();

🎯 Examples

Scraping Websites

Basic Scraping

import { smartScraper } from 'scrapegraph-js';

const apiKey = 'your-api-key';
const url = 'https://example.com';
const prompt = 'Extract the main heading and description.';

(async () => {
  try {
    const response = await smartScraper(apiKey, url, prompt);
    console.log(response.result);
  } catch (error) {
    console.error('Error:', error);
  }
})();

Scraping with Custom Output Schema

Note

To use this feature, it is necessary to employ the Zod package for schema creation.

Here is a real-world example:

import { smartScraper } from 'scrapegraph-js';
import { z } from 'zod';

const apiKey = 'your-api-key';
const url = 'https://scrapegraphai.com/';
const prompt = 'What does the company do? and ';

const schema = z.object({
  title: z.string().describe('The title of the webpage'),
  description: z.string().describe('The description of the webpage'),
  summary: z.string().describe('A brief summary of the webpage'),
});

(async () => {
  try {
    const response = await smartScraper(apiKey, url, prompt, schema);
    console.log(response.result);
  } catch (error) {
    console.error('Error:', error);
  }
})();

Search Scraping

Search and extract information from multiple web sources using AI.

import { searchScraper } from 'scrapegraph-js';

const apiKey = 'your-api-key';
const prompt = 'What is the latest version of Python and what are its main features?';

(async () => {
  try {
    const response = await searchScraper(apiKey, prompt);
    console.log(response.result);
  } catch (error) {
    console.error('Error:', error);
  }
})();

Scraping local HTML

Extract structured data from local HTML content

import { localScraper } from 'scrapegraph-js';

const apiKey = 'your_api_key';
const prompt = 'What does the company do?';

const websiteHtml = `<html>
                      <body>
                        <h1>Company Name</h1>
                        <p>We are a technology company focused on AI solutions.</p>
                        <div class="contact">
                          <p>Email: contact@example.com</p>
                        </div>
                      </body>
                    </html>`;
(async () => {
  try {
    const response = await localScraper(apiKey, websiteHtml, prompt);
    console.log(response);
  } catch (error) {
    console.error(error);
  }
})();

Markdownify

Converts a webpage into clean, well-structured markdown format.

import { smartScraper } from 'scrapegraph-js';

const apiKey = 'your_api_key';
const url = 'https://scrapegraphai.com/';

(async () => {
  try {
    const response = await markdownify(apiKey, url);
    console.log(response);
  } catch (error) {
    console.error(error);
  }
})();

Checking API Credits

import { getCredits } from 'scrapegraph-js';

const apiKey = 'your-api-key';

(async () => {
  try {
    const credits = await getCredits(apiKey);
    console.log('Available credits:', credits);
  } catch (error) {
    console.error('Error fetching credits:', error);
  }
})();

Submitting Feedback

import { sendFeedback } from 'scrapegraph-js';

const apiKey = 'your-api-key';
const requestId = '16a63a80-c87f-4cde-b005-e6c3ecda278b';
const rating = 5;
const feedbackText = 'This is a test feedback message.';

(async () => {
  try {
    const response = await sendFeedback(apiKey, requestId, rating, feedbackText);
    console.log('Feedback response:', response);
  } catch (error) {
    console.error('Error sending feedback:', error);
  }
})();

📚 Documentation

For detailed documentation, visit docs.scrapegraphai.com

🛠️ Development

Setup

  1. Clone the repository:

    git clone https://github.com/ScrapeGraphAI/scrapegraph-sdk.git
cd scrapegraph-sdk/scrapegraph-js

  2. Install dependencies:

    npm install

  3. Run linting and testing:

    npm run lint
npm test

Running Tests

# Run all tests
npm test

# Run tests with coverage
npm run test:coverage

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add some AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

🔗 Links

💬 Support

Made with ❤️ by ScrapeGraph AI