Skip to main content

Scrape Loop API Overview

Introduction​

The Scrape Loop API allows you to programmatically create and manage web scraping recipes, run scraping jobs, and retrieve scraped data. This guide will help you get started with the API.

Base URL​

All API requests should be made to:

https://api.scrapeloop.com/v1

Authentication​

All API endpoints require authentication. You can authenticate your requests in two ways:

Add your API key to the request headers:

curl -X GET "https://api.scrapeloop.com/v1/recipes" \
  -H "x-api-key: YOUR_API_KEY"

Response Format​

All API responses follow a consistent JSON format:

{
  "success": true,
  "data": {
    // Response data
  }
}

Core Concepts​

1. Recipes​

Recipes define what data to scrape and how to scrape it:

  • Target URL and selectors
  • Data extraction rules
  • Pagination settings
  • Execution configuration

2. Jobs​

Jobs are individual executions of recipes:

  • Track scraping progress
  • Store results
  • Handle errors and retries
  • Manage rate limiting

3. Results​

Scraped data is returned in a structured format:

  • JSON formatted
  • Paginated responses
  • Filterable and sortable
  • Downloadable in various formats

Quick Start​

1. Create a Recipe​

POST
/recipes
curl -X POST "https://api.scrapeloop.com/v1/recipes" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My First Recipe",
    "url": "https://example.com/products",
    "listSelector": ".product-list .product-item"
  }'

2. Run a Job​

POST
/recipes/{id}/jobs
curl -X POST "https://api.scrapeloop.com/v1/recipes/recipe_123/jobs" \
  -H "x-api-key: YOUR_API_KEY"

3. Get Results​

GET
/recipes/{id}/jobs/{jobId}/results
curl -X GET "https://api.scrapeloop.com/v1/recipes/recipe_123/jobs/job_456/results" \
  -H "x-api-key: YOUR_API_KEY"

Support​