API Basics

All you need to know for start using ScrapingAnt API.

Start using API#

The only 2 things needed to start working with ScrapingAnt API: URL you'd like to scrape, and the API key. The second one can be found at the user's dashboard.

API info#

ScrapingAnt Web Scraping API is available at the following endpoint:

https://api.scrapingant.com/v1/general

It supports both GET and POST methods, so every consumer is available to choose the most preferable one, as both methods do the same.

caution

POST method is only added for more comfortable API usage. It acts the same as GET endpoint, so only format is different.

The web scraping endpoint requires 2 parameters: x-api-key header with API key and url with the target URL to scrape.

url parameter should be passed as a query parameter when using GET method, and as a part of body JSON when using POST.

Below you can find example.com scraping requests with using of CURL as an executor.

GET request:

curl --request GET \
--url 'https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com' \
--header 'x-api-key: <YOUR-API-KEY>'

POST request:

curl --request POST \
--url 'https://api.scrapingant.com/v1/general' \
--header 'x-api-key: <YOUR-API-KEY>' \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '{ "url": "https://example.com"}'

Both requests will return the JSON result:

{
  "content":"<!DOCTYPE html><html><head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\">\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\">\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n    <style type=\"text/css\">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 2em;\n        background-color: #fdfdff;\n        border-radius: 0.5em;\n        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        div {\n            margin: 0 auto;\n            width: auto;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.</p>\n    <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n\n\n</body></html>",
  "cookies":""
}

To know more about optional parameters, and the response format, please, follow the Request and Response Format page.

Execution time#

Maximum page load time is restricted to 30 seconds, so each URL that failed will be tried as many times as possible for 30 seconds.

Generate API request via request generator#

The simplest way to generate required code snippet for starting consuming API is to use ScrapingAnt's request generator. You can find it at the user dashboard and just fill the form with required parameters to get code samples for CURL, Python and NodeJS. Also, the direct execution function is available, so you will be able to execute request right from the user's admin panel.