Request and response format
Information about supported parameters and format of API request.
Swagger
You can find the full API documentation in the Swagger format.
Authentication
All the requests to web scraping API should be authenticated using the API key.
You can find your API token on the dashboard page in the user's admin panel.
API parameter-based authentication
The API request can be authenticated using x-api-key query parameter:
curl 'https://api.scrapingant.com/v2/general?url=https%3A%2F%2Fexample.com&x-api-key=<YOUR_SCRAPINGANT_API_KEY>'
Request format
The web scraping endpoint is available at the following URL:
https://api.scrapingant.com/v2/general
Passing parameters via GET method
Parameters should be provided as a query string parameters in urlencoded format.
Example below shows how will look URL with the scraping request to example.com:
https://api.scrapingant.com/v2/general?url=https%3A%2F%2Fexample.com
How to encode query parameters in different programming languages:
- Go: url.QueryEscape
- Java: URLEncoder.encode
- NodeJS: encodeURIComponent
- PHP: urlencode
- Python: urllib.parse.quote
- Ruby: URI::escape
Available parameters
Below you can find the full list of the available parameters:
| Parameter | Required | Type | Description |
|---|---|---|---|
url | required | String | URL to scrape |
x-api-key | required | String | Your ScrapingAnt API key. |
browser | optional | Boolean | Enables using headless browser for scraping. Default: true. See Headless browser for more info. |
return_page_source | optional | Boolean | Enables returning data returned by the server and unaltered by the browser. Default: false. When true - JS won't be rendered. This feature works only with browser=true. See Scraping without JS rendering for more info. |
cookies | optional | String | Cookies to pass with a scraping request to the target site. See Custom cookies for more info. |
js_snippet | optional | String | Base64 encoded JS snippet to run once page being loaded in the ScrapingAnt browser. This feature works only with browser=true. See more at Javascript execution. |
proxy_type | optional | String | Specifies proxy type to make request from. Default: datacenter. See Proxy settings for more info. |
proxy_country | optional | String | Specifies proxy country to make request from. See Proxy settings for more info. |
wait_for_selector | optional | String | The CSS selector of the element our service will wait for before returning result. This feature works only with browser=true. See Wait for CSS selector for more info. |
block_resource | optional | String (repeated param) | Block resource types. Prevents cloud browser from loading specified resource types. Can be repeated in case several resource types should be blocked. This feature works only with browser=true. Available resource types: document, stylesheet, image, media, font, script, texttrack, xhr, fetch, eventsource, websocket, manifest, other |
Response structure
ScrapingAnt web scraping API endpoint returns plain html content of the scraped web page:
<html><body><h1>Hello, World!</h1></body></html>
In case of an error, the response has the HTTP status code in the range of 4xx or 5xx and the response properties will be replaced with detail property.
For example:
{
"detail": "Human readable error description"
}
If you are looking for JSON response format, please check ScrapingAnt's JSON response endpoint.
See Errors for more details.