Request and response format
Information about supported parameters and format of API request.
Authentication
You can find your API token on the dashboard page in the user's admin panel.
API parameter-based authentication
The API request can be authenticated using x-api-key query parameter:
curl 'https://api.scrapingant.com/v2/general?url=https%3A%2F%2Fexample.com&x-api-key=<YOUR_SCRAPINGANT_API_KEY>'
Request format
The web scraping endpoint is available at the following URL:
https://api.scrapingant.com/v2/general
Passing parameters via GET method
Parameters should be provided as a query string parameters in urlencoded format.
Example below shows how will look URL with the scraping request to example.com:
https://api.scrapingant.com/v2/general?url=https%3A%2F%2Fexample.com
How to encode query parameters in different programming languages:
- Go: url.QueryEscape
- Java: URLEncoder.encode
- NodeJS: encodeURIComponent
- PHP: urlencode
- Python: urllib.parse.quote
- Ruby: URI::escape
Available parameters
Below you can find the full list of the available parameters:
| Parameter | Required | Type | Description |
|---|---|---|---|
url | required | String | URL to scrape |
x-api-key | required | String | Your ScrapingAnt API key. |
browser | optional | Boolean | Enables using headless browser for scraping. Default: true. See Headless browser for more info. |
cookies | optional | String | Cookies to pass with a scraping request to the target site. See Custom cookies for more info. |
js_snippet | optional | String | Base64 encoded JS snippet to run once page being loaded in the ScrapingAnt browser. This feature works only with browser=true. See more at Javascript execution. |
proxy_type | optional | String | Specifies proxy type to make request from. Default: datacenter. See Proxy settings for more info. |
proxy_country | optional | String | Specifies proxy country to make request from. See Proxy settings for more info. |
wait_for_selector | optional | String | The CSS selector of the element our service will wait for before returning result. This feature works only with browser=true. See Wait for CSS selector for more info. |
return_page_source | optional | Boolean | Enables returning HTML returned by the server and unaltered by the browser. Default: false. This feature works only with browser=true. |
Response structure
ScrapingAnt web scraping API endpoint returns a JSON object with the content, cookies and status_code properties:
{
"content": "<html><body><h1>Hello, World!</h1></body></html>",
"cookies": "cookie_name_1=cookie_value_1;cookie_name_2=cookie_value_2",
"status_code": 200
}
| Parameter | Type | Description |
|---|---|---|
content | String | Content from the scraped web page |
cookies | String | Response cookies from the scraped web page |
status_code | Number | Status code received from the scraped page |
In case of an error, the response has the HTTP status code in the range of 4xx or 5xx and the response properties will be replaced with detail property.
For example:
{
"detail": "Human readable error description"
}
See Errors for more details.