Request and response format
Information about supported parameters and format of API request.
Authentication
You can find your API token on the dashboard page in the user's admin panel.
Header-based authentication
The API request can be authenticated using the x-api-key header:
curl --request GET \
--url 'https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com' \
--header 'x-api-key: <YOUR_SCRAPINGANT_API_KEY>'
API parameter-based authentication
The other authentication option is to use the x-api-key API parameter:
curl 'https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com&x-api-key=<YOUR_SCRAPINGANT_API_KEY>'
Request format
The web scraping endpoint is available at the following URL:
https://api.scrapingant.com/v1/general
It supports both GET and POST methods.
Passing parameters via GET method
Parameters for GET method should be passed as a query string parameters in urlencoded format.
Example below shows how will look URL with scraping request to example.com:
https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com
How to encode query parameters in different programming languages:
- Go: url.QueryEscape
- Java: URLEncoder.encode
- NodeJS: encodeURIComponent
- PHP: urlencode
- Python: urllib.parse.quote
- Ruby: URI::escape
Passing parameters via POST method
To pass parameters via POST method you just have to place them into JSON body, like the following:
{
"url": "https://example.com"
}
Available parameters
Below you can find the full list of the available parameters:
| Parameter | Required | Type | Description |
|---|---|---|---|
url | required | String | URL to scrape |
x-api-key | optional | String | Your ScrapingAnt API key. The field is optional only when the x-api-key header is set. Otherwise it is required for authentiation in ScrapingAnt API |
browser | optional | Boolean | Enables using headless browser for scraping. Default: true. See Headless browser for more info. |
cookies | optional | String | Cookies to pass with a scraping request to the target site. See Custom cookies for more info. |
js_snippet | optional | String | Base64 encoded JS snippet to run once page being loaded in the ScrapingAnt browser. This feature works only with browser=true. See more at Javascript execution. |
proxy_type | optional | String | Specifies proxy type to make request from. Default: datacenter. See Proxy settings for more info. |
proxy_country | optional | String | Specifies proxy country to make request from. See Proxy settings for more info. |
wait_for_selector | optional | String | The CSS selector of the element our service will wait for before returning result. This feature works only with browser=true. See Wait for CSS selector for more info. |
return_text | optional | Boolean | Enables returning text only content from the page instead of full HTML. Default: false |
Response structure
ScrapingAnt web scraping API endpoint returns a JSON object with the content, cookies and status_code properties:
{
"content": "<html><body><h1>Hello, World!</h1></body></html>",
"cookies": "cookie_name_1=cookie_value_1;cookie_name_2=cookie_value_2",
"status_code": 200
}
| Parameter | Type | Description |
|---|---|---|
content | String | Content from the scraped web page |
cookies | String | Response cookies from the scraped web page |
status_code | Number | Status code received from the scraped page |
In case of an error, the response has the HTTP status code in the range of 4xx or 5xx and the response properties will be replaced with detail property.
For example:
{
"detail": "Human readable error description"
}
See Errors for more details.