Request and response format
Information about supported parameters and format of API request.
#
AuthenticationYou can find your API token on the dashboard page in the user's admin panel.
#
Header-based authenticationThe API request can be authenticated using the x-api-key
header:
#
API parameter-based authenticationThe other authentication option is to use the x-api-key
API parameter:
#
Request formatThe web scraping endpoint is available at the following URL:
It supports both GET
and POST
methods.
#
Passing parameters via GET methodParameters for GET
method should be passed as a query string parameters in urlencoded format.
Example below shows how will look URL with scraping request to example.com
:
#
How to encode query parameters in different programming languages:- Go: url.QueryEscape
- Java: URLEncoder.encode
- NodeJS: encodeURIComponent
- PHP: urlencode
- Python: urllib.parse.quote
- Ruby: URI::escape
#
Passing parameters via POST methodTo pass parameters via POST method you just have to place them into JSON body, like the following:
#
Available parametersBelow you can find the full list of the available parameters:
Parameter | Required | Type | Description |
---|---|---|---|
url | required | String | URL to scrape |
x-api-key | optional | String | Your ScrapingAnt API key. The field is optional only when the x-api-key header is set. Otherwise it is required for authentiation in ScrapingAnt API |
browser | optional | Boolean | Enables using headless browser for scraping. Default: true . See Headless browser for more info. |
cookies | optional | String | Cookies to pass with a scraping request to the target site. See Custom cookies for more info. |
js_snippet | optional | String | Base64 encoded JS snippet to run once page being loaded in the ScrapingAnt browser. This feature works only with browser=true . See more at Javascript execution. |
proxy_type | optional | String | Specifies proxy type to make request from. Default: datacenter . See Proxy settings for more info. |
proxy_country | optional | String | Specifies proxy country to make request from. See Proxy settings for more info. |
wait_for_selector | optional | String | The CSS selector of the element our service will wait for before returning result. This feature works only with browser=true . See Wait for CSS selector for more info. |
return_text | optional | Boolean | Enables returning text only content from the page instead of full HTML. Default: false |
#
Response structureScrapingAnt web scraping API endpoint returns a JSON object with the content
, cookies
and status_code
properties:
Parameter | Type | Description |
---|---|---|
content | String | Content from the scraped web page |
cookies | String | Response cookies from the scraped web page |
status_code | Number | Status code received from the scraped page |
In case of an error, the response has the HTTP status code in the range of 4xx
or 5xx
and the response properties will be replaced with detail
property.
For example:
See Errors for more details.