Request and response format

Information about supported parameters and format of API request.

Authentication#

You can find your API token on the dashboard page in the user's admin panel.

Header-based authentication#

The API request can be authenticated using the x-api-key header:

curl --request GET \
--url 'https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com' \
--header 'x-api-key: <YOUR_SCRAPINGANT_API_KEY>'

API parameter-based authentication#

The other authentication option is to use the x-api-key API parameter:

curl 'https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com&x-api-key=<YOUR_SCRAPINGANT_API_KEY>'

Request format#

The web scraping endpoint is available at the following URL:

https://api.scrapingant.com/v1/general

It supports both GET and POST methods.

Passing parameters via GET method#

Parameters for GET method should be passed as a query string parameters in urlencoded format.

Example below shows how will look URL with scraping request to example.com:

https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com

How to encode query parameters in different programming languages:#

Passing parameters via POST method#

To pass parameters via POST method you just have to place them into JSON body, like the following:

{
  "url": "https://example.com"
}

Available parameters#

Below you can find the full list of the available parameters:

Parameter	Required	Type	Description
`url`	required	String	URL to scrape
`x-api-key`	optional	String	Your ScrapingAnt API key. The field is optional only when the `x-api-key` header is set. Otherwise it is required for authentiation in ScrapingAnt API
`browser`	optional	Boolean	Enables using headless browser for scraping. Default: `true`. See Headless browser for more info.
`cookies`	optional	String	Cookies to pass with a scraping request to the target site. See Custom cookies for more info.
`js_snippet`	optional	String	Base64 encoded JS snippet to run once page being loaded in the ScrapingAnt browser. This feature works only with `browser=true`. See more at Javascript execution.
`proxy_type`	optional	String	Specifies proxy type to make request from. Default: `datacenter`. See Proxy settings for more info.
`proxy_country`	optional	String	Specifies proxy country to make request from. See Proxy settings for more info.
`wait_for_selector`	optional	String	The CSS selector of the element our service will wait for before returning result. This feature works only with `browser=true`. See Wait for CSS selector for more info.
`return_text`	optional	Boolean	Enables returning text only content from the page instead of full HTML. Default: `false`

Response structure#

ScrapingAnt web scraping API endpoint returns a JSON object with the content, cookies and status_code properties:

{
    "content": "<html><body><h1>Hello, World!</h1></body></html>",
    "cookies": "cookie_name_1=cookie_value_1;cookie_name_2=cookie_value_2",
    "status_code": 200
}

Parameter	Type	Description
`content`	String	Content from the scraped web page
`cookies`	String	Response cookies from the scraped web page
`status_code`	Number	Status code received from the scraped page

In case of an error, the response has the HTTP status code in the range of 4xx or 5xx and the response properties will be replaced with detail property.

For example:

{
    "detail": "Human readable error description"
}

See Errors for more details.