Request and response format

Information about supported parameters and format of API request.

Authentication#

You can find your API token on the dashboard page in the user's admin panel.

To be able to execute API request add the token as the x-api-key header to your request.

Request format#

The web scraping endpoint is available at the following URL:

https://api.scrapingant.com/v1/general

It supports both GET and POST methods.

Passing parameters via GET method#

Parameters for GET method should be passed as a query string parameters in urlencoded format.

Example below shows how will look URL with scraping request to example.com:

https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com

How to encode query parameters in different programming languages:#

Passing parameters via POST method#

To pass parameters via POST method you just have to place them into JSON body, like the following:

{
  "url": "https://example.com"
}

Available parameters#

Below you can find the full list of the available parameters:

Parameter	Required	Type	Description
`url`	required	String	URL to scrape
`browser`	optional	Boolean	Enables using headless browser for scraping. Default: `true`. See Headless browser for more info.
`cookies`	optional	String	Cookies to pass with a scraping request to the target site. See Custom cookies for more info.
`js_snippet`	optional	String	Base64 encoded JS snippet to run once page being loaded in the ScrapingAnt browser. This feature works only with `browser=true`. See more at Javascript execution.
`proxy_type`	optional	String	Specifies proxy type to make request from. Default: `datacenter`. See Proxy settings for more info.
`proxy_country`	optional	String	Specifies proxy country to make request from. See Proxy settings for more info.
`wait_for_selector`	optional	String	The CSS selector of the element our service will wait for before returning result. This feature works only with `browser=true`. See Wait for CSS selector for more info.
`return_text`	optional	Boolean	Enables returning text only content from the page instead of full HTML. Default: `false`

Response structure#

ScrapingAnt web scraping API endpoint returns a JSON object with the content and cookies properties:

{
    "content": "<html><body><h1>Hello, World!</h1></body></html>",
    "cookies": "cookie_name_1=cookie_value_1;cookie_name_2=cookie_value_2"
}

In case of an error, the response has the HTTP status code in the range of 4xx or 5xx and the response properties will be replaced with detail property.

For example:

{
    "detail": "Human readable error description"
}

See Errors for more details.