API v1 request parameters and JSON response

Information about supported parameters and format of API request.

Authentication

You can find your API token on the dashboard page in the user's admin panel.

Header-based authentication

The API request can be authenticated using the x-api-key header:

curl --request GET \
--url 'https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com' \
--header 'x-api-key: <YOUR_SCRAPINGANT_API_KEY>'

API parameter-based authentication

The other authentication option is to use the x-api-key API parameter:

curl 'https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com&x-api-key=<YOUR_SCRAPINGANT_API_KEY>'

Request format

The web scraping endpoint is available at the following URL:

https://api.scrapingant.com/v1/general

It supports both GET and POST methods.

Passing parameters via GET method

Parameters for GET method should be passed as a query string parameters in urlencoded format.

Example below shows how will look URL with scraping request to example.com:

https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com

How to encode query parameters in different programming languages:

• Go: url.QueryEscape
• Java: URLEncoder.encode
• NodeJS: encodeURIComponent
• PHP: urlencode
• Python: urllib.parse.quote
• Ruby: URI::escape

Passing parameters via POST method

To pass parameters via POST method you just have to place them into JSON body, like the following:

{
  "url": "https://example.com"
}

Available parameters

Below you can find the full list of the available parameters:

Parameter	Required	Type	Description
url	required	String	URL to scrape
x-api-key	optional	String	Your ScrapingAnt API key. The field is optional only when the x-api-key header is set. Otherwise it is required for authentiation in ScrapingAnt API
browser	optional	Boolean	Enables using headless browser for scraping. Default: true. See Headless browser for more info.
cookies	optional	String	Cookies to pass with a scraping request to the target site. See Custom cookies for more info.
js_snippet	optional	String	Base64 encoded JS snippet to run once page being loaded in the ScrapingAnt browser. This feature works only with browser=true. See more at Javascript execution.
proxy_type	optional	String	Specifies proxy type to make request from. Default: datacenter. See Proxy settings for more info.
proxy_country	optional	String	Specifies proxy country to make request from. See Proxy settings for more info.
wait_for_selector	optional	String	The CSS selector of the element our service will wait for before returning result. This feature works only with browser=true. See Wait for CSS selector for more info.
return_text	optional	Boolean	Enables returning text only content from the page instead of full HTML. Default: false

Response structure

ScrapingAnt web scraping API endpoint returns a JSON object with the content, cookies and status_code properties:

{
    "content": "<html><body><h1>Hello, World!</h1></body></html>",
    "cookies": "cookie_name_1=cookie_value_1;cookie_name_2=cookie_value_2",
    "status_code": 200
}

Parameter	Type	Description
content	String	Content from the scraped web page
cookies	String	Response cookies from the scraped web page
status_code	Number	Status code received from the scraped page

In case of an error, the response has the HTTP status code in the range of 4xx or 5xx and the response properties will be replaced with detail property.

For example:

{
    "detail": "Human readable error description"
}

See Errors for more details.