Skip to main content

Request and response format

Information about supported parameters and format of API request.

Authentication#

You can find your API token on the dashboard page in the user's admin panel.

Header-based authentication#

The API request can be authenticated using the x-api-key header:

curl --request GET \
--url 'https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com' \
--header 'x-api-key: <YOUR_SCRAPINGANT_API_KEY>'

API parameter-based authentication#

The other authentication option is to use the x-api-key API parameter:

curl 'https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com&x-api-key=<YOUR_SCRAPINGANT_API_KEY>'

Request format#

The web scraping endpoint is available at the following URL:

https://api.scrapingant.com/v1/general

It supports both GET and POST methods.

Passing parameters via GET method#

Parameters for GET method should be passed as a query string parameters in urlencoded format.

Example below shows how will look URL with scraping request to example.com:

https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fexample.com

How to encode query parameters in different programming languages:#

Passing parameters via POST method#

To pass parameters via POST method you just have to place them into JSON body, like the following:

{
"url": "https://example.com"
}

Available parameters#

Below you can find the full list of the available parameters:

ParameterRequiredTypeDescription
urlrequiredStringURL to scrape
x-api-keyoptionalStringYour ScrapingAnt API key. The field is optional only when the x-api-key header is set. Otherwise it is required for authentiation in ScrapingAnt API
browseroptionalBooleanEnables using headless browser for scraping. Default: true. See Headless browser for more info.
cookiesoptionalStringCookies to pass with a scraping request to the target site. See Custom cookies for more info.
js_snippetoptionalStringBase64 encoded JS snippet to run once page being loaded in the ScrapingAnt browser. This feature works only with browser=true. See more at Javascript execution.
proxy_typeoptionalStringSpecifies proxy type to make request from. Default: datacenter. See Proxy settings for more info.
proxy_countryoptionalStringSpecifies proxy country to make request from. See Proxy settings for more info.
wait_for_selectoroptionalStringThe CSS selector of the element our service will wait for before returning result. This feature works only with browser=true. See Wait for CSS selector for more info.
return_textoptionalBooleanEnables returning text only content from the page instead of full HTML. Default: false

Response structure#

ScrapingAnt web scraping API endpoint returns a JSON object with the content, cookies and status_code properties:

{
"content": "<html><body><h1>Hello, World!</h1></body></html>",
"cookies": "cookie_name_1=cookie_value_1;cookie_name_2=cookie_value_2",
"status_code": 200
}
ParameterTypeDescription
contentStringContent from the scraped web page
cookiesStringResponse cookies from the scraped web page
status_codeNumberStatus code received from the scraped page

In case of an error, the response has the HTTP status code in the range of 4xx or 5xx and the response properties will be replaced with detail property.

For example:

{
"detail": "Human readable error description"
}

See Errors for more details.