Skip to main content
Version: v2

Headers

Headers usage with ScrapingAnt API and how to send them along with a scraping request.

Headers​

ScrapingAnt allows you to send custom headers to the target web page while scraping. Also, the system sends some additional headers with every response.

Below you can find the list of the headers that are sent with every response:

Header nameDescription
Ant-credits-costThe amount of credits that were spent for the request
Ant-page-status-codeThe status code of the target web page

Request​

You can pass custom headers to the target webpages while scraping with the ScrapingAnt web scraping API.

For sending your headers with a scraping request just add the prefix ant- to your header names.

This prefix will be trimmed by ScrapingAnt and headers will be forwarded to the target web page.

tip

If you are using the ScrapingAnt Python or Javascript library, no need to prefix headers with "ant-"

Examples​

Sending your headers using cURL request to ScrapingAnt​

Request​

curl --request GET \
--url 'https://api.scrapingant.com/v2/general?url=https%3A%2F%2Fhttpbin.org%2Fheaders&x-api-key=<YOUR_SCRAPINGANT_API_KEY>' \
--header 'Ant-Custom-Header: I <3 ScrapingAnt'

Response​

<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">{
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.9",
    "Cache-Control": "no-cache",
    "Content-Length": "0",
    "Custom-Header": "I &lt;3 ScrapingAnt",
    "Host": "httpbin.org",
    "Pragma": "no-cache",
    "Sec-Ch-Ua": "\"Chromium\";v=\"106\", \"Google Chrome\";v=\"106\", \";Not A Brand\";v=\"99\"",
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": "\"Windows\"",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.5237.0 Safari/537.36",
    "X-Amzn-Trace-Id": "Root=1-6437bb94-3d4c3b1b0950bbfc06ad37dd"
  }
}
</pre></body></html>

As you can observe, HTTPBin returned our custom header as:

"Custom-Header": "I &lt;3 ScrapingAnt"

Sending your headers using Python client request to ScrapingAnt​

from scrapingant_client import ScrapingAntClient


client = ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')
result = client.general_request(
    'https://httpbin.org/headers', 
    headers={
        'test-header': 'test-value'
    }
)

print(result.content)

Sending your headers using Javascript client request to ScrapingAnt​

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

client.scrape('https://httpbin.org/headers', { headers: { scraping: "is cool!" } })
    .then(res => console.log(res))
    .catch(err => console.error(err.message));

Response​

The response will contain the same headers as the target web page returned, but prefixed with ant-original-header- prefix.

Given the same curl example as above but with -v (verbose) parameter to see headers, the response will contain the following headers:

curl -v --request GET \
--url 'https://api.scrapingant.com/v2/general?url=https%3A%2F%2Fhttpbin.org%2Fheaders&x-api-key=<YOUR_SCRAPINGANT_API_KEY>' \
--header 'Ant-Custom-Header: I <3 ScrapingAnt'
...
ant-original-header-access-control-allow-credentials: true
ant-original-header-access-control-allow-origin: *
ant-original-header-content-length: 980
ant-original-header-content-type: application/json
ant-original-header-date: Mon, 28 Aug 2023 08:51:36 GMT
ant-original-header-server: gunicorn/19.9.0
...

Alternatively, you can get the response headers without the prefix using JSON response endpoint