Custom headers

Custom headers sending via ScrapingAnt API and how to send them along with a scraping request.

Request#

You can pass custom headers to the target webpages while scraping with the ScrapingAnt web scraping API.

For sending your headers with a scraping request just add the prefix ant- to your header names.

This prefix will be trimmed by ScrapingAnt and headers will be forwarded to the target web page.

tip

If you are using the ScrapingAnt Python or Javascript library, no need to prefix headers with "ant-"

Examples#

Sending your headers using cURL request to ScrapingAnt#

Request#

curl --request GET \
--url 'https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fhttpbin.org%2Fheaders&x-api-key=<YOUR_SCRAPINGANT_API_KEY>' \
--header 'Ant-Custom-Header: I <3 ScrapingAnt'

Response#

{
  "content":"<html><head></head><body><pre style=\"word-wrap: break-word; white-space: pre-wrap;\">{\n  \"headers\": {\n    \"Accept\": \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\", \n    \"Accept-Encoding\": \"gzip, deflate, br\", \n    \"Accept-Language\": \"en-US,en;q=0.9\", \n    \"Custom-Header\": \"I &lt;3 ScrapingAnt\", \n    \"Host\": \"httpbin.org\", \n    \"Sec-Fetch-Dest\": \"document\", \n    \"Sec-Fetch-Mode\": \"navigate\", \n    \"Sec-Fetch-Site\": \"none\", \n    \"Sec-Fetch-User\": \"?1\", \n    \"Upgrade-Insecure-Requests\": \"1\", \n    \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36\", \n    \"X-Amzn-Trace-Id\": \"Root=1-61f2b754-482db58c3f0f589a5c57595d\"\n  }\n}\n</pre></body></html>",
  "cookies":""
}

As you can observe, HTTPBin returned our custom header as:

"Custom-Header": "I <3 ScrapingAnt"

Sending your headers using Python client request to ScrapingAnt#

from scrapingant_client import ScrapingAntClient


client = ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')
result = client.general_request(
    'https://httpbin.org/headers', 
    headers={
        'test-header': 'test-value'
    }
)

print(result.content)

Sending your headers using Javascript client request to ScrapingAnt#

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

client.scrape('https://httpbin.org/headers', { headers: { scraping: "is cool!" } })
    .then(res => console.log(res))
    .catch(err => console.error(err.message));