Skip to main content
Version: v2

Custom headers

Custom headers sending via ScrapingAnt API and how to send them along with a scraping request.

Request

You can pass custom headers to the target webpages while scraping with the ScrapingAnt web scraping API.

For sending your headers with a scraping request just add the prefix ant- to your header names.

This prefix will be trimmed by ScrapingAnt and headers will be forwarded to the target web page.

tip

If you are using the ScrapingAnt Python or Javascript library, no need to prefix headers with "ant-"

Examples

Sending your headers using cURL request to ScrapingAnt

Request

curl --request GET \
--url 'https://api.scrapingant.com/v2/general?url=https%3A%2F%2Fhttpbin.org%2Fheaders&x-api-key=<YOUR_SCRAPINGANT_API_KEY>' \
--header 'Ant-Custom-Header: I <3 ScrapingAnt'

Response

{
"content":"<html><head></head><body><pre style=\"word-wrap: break-word; white-space: pre-wrap;\">{\n \"headers\": {\n \"Accept\": \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\", \n \"Accept-Encoding\": \"gzip, deflate, br\", \n \"Accept-Language\": \"en-US,en;q=0.9\", \n \"Custom-Header\": \"I &lt;3 ScrapingAnt\", \n \"Host\": \"httpbin.org\", \n \"Sec-Fetch-Dest\": \"document\", \n \"Sec-Fetch-Mode\": \"navigate\", \n \"Sec-Fetch-Site\": \"none\", \n \"Sec-Fetch-User\": \"?1\", \n \"Upgrade-Insecure-Requests\": \"1\", \n \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36\", \n \"X-Amzn-Trace-Id\": \"Root=1-61f2b754-482db58c3f0f589a5c57595d\"\n }\n}\n</pre></body></html>",
"cookies":""
}

As you can observe, HTTPBin returned our custom header as:

"Custom-Header": "I &lt;3 ScrapingAnt"

Sending your headers using Python client request to ScrapingAnt

from scrapingant_client import ScrapingAntClient


client = ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')
result = client.general_request(
'https://httpbin.org/headers',
headers={
'test-header': 'test-value'
}
)

print(result.content)

Sending your headers using Javascript client request to ScrapingAnt

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

client.scrape('https://httpbin.org/headers', { headers: { scraping: "is cool!" } })
.then(res => console.log(res))
.catch(err => console.error(err.message));