Migration from ScrapingBee

Migrate from ScrapingBee to ScrapingAnt API effortlessly.

What is ScrapingBee?

ScrapingBee is a web scraping service that provides variety of features, but the main one is a web scraping API. It allows data extraction at scale through a REST-like API.

What ScrapingAnt API could offer?

ScrapingAnt is an alternative to ScrapingBee with it own set of features and technology wrapped to a similar API, so the migration itself is a simple process.

In addition to the features that ScrapingBee provides, ScrapingAnt API offers:

• Unlimited concurrent requests
• Price range includes small and medium businesses needs
• Reliable and diverse proxy pool with unique proxy rotation technology
• Cloud browser technology with ability of running JavaScript scenarios

It means that for your particular case our technology could provide a better alternative across:

• Success rate (number of successful requests per total number of requests)
• Speed (API request response time)
• Cost efficiency (price per request)

Prerequisites

To start the migration process you need to have:

• ScrapingAnt API key

That's it! You are ready to start the migration process.

tip

Please, be aware that ScrapingAnt's free plan limits the number of concurrent requests to 1 and mainly used for testing/development purposes. We suggest performing production testing on smaller paid plans.

Contact us via support@scrapingant.com for pricing inquiries.

Migration process

Step 1. Make API endpoint changes

ScrapingBee web scraping API endpoint is https://app.scrapingbee.com/api/v1/. It processes all the requests for web scraping tasks.

ScrapingAnt's alternative is https://api.scrapingant.com/v2/general. It's a direct replacement for ScrapingBee API endpoint for the most cases.

Check out our extended endpoint documentation in case you're using advanced features of ScrapingBee API like JSON Response.

Step 2. Make API request changes

Basic requests for both APIs require 2 parameters:

• url - URL of the target website
• api_key for ScrapingBee or x-api-key for ScrapingAnt API - API key to authenticate the request

Both APIs renders web pages in a headless browser by default, so in case you need to disable this feature - it should be performed explicitly.

For example, the basic request for ScrapingBee API looks like:

curl "https://app.scrapingbee.com/api/v1/?api_key=SCRAPINGBEE-API-KEY&url=YOUR-URL"

The same request for ScrapingAnt API:

curl "https://api.scrapingant.com/v2/general?x-api-key=SCRAPINGANT_API_KEY&url=YOUR-URL"

So for the basic cases the only need is to replace the API endpoint and API key parameter name.

In case of the more advanced usage there is a table of the most common ScrapingBee API parameters and their ScrapingAnt API alternatives:

ScrapingBee API parameter	ScrapingAnt API alternative	Description
country_code	proxy_country	Country code of the proxy to use for the request. Available at all the proxy types. More info here.
premium_proxy	proxy_type	Use premium proxy. ScrapingAnt allows configure proxy type via enum parameters. More info here.
stealth_proxy	proxy_type	Use stealth proxy. ScrapingAnt allows configure proxy type via enum parameters. More info here.
render_js	browser	Enables browser rendering. More info here.
js_scenario	js_snippet	ScrapingAnt provides clear JS in-page interface, which allows custom JS code execution. More info here
wait_for	wait-for-selector	Wait for a specific CSS selector to appear on the page before returning the result. No changes between both APIs. More info here
forward_headers	None. Enabled by default	This feature is enabled by default, the only prefix should be changed from Spb- to ant-. More info here.
return_page_source	return_page_source	The same behavior for both APIs. Enables receiving data returned by the server and unaltered by the browser.

There are plenty other parameters that could be used to customise your request at both APIs. You can find the full list of parameters for ScrapingAnt API here.

Step 3. Test your changes and integration

Despite the migration process is smooth and simple, it's always a good idea to test your changes and integration before going live.

As ScrapingAnt doesn't bill for unsuccessful requests, you can test your integration without any risk. Also, we encourage our users to implement retry mechanism to handle possible detections and other possible cases.

The sample retry mechanism could look like this:

import requests
import time


def get_response(url, api_key):
    response = requests.get(
        f"https://api.scrapingant.com/v2/general?x-api-key={api_key}&url={url}"
    )
    if response.status_code == 200:
        return response
    elif response.status_code == 409:
        # Handle rate limit for free plan
        time.sleep(1)
        return get_response(url, api_key)
    elif response.status_code == 403:
        # Handle detection
        time.sleep(1)
        return get_response(url, api_key)
    else:
        # Handle other errors
        return None

You can learn more about different not-billable response codes here and prepare your own logic based on the error-type.

It's a common practice to try different combinations of browser rendering and proxy settings automatically when receiving a detection rate more than a specific threshold to keep the cost and performance balance.

For example:

import requests
import time


def get_response(url, api_key, browser, proxy_type):
    response = requests.get(
        f"https://api.scrapingant.com/v2/general?x-api-key={api_key}&url={url}&browser={browser}&proxy_type={proxy_type}"
    )
    if response.status_code == 200:
        return response
    elif response.status_code == 409:
        # Handle rate limit for free plan
        time.sleep(1)
        return get_response(url, api_key, browser, proxy_type)
    elif response.status_code == 403:
        # Handle detection
        if browser == True:
            # Try without browser rendering
            return get_response(url, api_key, False, proxy_type)
        elif proxy_type == "residential":
            # Try with datacenter proxy
            return get_response(url, api_key, browser, "datacenter")
        else:
            # Try with residential proxy
            return get_response(url, api_key, browser, "residential")
    else:
        # Handle other errors
        return None

It's just a basic sample, but it shows a flexibility of the automation process that could be implemented based on the response information from ScrapingAnt API.

Further support

This document provides the basic information about the migration process from ScrapingBee to ScrapingAnt web scraping API.

We recommend checking the latest documentation for both APIs to have better overview across feature alternatives at ScrapingAnt.

Complex integrations could require additional support, so don't hesitate to reach us via support@scrapingant.com in case of any questions.

FAQ

What is the difference between ScrapingBee and ScrapingAnt?

ScrapingAnt is an alternative to ScrapingBee with it own set of features and technology wrapped to a simple API.

ScrapingAnt provides unique browser rendering technology that enables data extraction without getting blocked by anti-bot systems.

Where can I find the full list of parameters for ScrapingAnt API?

You can find the full list of parameters for ScrapingAnt API here.

Does ScrapingAnt provide libraries for different programming languages?

Yes, ScrapingAnt provides libraries for different programming languages:

• Python - Python client for ScrapingAnt API
• JS - Javascript client for ScrapingAnt API

Does ScrapingAnt limit the number of concurrent requests?

No, ScrapingAnt provides unlimited parallel web scraping API. You can send as many concurrent requests as you need.

Does ScrapingAnt good with website X?

ScrapingAnt is a general-purpose web scraping API, so it works well with most of the websites. However, in order to test it out you can try our free plan with 10.000 API credits per month and perform all the tests related to the target websites.

Does ScrapingAnt better than ScrapingBee?

Each web scraping company creates its own technology and provides unique features. ScrapingAnt provides unique browser rendering technology that enables data extraction without getting detected by anti-bot systems and support great scale for most of the businesses.

Still, the only and the best way is to try it out with your specific use case, as our technology could be a better fit for your needs.

Does ScrapingAnt allow subscription cancellation?

Yes, you can cancel your subscription at any time.