Wait for CSS selector

How to improve web scraper performance and wait for the full page load

Waiting for CSS selector

Generally, our system waits until the entire page load from the networking perspective. It means that the result will be returned right after the networking is performed.

Still, in some cases, a web scraper may require waiting for a particular DOM element appearance. This DOM element may appear in the HTML before and after the entire page load. So, with waiting for CSS selector feature, it's possible to decrease the ScrapingAnt API response time or make a delay after the whole page load to wait for the DOM element.

wait_for_selector parameter

Once wait_for_selector parameter is specified, the ScrapingAnt service will no longer wait until the entire page load but wait for the desired DOM element appearance instead.

For example, with waiting for the element:

<div class="price-container"></div>

Your request will have the following look:

GET request

?url=...&wait_for_selector=.price-container

note

Don't forget to urlencode wait_for_selector parameter while using GET request

POST request

{
    "url": "...",
    "wait_for_selector": ".price-container"
}

How to create a CSS selector?

Check out CSS selectors cheat sheet and CSS Selector Tester to prepare your CSS selector.

Also, the most common selectors can be found below:

Selector	Instance	Output
*	*	Selects all elements
[attribute]	[age]	Selects all elements with age attribute
[attribute=value]	[age=50]	Selects all elements with age=50 attribute
[attribute~=value]	[color~=cyan]	Selects all elements with color attribute containing the word “cyan”
[attribute*=value]	a[href*=”voted”]	Selects all <a> elements whose href attribute contains the string “voted”
.class	.h1	Selects all elements in the HTML file with the class “h1”
.class1.class2	.postition1.position2	Selects all elements that have both “position1” and “position2” in its class attribute
.class1 .class2	.position1 .position2	Selects all parent elements that contain position1 as the attribute and with children as position2 attribute
:default	jump:input	Selects the default <jump> element
element	div	Selects all <div> elements
element.class	div.title	Selects all <div> elements with class=“title”
element,element	div,p	Selects all <div> elements and all <p> elements
element1~element2	end~paragraph	Selects all <paragraph> elements preceded by <end> element
element1+element2	middlename+firstname	Selects all <middlename> elements that are placed immediately after <firstname> element
:focus	name:focus	Selects the name element which has focus