All Collections
About Robots
About Extractors
How can I navigate paginated Web sites?
How can I navigate paginated Web sites?
Henrik Hofmeister avatar
Written by Henrik Hofmeister
Updated over a week ago

How can I navigate paginated Web sites?

Paginated sites are those with page numbers and Previous/Next page navigation links near the bottom of the page. There are often other pagination elements, as well, such as arrows.

The Iterate Pages step often works right out of the box by clicking an element that navigates to the next page, such as a Next or arrow link, and clicking Iterate Pages in the element panel. If it works, then great. You are all set.

During it's first iteration, it will do nothing (since it's the first page). Its second and all other iterations will click thenext pageelement. It will repeat until the target element either no longer is found or doesn't cause the page to change.

However, pagination elements can behave very differently from each other, so sometimes you have to troubleshoot in order to get it to work.

A common pattern of pagination interface design is that once you navigate to Page 2, aPreviousbutton appears. There is now a new element on the page, and this may cause the CSS path assigned to theIterate Pagesstep to become incorrect. In fact, it can sometimes actually point to thelast pagebutton instead of thenext pagebutton. In practice, this means that it will first go to Page 2, then back to Page 1.

An example:

Page 1:

1,2,3,4,5,6,last,next

CSS path: UL > LI:nth-child(8) - points tonext

Page 2:

previous,1,2,3,4,5,6 last,next

CSS path: UL > LI:nth-child(8) - now points tolast

The iteration step should point to a Next button or similar clickable element that navigates to the next page. There are two ways to do this: :last and :text_contains().

:last

One way to solve the problem is to navigate to Page 2 in the robot editor and verify that the CSS path of theIterate Pagesstep still points to thenextbutton. If it doesn't, modify the CSS path accordingly, which can often be as simple as adding:lastto the end. This will always select the last element.

:text_contains("Next")

Use this to match the wording on the elements which skips to the next page. Just replace the text in quotes (" ") with the text from the button/element.

Did this answer your question?