Data Scraping Activity - trigger "next page" when there is no explicit button


What is the use case

Extend the functionality of the Data Scraping activity so that a whole list can be captured from a Web page when there is no explicit “Next Page” button

Data Scraping requires a selector to be identified explicitly in order to display the next page of a Web page list. But some sites (Twitter and YouTube, so probably a lot more) don’t use a “Next Page” button. Instead, the next page of the list (Tweets or Comments in the above examples) displays when the page slider gets close to the limit of its downward travel or Fn + Pg-dn is pressed on the keyboard. The UiPath robot doesn’t offer this facility so there isn’t a way of capturing the whole list.

The following YouTube video has a (very) large number of comments so it’s useful for investigation and testing

Note: no other solution to this problem was offered after posting in the Rookies forum Data Scraping on Twitter - need a way to trigger "next page" when there is no button

How do you see a solution for the use case?

In the Data Scraping activity:

  1. Offer the option to display the whole list without having to specify a “next page” button (selector)
  2. In addition to a selector, offer the option of specifying a “next page” or “end” keyboard sequence

Scope: ______________* Custom Activity

  • Custom Activity
  • Reusable Component
  • Template
  • Automation Framework
  • Application Connector
  • Data Connector
  • RPA Documentation
  • Machine learning model
  • Dashboard


hi @b4bbler
Check the below post

Ashwin S


Ok, I’ve checked out the linked document, but I’m no wiser. Please indicate where in the document or examples it illustrates how to data scrape from multiple pages when there is no “Next page” button. If you know how it’s done, please post an example using the UiPath forum search facility, I look forward to your revelation.