Perform Data Extraction on web page that delays to navigate to next page

Hi Devs, I want to extract all the data from a web page but when the bot clicks the next page it takes up to 4 minutes to navigate to that page so the extract data operation doesn’t wait till that time as a result extracts only the first page.Please how do I resolve this as the number of data to extract from that page spans upt to 33 pages please.Regards

Hello @stanstilo

You can modify the DelayBetweenPagesMS property in Data extraction.

  • DelayBetweenPagesMS - The amount of time, in milliseconds, to wait until the next page is loaded. (If the loading time of the page is longer, this value should be higher.)


You have two ways:

1- Increases the delaybetweenPagesMS;
2 - Iterates between pages “manually”. For example:

  • Datascrapping the current page;
  • Element exist to validate if next page exists;
  • If yes, click “next”…
  • Validate if the page has been updated.
  • Go back to the beginning and do datascrapping.


1 Like