Perform Data Extraction on web page that delays to navigate to next page

Hi Devs, I want to extract all the data from a web page but when the bot clicks the next page it takes up to 4 minutes to navigate to that page so the extract data operation doesn’t wait till that time as a result extracts only the first page.Please how do I resolve this as the number of data to extract from that page spans upt to 33 pages please.Regards

Hello @stanstilo

You can modify the DelayBetweenPagesMS property in Data extraction.

  • DelayBetweenPagesMS - The amount of time, in milliseconds, to wait until the next page is loaded. (If the loading time of the page is longer, this value should be higher.)
3 Likes

Hello.

You have two ways:

1- Increases the delaybetweenPagesMS;
2 - Iterates between pages “manually”. For example:

  • Datascrapping the current page;
  • Element exist to validate if next page exists;
  • If yes, click “next”…
  • Validate if the page has been updated.
  • Go back to the beginning and do datascrapping.

Hug

1 Like