Not scraping data from entire page

[lukasziebold]

@kirshkyle

Is there a way to set the data scraping activity to wait until the webpage is fully loaded?

Something like this has been added for scraping of SAP tables, but I don’t think it is implemented in classic data scraping! Check out this thread aswell: DataScrapping on an infinite scrolling page

@kirshkyle

And do you have any idea as to why the data scraping works for other people’s bots on the same webpage and why mine may have issues?

You could try around with different browsers! Chrome/Firefox/Edge might give different results.

@kirshkyle

Do you have any other ideas?

Actually I would do it like you mentioned before and I’ve done this before in an old academy assignment:

@kirshkyle

The only idea that I have is to create a loop of attaching the browser, scrolling to the bottom of the page, extracting the data and then simulating a click in the next page button

• Use an Element Exists activity to check if a button for the next page exists.
• Then add a While activity using that Element Exists output-boolean as a condition.
• Add the same Element Exists activity (you might need to wildcard the selector) to the While sequence
• then click the next page button in the While sequence (this click will fail on the last page, because there is no button. You can catch it with Try/Catch or only execute it in if with the element exist output-boolean).

You should have your structure to loop through pages until you have reached the last page now.

@kirshkyle

After adding 7 pgdn hot keys (I am assuming there is a better way to do this)

There is! Put a Repeat activity with pgdn hotkey and repeat 7 at the start of the While sequence. Then do your data scraping, but do not indicate there is a next page, we use our own structure for this. And there you should have it:

While the button for the next page exists, your bot will scroll down, scrape data, click next page. Then on the last page it will check for the next page, scroll down, scrape data and not click.

You will probably need to merge the resulting DataTable from the current page to a master DataTable, but then you should get all the results for all the pages in one DataTable

Hope this helps!