Unable to data scrape entire website

[Acerdins (A)]

Hi there,

I’m using table extraction to scrape data on SAP Fiori to a data table. The problem is that it won’t scrape all the necessary data, just the data that is visible on the screen. The more I scroll down, the more data loads, resulting in more data that is scraped. I can achieve this result by sending a hotkey to scroll all the way down, but this seems so inefficient. Are there any better ways to solve this?

Thank you!

[postwick (Paul)]

There is no better way than just scrolling all the way down. Sometimes it’s just necessary.

[vrdabberu (Varunraj)]

@Acerdins

Use extract data table activity which is available in modern experience so that u can extract the data from the entire page.

Note : Extract data table activity must be used inside the use browser/application activity.

Hope it works!!

[Vikas_M]

@Acerdins
Please refer below thread

What is the best way to create queue items in SAP Fiori?

Hi there, I am trying to create queue items of orders that are generated in SAP Fiori. More specifically, I’m trying to put the ‘‘position numbers’’ of each order in the queue item, check if they’re unique (throw a BRE if they’re not, and throw a BRE if no position number has been found), and I would like to filter whether the orders have already been edited or not. I would only like to process the ones that have not been edited. What’s the best way to go about this? Build a data table with ju…

It might help you out

[raja.arslankhan (Raja Muhammad Arslan khan)]

@Acerdins
Try this one:

[postwick (Paul)]

@raja.arslankhan

That won’t work because the problem is that the data doesn’t all load until you scroll to the bottom of the page.

[Acerdins (A)]

Hi, this doesn’t work as @postwick said.

[Acerdins (A)]

Hi, thank you for your reply. This doesn’t work as @postwick said.

[raja.arslankhan (Raja Muhammad Arslan khan)]

@Acerdins @postwick Yes Right if your data comes after scroll then you need to put parallel scrolling through parallel for each.

[vrdabberu (Varunraj)]

@Acerdins

Try the keyboard shortcuts or else the send hot keys so that the page can be loaded till the end.
Give Cntrl+End so that the page can be loaded till end and indicate the last row of table using extract data table

Regards

[postwick (Paul)]

You wouldn’t use Parallel. You’d just scroll to the bottom first, then do the Extract.

[lrtetala (Lakshman Reddy)]

Hi @Acerdins

Try this way
Use Application/Browser activity inside Extract DataTable activity

You need to indicate another item within the range of next 10 rows(beacuse it’s tend to capture 10 fields only) then check if all the fields is selected or not. If not selected then indicate the next item in the following 10 rows.

If after indicating the next item in the following 10 rows tends to clearing the first 10 rows selection then try indicate any item in that first 10 rows again except first and last fields.

I hope it helps!!