Title: Data Scraping Only Gets 50 Rows from Lazy-Loaded Table - How to Extract All 500+ Rows?

Help Studio
1

Question:
Using Data Scraping on a web table that loads 50 rows initially with lazy loading on scroll. Total dataset has 500+ rows but extraction only captures the first 50 visible rows.
How to extract all rows from a lazy-loaded table without pagination? Should I scrape in batches after each scroll or use a different activity than Data Scraping wizard?

2

Hi @Mahendra_Choudhary,

To remove duplicates
use the Assign activity:
dt_Final = dt_Final.DefaultView.ToTable(True)

  1. Use Table Extraction. In the Properties panel, check “Scroll down to retrieve data” and set MaxItems to 0. This is the built-in way to handle lazy loading.
  2. If that fails, wrap Extract Table Data in a While Loop. Inside the loop, add a Keyboard Shortcut for Page Down and a 1-second Delay to trigger the load.
  3. Use Merge Data Table to combine each scroll’s results into one master table.
  4. Use the Assign activity mentioned above at the very end to delete the overlapping rows captured during scrolling.
2 Likes
Forum Engagement Daily Reports
3

Extract visible rows in batches, then use merge datatable and after extraction merge the extracted data to a different datatable, that way you can get all the data.

4

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.