Data Scraping of hidden table with multiple column

kratikoza · May 26, 2021, 1:49pm

Hello Everyone,
Hopefully you guys are safe and healthy,

Suppose there is a website have having Data Table with many rows, each row has ‘view detail’ button, After clicking on view detail button there is Hidden table Having 3 columns(A,B,C) so want to scrape main data table & Hidden data table with “All Column” Data.

So i have tried following solution need help getting all the data in less time…

Used Data Scraping by selecting each row completely(All data from main table is scraped & only Column A data is scraped from Hidden Table) process takes around 3-5mins max. So is it possible to edit add Hidden Column B & C in XML-MetaData in Data Scraping via XML Editor.(Note i found out that when we click View Details button to show Hidden Data Table Column B & C data is displayed only after clicking respective column names)
Added following to above solution at the end, manually clicking all the rows hidden data table column B then scraping column B data then manually clicking all the rows hidden data table column C then scraping column C data whole process take more then 3-4hrs because there are more than 1000+ rows on the website data table…

What can be fixed to get all the data scraped in minimum time possible…
please feel free to suggest different approach would appreciate if we can edit add XML meta data and scrape all the data…

Stay Safe
Regards
Kratik

Pankaj.Patil · May 26, 2021, 2:50pm

@kratikoza,

This is shocking
Data Scraping does not take this much time it’s very fast.

I would suggest that you could ask your PO(Process Owner) if PO has different way to do this task.

I guess it’s application issue which does not allow to retrieve data so fast.

kratikoza · May 27, 2021, 1:26am

@Pankaj.Patil
Thank you for replying.

As i said earlier data scraping method done in solution 1 is taking only 3-5 mins but not able to scrape full data from hidden table. When Data scraping combined with clicking view detail & column selection button of each row then whole process take 3-4hrs…