Get Web table data from multiple pages and process each page

Help Studio Web
1

Hello Developers,

My workflow is as follows:

  1. Extract a table from the web having multiple pages
  2. after reading one page push each row to the UiPath queue
  3. then proceed with next page push each row to the queue

Currently I am able to get data from table with multiple pages but it collects the table data from all pages in one go then allow to write on queue or excel, As my data is huge like millions of rows I want to process and write to queue page wise

I can write my custom activities rather than UiPath OOTB get table data activity but looking for any better option to handle the above use case

2

Hi @taxacox851 ,
You can collect the data from one page and add it to the Queue items. Then select the next page, scrape the data, and add the details to the Queue again.

For looping through the pages, I would prefer using a Do While loop.

I hope these suggestions help you in your automation!

3

@taxacox851

First thing its always better to collect the data and then add

You can use bulk add queue items which can handle upto 15k rows

If you have mor eeven then extracting all at once makes sense..after extraction you can divide or make chunks of data and load if you need

But if you ask is to get each pge only..then use extract table without next option and include a logic to click on next and extract again sothat it is pgewise..but i would advise going with extract all and then chunks is better..dt.AsEnumerable.chunks or .skip and .take can be used for seggregating data

Cheers

1 Like
4

its not about loading its about how much data UiPath can hold into memory

Like I have 50lakhs rows in the table and collecting all that rows in one go and store in memory before processing is bit heavy task for any machine

That’s why I am preferring read a page, store 1000 records in memory, process them, then off load the page before get the next page

1 Like
5

@taxacox851

Then please follow the second option provided

cheers