Data Scraping over hundreds of pages

Help Studio
, ,
1

Hi,

I need to extract around 42k files from an internal website. Each page displays only 20 results, we need to open each result from the page and then download the file.

I am using data scraping to extract the links but the data is spanned over 2200 pages, since data scraping doesn’t scrape for those many pages…I am trying to scrape one page at a time. But somehow that doesn’t seem to work. It is not extracting any data, the output Data table is empty. Could it be because the results differ from when I have added Data Scraping?

Could anyone please help me with this?

2

hi @Shivani_Sama ,

It will be helpful if you can share a sample screen of the pages that you’re trying to scrape.

Kind regards,
Kenneth

3

Quick hint, in the final url, is there any variable that you can find in the original datatable?
For example

ID NAME NUMBER
1 XXX 123

If the URL is like: www.url/NAME/NUMBER you could scrape all the info and then use a go to url to access the direct page