Scrape data from a website without "Next" Button -does not save all data

Learn Academy Feedback
1

Hi All,

I am trying to scrape data from a web page that doesn’t have a next button and this inhibits me from navigating to the next page. I did the following but it still doesn’t work:

1 Process scrapes first page and saves in a csv file, 2 checks if element (page 2 button) exists - using a while loop, if it does it then clicks that page number and then scrapes the data again, saves in the same csv file.
But i noticed not all data is scraped and even the ones that are saved are from different pages.

How do i solve this issue please? I attached the link to the website i’m scraping from

Thanks so much

2

Hi @Olaoluwa

In the data scraping wizard, have checked the option that specifies the row limit? By default it’s set to 100. Make it to 0 so that it will extract all records in that page?

3

hi @Olaoluwa

try the below
ExtractData.xaml (11.6 KB)

Regards
Ajay

4

Hello,

Try increase the property ‘DeleyBetweenPagesMS’, the default value 300 ms. Some pages need more time to redraw informations.

5

Yes, i increased the number to 900 but the issue is that its nit scraping the date from other pages and adding to the data gotten from the first page. Here’s my xaml file
~~~Main.xaml (199.8 KB)

6

Hi Ajju,

I looked through but this doesn’t seem to be the case with my workflow.

Ola

7

Hello @Olaoluwa do not keep write range activity inside the loop… keep it at the end…
extract data activity will automatically gonna append the data in datatable…

if you log the number of rows in ExtractedDatatable… in first loop you might see there are 50 rows… in second loop you will see number of rows have increased to 100… in third loop rows count will be 150 etc…

8

Hi Akshay,
Thanks for your prompt reply. I logged the rows.count but it gave the same number -10, for each page, i dont think the data scraping is adding the scraped results to the previous data scraped. Can you please look through my workflow again?

Thanks,
Ola

9

working fine for me please check attached xaml…

___Main.xaml (19.2 KB)

try changing ExtractedDataTable scope to higher sequence

10

Thanks so much Akshay! I think this works. I appreciate it.

1 Like
11

great…
Please test thoroughly and if it works please mark my response as solution…

1 Like
12

This worked well. I’m not too familiar with the “do while” but was able to recreate what you sent with the “while” loop.
Thank you

1 Like
13

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.