Data scraping remains in the same web page

I am trying to extract from a website. the data is presented in more than one page, the datascraping can only extract the first page data only, so I listed all the pages in CSV file, then for each row, open browser, and called the data scrapper and save the results into CSV file.

all the pages open, but the scrapping can only read the first page data and doesn’t move to the next one.

appreciating you help and advise.

Hi @hamdan.hadeel , can you validate the selector for the other pages and highlight to see if that covers the desired table that you want to extract?

how can I do that?

@hamdan.hadeel after indicating the table, you should see the selector editor. There on the top you will find the validate button and the highlight button.

looks like it is ok as below

it replicates the entries of the first page with the number of pages it has to open, i.e. it reads the same page 7 times instead of reading the 7 pages

@hamdan.hadeel Could you please share the webpage that you are trying to scrape?

Compare Broadband Deals and Plans - Broadband Compare

test (3.4 KB) (380 Bytes)

@hamdan.hadeel looked over the webpage and faced the same issue. I think this is because of the structure of that particular webpage. Try data scraping again after zooming out the screen size to 25%.
If still that does not work you might have to build dynamic selector like below:

Here BoxNo value should be an integer type variable which you will have to increment after scraping every box. In the same way you can build a dynamic selector for clicking the next page.

Hope that helps you in any manner. :slightly_smiling_face:

Thanks for your help and support.
Finally it works in a different way.
I had to use attach browser and then navigate to the required URL instead of open URL.


Happy Automation!