How to scrape data from page with in a website

Hii
I need to scrape data from a website(Web of Science Master Journal List - WoS MJL by Clarivate). I need to scrape data field is impact factor of every journal. This field is present on the page within yeh webpage and to go to that page I need to click on a button (view profile page). This needs to be done for every journal and there a number of journal on each page. And I have to do this process for whole website.
I m using datascraping and for each row activities. But the problem is as this is a button and I m not getting any url for the journal or the button . Can any one help me out .

Did you try with Data scraping?

1 Like

Yes, but to go to the page where the information is I need to click on a button. And this needs to be done for all journals information which is in thousands on entire website

.Yes, but to go to the page where the information is I need to click on a button. And this needs to be done for all journals information which is in thousands on entire website

As I see there might be 2 challenges mentioned, one is "not able to get the reference for the button " so for this you can get it buy inspecting the page then hovering over that button and then copying it’s x-path, so this way it is possible.
Now coming to the 2nd challenge which is you need to repeatedly follow the process of clicking on “view profile page” then get the desired data again go back and do the same, so for this you can try web scraping with web automation(selenium), where you can write selenium code in for loop for the automation and extract the data.
Hope this helps :slight_smile: