Web Scraping Unable To Click Load More Button

Hey everyone,

I’ve been trying to scrape some data from a website and the scraping part has been working fine, but the load more button doesn’t want to click to scrape the rest of the data.
I’ve tried different browsers, and I’ve also tried setting the max results value to 100 and 0 but no change.

The page I’m trying to scrape is: https://www.thebash.com/search/20s-band-new-york-city-ny

Please see my workflow attached.
Main.xaml (6.6 KB)

Regards

hengie

Hi ,

Have you tried Data Scraping method?

Sorry, I wasn’t clear
I have been using this method but the next page selector isn’t working properly.

Regards

hengie

yes, u need to make dynamic the selector if you are using datascrabing activity

Sorry, I’m fairly new at this.

How would I go about doing this?
I don’t see any options to do that within the NextLinkSelector.

Regards

hengie

ok,
arrow button to navigate to the next page in Data Scraping
fyi

Sorry, I don’t think I’m understanding you
I’ve made a selector for the load more button on my NextLinkSelector

What am I doing wrong?

image

Regards

hengie

correct me if am wrong page is not loading to the next page ryt?this is the problem ur facing ?

check the selector for next page as well,
copy the selector of 1st page and paste it in notepad and again copy the next page selector and paste into notepad below first selector compare both the selector if any thing difference is there please make it constant

correct me if am not wrong page is not loading to the next page ryt?this is the problem ur facing ?

Kinda, I’ve noticed that it manages to click the load more button the first time, but then never clicks it again, very bizarre.
I’ve checked the selector and it doesn’t change.
More content loads with ajax so now I’m stuck on what to do here.

I tried copying the entire button element from inspect element but all it does is click it once.
When scraping I want it to click the load more button more than once so it loads all the content.

Regards

hengie

Nevermind, I got it to work
the reason it wasn’t clicking it a second time is because it couldn’t find a pattern so it assumed there was nothing else to get

Thanks for the help though :slight_smile:

okay :slight_smile:

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.