Datascraping in a for each loop but a page has a set number of results, some have less some have more. But it seems to only scrape based on the number maximum number per page

Help Studio
1

Okay that sounded very complicated, but it is based on the last topic that I posted here: Webscraping the same website but through a loop - #10 by Pratik_Wavhal

Basically, I was able to go through the for each loop and extract the results I needed. The problem comes down to this:

My webscraping is based on the next selector. So when there is no “next” button, the process would not scrape the page.

I tried to do an elements exist (for the next button). So basically, if the next button is there, scrape it with the next selector (multiple pages marked yes) . If it isn’t, scrape it without the next selector (multiple pages marked no).

I also tried it with the number of schools in the area. If it was less than or equal to 25 scrape without selector, and if it is more, scrape with the next selector.

However, both times it did not work.

This is an example of the webpage with more than 25 results: Schools in 94087, 1-25 | GreatSchools

Here is one with less than 25 results:

Attached is the excel file. The zip codes to loop through are under “Test”. The first one has more than 25 results and the rest are less than 20. The sheets before that (ZipCode+V5) are basically what happened when I ran the process.

Schools.xlsx (27.4 KB)

Any help would be greatly appreciated! Thank you!

3

Oh! This was the trial with using the number of results in a page:
Trial_14.xaml (22.2 KB)

This one makes use of the next button but I only did it for 94087 (more than 1 page of results) to see what would happen. It basically threw me off:
Trial-15.xaml (14.2 KB)

4

Sorry I had to type in many replies to the same post. This will be the last! This is also the final trial I did for Webscraping the same website but through a loop - #10 by Pratik_Wavhal. Which worked fine until it got to the zip code with only 1 page.
Trial_12.xaml (14.0 KB)
Excel file for this: school.xlsx (16.9 KB)