Okay that sounded very complicated, but it is based on the last topic that I posted here: Webscraping the same website but through a loop
Basically, I was able to go through the for each loop and extract the results I needed. The problem comes down to this:
My webscraping is based on the next selector. So when there is no “next” button, the process would not scrape the page.
I tried to do an elements exist (for the next button). So basically, if the next button is there, scrape it with the next selector (multiple pages marked yes) . If it isn’t, scrape it without the next selector (multiple pages marked no).
I also tried it with the number of schools in the area. If it was less than or equal to 25 scrape without selector, and if it is more, scrape with the next selector.
However, both times it did not work.
This is an example of the webpage with more than 25 results: https://www.greatschools.org/search/search.zipcode?sort=rating&zip=94087
Here is one with less than 25 results:
Attached is the excel file. The zip codes to loop through are under “Test”. The first one has more than 25 results and the rest are less than 20. The sheets before that (ZipCode+V5) are basically what happened when I ran the process.
Schools.xlsx (27.4 KB)
Any help would be greatly appreciated! Thank you!