Beginner - If statement + data scraping

I am trying to create a automation which will web-scrape a website of soccer players and their info. I have created the automation for the web-scraping and it works fine.
My problem is that I want it to scan every page, I would have to create an if statement to see if there are any soccer players left on screen to scan, else start the sequence that goes to the next screen.
On top of that, it would be nice if I could get an explanation on how to loop that until the countries/soccer players scanned runs out.
Appreciate any help and I will try to provide with as much information as I can to try and make it easy to understand.

project.json (1.0 KB)

@Robert_Wennberg Is that Website a dynamic page? Do the contents increase as you go scroll down the page?

Thanks for the quick response.
For some more context, this is the website:
https://www.transfermarkt.com/spieler-statistik/marktwertaenderungen/marktwertetop

What does it mean when a website is “dynamic”…?
The actual scraping itself is not a problem, I have managed to scrape each page respectively and put it into a CSV file.
What I would want though, as you may be able to see if you open up the project file, is to change the country through the dropdown list and keep on scraping for each respective country.

@Robert_Wennberg You should have Uploaded the Xaml Files not the Project Json File :sweat_smile:

Main.xaml (17.1 KB)

My apologies, trying to wrap my head around all this is making me tired haha!

1 Like

@Robert_Wennberg If you want to change the Country after each Country is Processed, You need to get the List of All the Countries Present, and each time you need to select the Country needed and then Scrape the Data for that Country, My Apologies as I’m not able to access the Website you provided, through my work environment :sweat_smile:

Thanks for taking your time to help me out.

I have already created sequences for scraping and also going to the next country, where the scraping sequence works as well. The only problem I have is that I want to set that up into the previously mentioned IF statement (to see if there are any players left to scrape, if not, start the sequence of going to the next country) and also set it up into a loop.
Seeing as the web site itself is a bit difficult to navigate, it made everything a bit more difficult unlike other forms of data scraping.

Bump.

Found an easier strategy but still stuck as to how to get out of the loop.
Looking at tutorials and reading it seems very easy to do a loop when using an Int variable, but I am wondering how you would go by when the automation reaches the last page of something?
In my case, I would like the automation do write the extracted data into a CSV file when it “hits a wall” or as mentioned earlier, when there are no more soccer players left to scan/last page.
Any help is helpful since I am just starting out!

Main.xaml (24.7 KB)