How to set up web scraping for each page when the "next page" button is different for each page?

I want to scrape all the URLs of the sub-lessons in this course page. However, the location of the “next page” button is different for pages that have quizzes or assignments interspersed. How can I solve this?

There are three types of pages with fixed locations of the “next page” button:

1 Like

Can you share the screenshots please? When I go to the links it did not show me a list.

sure here are some screenshots

Thanks -

Try using Check app state activity.

Output:

  • Result - Indicates if the element exists or not. The field supports only Boolean variables. Optionally, you can save this value (True/False) for later use, or save it in a cell in a parent Excel file or the Project Notebook.

Check if the next page elements exists
if it exist then click it
Else check other elements to go to next topic (skip quiz or skip assignment)
if those exist then click it.

6 Likes

thanks for the reply this is the workflow that I’ve written so far, wonder how to set up the condition for the do-while and how do I write the collect the URL strings into an excel file. (sorry I’m quite new to path :smiling_face_with_tear:)

testing.xaml (44.1 KB)

Hi mark,I can now write down the lines of each page’s url. How do I collect these URLs and save it to excel? thank you again for the help!

Multiple ways to do it.
The simplest I can think of is Write Cell Activity.

image

Set the cell value as “A” + rowNumber

Where rowNumber is a variable to increment on each loop.

1 Like

shd I place these activities below " Get url"?

Yes. make sure to initialize first the rowNumber variable to the start of the row where you want to write the URL in excel file.

hi I tried using your method but an error occurred, do you mind doing an example for me, attached can find my workflow file (sorry for keeping asking you questions cuz I’m pretty new to Uipth :smiling_face_with_tear:)

testing 002.xaml (47.2 KB)

for some reason, I cannot open the file.

what’s the error? it looks like an error on the rowNumber variable.

Make sure to set the variable type to Int32. You can also initial the value to the starting row of your excel table. In this example, I set it at 1st row of the excel rows.

hi sorry for the late reply, this is the error code that pops up,plz try out the file attached this time, thank you so much for the help lots appreciate

test.xaml (46.4 KB)

The cell should be “A”+rowNumber.toString

Thanks it works perfectly thank you so much!

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.