Extract structured data by Data Scrap Wizard when there is no NEXT button, only 1234 pages buttons

Help
1

Hi guys,

Could you please help with this solution – no NEXT button or ARROW, just only 1234 page buttons.
Q4-screenshot1

Structure of the website is the same, so selectors are gona be the same.

I read lots of posts, but for a beginner it’s very difficult to understand what exact settings to put for particular activities and what is a workflow. So, screenshots or .xaml files would be ,much appreciated.

Thank you all in advance.

2

Hi
Well still we got Next 10 pages right
Did you click that and try while Data scrapping
Hope that would work for sure
That’s either the same
Cheers @nova001

3

Simplest solution would be using dynamic selector for clicking the next page.

Example for this page you can you the below selector in click activity :

Capture

4

I tried this solution but it is not working because it click on “Next 10 pages” and pages from 2-10 are ignored.
The solution might be from @Suman7.

  1. Create For each loop
  2. Scrape 1 Page of information
  3. Increment the counter
  4. Append the Data Table
1 Like
5

Hi @nova001, @srdjan.suc
Please find this sample workflow :

ScrapingData_noNextButton_135317.zip (12.4 KB)

All you need to update is the total number of pages in the while loop condition.

2 Likes
6

yea, this workflow is working! Thanks for that!

Question - where in the workflow I should put DataScrap?

7

Main.xaml (14.7 KB)

At the beginning of While.

Also, be careful when you get to increment of 11 (something like 11, 22, 33) because the selector changes, I’ve put sample for this flow, but if you could somehow dynamically change total number of iterations that would be awesome.

All credits to Suman7

8

Thank you all for your efforts!

I try to replicate the same workflow to use for other website, I have some questions.

  1. For ExtractStructuredData - when u scraped it by wizard, did u press yes or no here? if yes, then did u choose 2nd page?

    Q4-screenshot2%20-%20next%20page%20button
    Q4-screenshot2%20-%20next%20page%20button789×464 42.4 KB

  2. SendHotKey - what did u selected on the web page when u pressed “Indicate Element Inside the browser”?
    Q4-screenshot3%20-%20SendHotKey

  3. While / Condition / Counter - what if I want it to be unlimited (providing that selectors on all pages are identical), what is a syntaxis there?
    Q4-screenshot4%20-%20Counter

9

  1. You select NO because you are scraping only 1 page of information FOR EACH iteration.

  2. I think that since Send Hotkey is inside Open Browser activity it will send hotkey to the current opened window (hence no selector)

  3. I don’t understand the question

10
  1. Counter - I mean, if I want it to be unlimited, not scrap only 10 pages, but 20 or more, until the end - what should I put in Condition (instead of counter<10) ?
11

well, I am not sure, you would have to dynamically change that value for different cases.

My idea would be something like this :
You have total number of jobs found, there is 25 jobs per page, so i would divide the number of jobs by 25 and put that value in a variable

Untitled
Untitled1440×590 194 KB

1 Like
12

Listen, I tried to basically rebuild the process by myself, thank to you guys, I understood the concept, so “TimesJobs” website worked well for me.

I tried to replicate the same process for other website, but this time I got an error.
The bot scraps 1,2 pages, then scrolls down and clicks 3rd page, it loads and then stops scraping and returns an error - “cant find UI element for selector”. Excel file is filled only by data from 1 and 2nd page, not 3rd. Strange… the selectors for 3rd page should be the same…

I attached the workflow, could you please look into it? I’d like to know what I did wrong.

Q4-screenshot5%20-%20Error%20-%20Selector

13

Are you sure about the selector?
Put the static values and then try to validate the selector, something there must be different

14

Hello @srdjan.suc @Suman7

I was trying to replicate something like this for my need as well, but was unsuccessful despite trying for hours.
Could you help me out please - it is a private page that I cannot share here and I think if I could show you my screen and get some help, I would be obliged.

15

Is there sth like “page=2” in the link so that you can loop over the pages? After that merge the resulting DataTables.

16

Sorry brother I didn’t get you -
Yes there us page = 2 for that matter.

17

Hey! I figured out that the code stops working whenever it reaches page 11.
@srdjan.suc alluded to it in previous comments as well, but I guess I don’t get it.
He says something about dynamic selector.
Any help @moenk ?