Data scraping does not capture the data from the next page

Help Studio
,
1

Hello guys, I am new to uipath
I was trying to scrap the job title and the short description data from each page from this link.
https://www.autogeneral.com.au/careers/career-opportunities/all
When I was using the data scraping tool, I selected “next” to go to the next page and store the data into an excel sheet, however only the data from the first page is stored in the excel sheet, I selected the right selector. It seems that the bot did click on the next button, it was just the data from the second page that couldn’t be stored, Can you please help? really appreciate it.

2

@R_R
Welcome to the forum

I did crosschecked and it was paging. Can you just create a new xaml, recreate the extract data part and replicate the issue?

Feel free th share some screenshot from your flow implementation with us

3

job scraping.xaml (7.2 KB)

I set the max results to 10 and after running the workflow you can only see the results from the first page in the excel worksheet

4

if you set the ContinueonError to false, it will show a run time error on the extract data activity

5

@R_R
give a try on post edited extractMetadata config from below:

<extract>
	<row exact='1'>
		<webctrl tag='div' class='career-opportunities__job'/>
	</row>
	<column exact='1' name='Title' attr='text'>
		<webctrl tag='h3' idx='1'/>
	</column>
	<column exact='1' name='Description' attr='text'>
		<webctrl tag='p' class='p paragraph__p' idx='1'/>
	</column>
</extract>

for RnD purpose it was configured to take title and description. You can later modify it

Result:

grafik
grafik824×203 45.7 KB

6

Thanks man, This seems to loop through all the pages and stuck at the last page.
the max number of results can’t control the number of jobs being scrapped.

7

job scraping.xaml (7.0 KB)
I just copied and pasted your answer into my extract data activity

8

did run your xaml and was only extracting no of jobs as configured by max results.

9

is there anyway that you can still use the maxnumberofresults in the extract data activity to control the number of jobs being scrapped? somehow I can’t do it with your code, Thank you.

10

grafik
grafik439×556 68.6 KB

was using this parameter and the extraction result was limited as configured
0 = means All results:

grafik

In case of you are using variables for this ensure the right datatype: int32 and that the value is set (e.g. active or by default setting)
what was done at your end?

11

I was using a number for this parameter. however no matter which number I use, it always goes to the last page and stops there, it does not execute the next activity. when I set the CountinueOnError to false then manually close the browser, it throws an error. 21.4.4+Branch.master.Sha.a8b5101efe02651274064093e9c7410fedff7696

Source: Extract Structured Data ‘BODY’

Message: Invalid UI node.

Exception Type: UiPath.Core.InvalidUiElementException

RemoteException wrapping UiPath.Core.InvalidUiElementException: Invalid UI node.
at UiPath.Core.Activities.ScopeActivity.OnFaulted(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)
at System.Activities.Runtime.FaultCallbackWrapper.Invoke(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)
at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)