Append range keeps writing same data

Help
, , ,
1

Hi Guys,

I know this topic has been raised up a few times already, but I still can’t find the solution.

MAIN TASK is:
1 To crap data from the web from multiple pages (I limited it as 4 pages for now).
2 Merge all data from multiple pages into 1 Excel file. (So every loop new data will be added to the previous one)

My WORKFLOW:
1 I created WHILE loop
2 Then, by using Data Scraping Wizard I identified what data to scrap (when it asked me about multiple pages, I said NO)
3 All scraped data then goes to ExtractDataTable1 variable
4 Data gets saved by Append Range activity to Excel file
5 Clear Data Table (ExtractDataTable1)
6 Then is scrolls down to the bottom of the page
7 Press down page 2,3,4 etc.
(looping whole process again)

THE PROBLEM is, in resulting Excel file, I get same data only from Page1 to be saved repeatedly right after another, so If I scrap four pages, in result I get 100 records (25 from each page), but they just repeat each other. What I want to achieve is to have all data from four pages added one after another in one Excel file.

For reference, I put Write Line activity (ExtractDataTable1.Rows.Count.Tostring) to check if the DataTable get cleared to null or not. It returns nothing.

I have attached my workflow screenshot, please help me to figure out what I am doing wrong.

Thanks in advance.

Workflow - Data scrap append range
Workflow - Data scrap append range1080×1670 97.5 KB

2

You should indicate the next link selector property.(Button to go to next page)
image

3

Thank you for reply, but the thing is, there is no Next page button at the webpage, only 1234 …
So, if I say YES and click on 2 when Data Scrap Wizard is asking me about multiple pages, it will be an error, since it works only when you have “Next page” or > buttons.

4

Can you show button page1 selector? In while loop activity you should declare index variable, increase + 1 and put it into your next page selector after each loop.

5

If I uderstand you correctly it should be in “Click next page” step in my workflow. This is what is inside.
webctrl aaname=‘“+counter.ToString+”’ parentid=‘searchResultData’ tag=‘A’ />

FYI, the webpage is this - “https://www.timesjobs.com/candidate/job-search.html?from=submit&actualTxtKeywords=.net%20Developer&searchBy=0&rdoOperator=OR&searchType=personalizedSearch&luceneResultSize=25&postWeek=60&txtKeywords=0DQT0net%20developer0DQT0&pDate=I&sequence=1&startPage=1”

6

Yes. Exactly!.

7

You mean I should type this code:

webctrl aaname=’“+counter.ToString+”’ parentid=‘searchResultData’ tag=‘A’ />

To NextLinkSelector in DataScrap activity? Like u mentioned the screenshot before?image

8

Check attach file:
RecordingSequence.xaml (12.1 KB)

You need dynamic click activity to go to next page as below:

image
image861×202 10.6 KB

9

I have this selector set in “Click next page” actually. So, it goes without errors clicking all 23456 etc. buttons and scraping data and saving excel file.

And I get data in Excel file, but it’s only from 1st page and keeps repeating.

So, I think, it has something to do with:
1 Clear Data Table - so it doesn’t clear it, and keeps copying 1st page in Apend Range.
or maybe
2 Something wrong with variables / DataTables, maybe I should create one more variable for DataTable?
3 Or something is wrond with DataScrap activityitself, it’s probably scraping 1st page (because I showed it in Wizard, but then because I didn’t click multiple pages, it keeps scraping only 1st page), but then why it doesn’t scrap other pages altought the browser moves to 2345 pages??

I do no… ;(

10

So, I did this experiment - I’ve deleted Clear Data Table step, then ran it and suprisingly instead of 100 records (4 pages), it showed me 150! So, I guess it’s a result that stayed in the memory from previous session? so ClearDataTable actually is working… so probably something is wrond with DataScrap, since it doesn’t scrap other pages???

11

So, I tried to add the line below to NextLinkSelector in DataScrap activity:

webctrl aaname=’*’ parentid=‘searchResultData’ tag=‘A’

now actually it goes through all pages, scraps all data, BUT the result have changed - now it shows only last scraped page in Excel. (I tried to delete ClearDataTable, but then it starts repeating again)

so I guess he solution is somewhere in the middle??

anyone, any ideas?