I have been trying to fetch some data from web page here
But it works fine for first 24 records. But if I specify “Next” its going through all the pages and never end. So its not stopping after it fetch eg: 100 records. Similar topic (here) is also in the forum but can’t get my issue fixed with that solution. Any help would be highly appreciated. TestWebScrape.xaml (14.0 KB)
You have to tune your Data Scraping activity, as it doesn’t scrap all items. I let it run for a 1000 records and then closed the IE (which results in the scraping finishing and saving to file).
It only scraped 24 records out of the 1000 it saw, which is way below the cap of 100 and the reason it keeps running.
The documentation knowledge surrounding the xml code within the tool is a bit anecdotal and comes from the “forum experience” and playing around with the tool. It definitely requires some more input and I believe our documentation team is aware of it and will document it at one point
Basically, the xml in that field is a literal “path” to your element on the page. If it happens to be too specific, it will only find the values that match the path 100%. In this case, it was only catching a few records for thousands it was exposed to.
I started removing 1 line at a time and rerunning the project to see if it works. By removing 2 lines I must have removed enough of the “too specific” path to allow it to catch all the needed values.