Currently I am trying to parse all the company names and url from all the pages and save it excel. I am using DataScrapper to do the job but once it reaches the last page, it’s not doing anything and hangs.
This site has almost 85 pages and total records are around 3000. It works well when we try with max 800 records but after that it fails.
What activity/call is it hanging on? Are you able to get to that point in debug mode and see what it is trying to do that it is specifically getting caught on?
I am trying to use data scrap activity to fetch company details from the site - 3424+ Top Market Research Companies in the USA | Quirks.com
This site has around 84 pages and each page has 40 records(company name). I am trying to fetch all the records(3345 ) but data scrapping is hanging if we use to fetch approx. 850 records then it’s working.
That is good information, however to assist better, it would be helpful to know exactly what step it is hanging on? If you run this in debug mode, you could likely find out what step it gets stuck on and that would help understand how to then possibly fix it?
The bot has three steps
- Open the URL
- Use Data Scrap wizard to fetch company details of all the pages
- Save it excel
We put break point on Data Scrap wizard and also in Save it excel. After that we ran it in Debug mode, it is reaching Data Scrap wizard then it keeps on running after reaching the final page (84). It is not hitting next break point which we kept on next step.
But yesterday when we tried to split it into 2 jobs. First run will do till 840 records and then next run will run from 841 to till end. We are able to scrap the data but not working in a single run for all pages.
ashokkarale
Tudor_Gavriliuc
rama_krishna_Rao
maria.josephina