Hi all,
Trying to do table extraction for web page, web page consists of 85+ pages. Sometimes it extracts all pages, sometimes it stops in between. What to do. Please guide me on this.
Tried with data scrapping- there also same issue.
Hi @lakshmi.mp
Did it throw any error or just process stops?
Could you please try increasing the time out?
@supriya117 , Not showing any error. Execution stops.
@Dilip_Wakdikar_1996 ,
Delay between pages 3 seconds given.
Hi @lakshmi.mp
Increase the time out and Can you tried to use Data Scrapping instead of Table extraction
@lakshmi.mp
I am talking about the below timeout property
@Gokul001 tried with data scrapping same issue facing. Scrapping stops in between sometimes at 5h page, 20th page…
@Dilip_Wakdikar_1996 ,
Default time in seconds is 30sec. How to give…
Give 60000 in time out property. Means 60 sec
1.when extracting data from web pages with multiple pages using data scraping or other methods, it’s important to handle the pagination. To ensure extraction of all pages, make sure to include pagination logic to navigate through all pages systematically.
pagination meaning: Pagination refers to the process of navigating through multiple pages of a web application, typically used in data extraction scenarios to ensure all data from different pages is collected.
Check if it is in seconds or milliseconds!
Updated version of activity are dealing with seconds and older version has millisecond.
@sai_gupta , how to do the pagination. Can you explain briefly…
@lakshmi.mp Since you are able to manage scraping 20 different pages and stops at 85th page, I believe you have already achieved pagination
@Dilip_Wakdikar_1996 , Sometimes it will scraps all the pages data and stores in excel file, sometimes in between it stops and only those data stored into excel.
By setting a condition to continue data scraping as long as the “Next Page” button or link is enabled, looping through all available pages.
@sai_gupta
Please look into the screenshot
how to give condition for “Next”
1.This condition checks if the “Next” button’s “disabled” attribute is not set, indicating that the button is still clickable.
While Not String.IsNullOrEmpty(nextButtonElement.GetAttribute(“disabled”))
or
“Continue scraping while the ‘Next’ button is enabled or visible.”