I am currently navigating to a health website and scraping MD names and other pertinent info.
My inquiry is three part.
1.) When I am extracting a large amount of names there are instances where it will not extract all the data per page. However, I found the way to get around this is to include a fixed delay in the ‘DelayBetweenPages’ property & change the ‘WaitForReady’ to Complete.
Keep in mind, there are instances where there can be dozens of pages. Is there any other way I can put in a requirement where there has to be a fully loaded page and extract all line items before moving on? Using a fixed delay takes too long.
2.) For some reason, the column names are not being included in the Append to CSV activity. Only the line item content. Any work around for this?
3.) Finally, there are some formatting I want to do to some of the rows BEFORE appending to csv…is this possible?
One of the columns will include a phone number…except, the bot is also scraping the ‘Phone’ tab next to the actual number.
For instance, it will pull the number ‘(###)-###-#### Phone’…can I trim the phone before inputting into csv?
Thanks again for all the help.