I am entering a search text into youtube, and then want to extract the results, the number of views of each results, and the age of the video.
I am trying to do it Table Extraction, and it works for couple of values. Since there is “No Next Button” in Youtube, the extraction doesn’t continue. The same issue is faced when using Linkedin network page. or linkedin wall.
Is there a work around?
One thing to be understood about Data Extraction is that it will not work if there is a pattern in the data grid that is different from the main grid that contains the data.
For example: If you have a number of products being displayed as tiles, that pattern would be broken if there is an ad placed in between the products. Visually, the tile will look identical to the tiles containing the products, but internally its elements are completely different.
When the Extractor reaches this break in the pattern, it will stop extracting information.
Here is one case that I have experienced myself:
Notice that the AD looks like any other tile, but it will break the pattern after the first 2 tiles on Row 1 and therefore it will not get to the second line:
The Next button is not always present. You may have have to look at the page url to see if the page count is included in the URL and try to move to the next page by using NavigateTo activity inside your Browser scope. You may have to putt this into a Try catch so that when you reach a page that doesn’t exist, your Robot can trap the exception and stop the extraction process.
I hope this helps.
Thanks for Andy for your quick and clear response.
I am able to take care of the Ads that come in between the patterns in most of the E-commmerce or online shops. The challenge for me is in scraping data from youtube search results.
What I am trying to do is, trying to same level of data and insights that Google gets from youtube. I start by searching for a keyword in Youtube, then try to capture the search results name, likes, the age etc. By doing this over time, we can gather data on how the videos are doing in terms of likes. From that data, insights can be gained, by automation.
We can get it by paying to Google too, but why not use it as challenge and learn. I am stuck though.