How to extract data from this Website

Hi Everyone,
I have a browser task which i want to extract data from that website. Here the challenge is, it don’t have any next page option we need to scroll down for further data i got struck here to solve this. can anyone guide me how to do this.

Hi @Learner007

Use a loop and extract the data and after extraction use scroll activity and extract again. For ending the loop you can check if the retrieved data before(Save the data retrieved from previous iteration in a variable to compare) scrolling and after scrolling is same or not and end the loop

cheers

You can use the Set Focus activity to automatically move the scroll to the “Best dog food collection” text and then use a Check App State to see if the “loading” image appears. If it does, wait until it disappears and then Set Focus again. You can do this all in a Do While so it dynamically stops when the “loading” image doesn’t appear again.

Hi @Anil_G i have tried this before but this is taking lot of time because some have more data and some pages have less data at the time i can’t decide where to stop the loop and if i do scrolling and scraping it will give me duplicates but here also i don’t know where to stop the loop because there is no fixed data count.
I’m just looking is there any backend process to extract all data without loops.

You’d have to ask the company if they have an API or some other way to get the data from the web site.

Hi @Learner007

as these are dynamic loading pages we cant go with backend because in backed the data loads only when you scroll.

To stop the loop as I said you have to compare the data retrieved in previous iteration and current iteration and stop the loop accordingly when it matches

Unless you have an api to retrieve tham

cheers

Hi @postwick i will try this once but it might be difficult to capture the “Loading image” if i scroll down it will load data immediately but i will try once this

@Anil_G @postwick , I don’t think so they don’t have any API for my real time website. I will take clarity on this also once.

1 Like

I can’t get it to work on my system, but that may be because of security restrictions. You could try “reader mode” (distilled pages) and see if it presents the entire list all at once without have to scroll to load more.

https://www.tenforums.com/tutorials/134235-how-enable-disable-reader-mode-google-chrome.html

Ok @postwick i will check

in general you can handle as mentioned by Paul.

As an alternate you can also page by manipulating the url:

https://headsupfortails.com/collections/dog-food?page=2
https://headsupfortails.com/collections/dog-food?page=3

till you see:
grafik

But

first scroll all then scrap data

1 Like

Hi @ppr i will all these steps once.

Oh that’s fantastic. That’s the solution that would be the simplest. Good catch noticing it works like that.