Data scraping in a web page

Help
1

Good Evening,

I am using Data Scraping on website put the data to an excel. Data would be changing dynamically. It is copying only 20 records to excel. Here even i don’t have an option for spanning pages like 1,2,3 etc. It is a continuous page with no next button.

The thing is when i use data scrapping i observed that when i go down the page is loading then i can see the data.

Can someone help me.

2

i didnt get this
kindly elaborate pls
@Karthik_Kulkarni

3

If I understand, the problem is that not all data is downloaded when whole page is not loaded, am I right?

4

Yes @pllo2ptk,

Am only getting less than records which appear before the page loaded from (E-commerce site), actually i want 25 records.

so when i give 25 records at the time time of data scrapping still i can see less than that

Hi @Palaniyappan

Actually i need to scrape the ecommerce website,so when am scrapping i can only some records instead of giving specific records to be taken.the page is continuous it doesnt even have next button.

5

Hi,
Remember, in all Ecommerce websites the first rows/ads usualy are the Premium ads and the DataTable is diferent from the normal ads.

This can be your problem, if this ecomerce have premium.

If the ecommerce dont have premium ads can you post a print of the ads/lines you try to get ?

6

Hello,

Example the same problem in big basket while scrapping the data from it gives the first 20 items for a product, how to get the items which are more than 20 , as it is continous a page it is taking time to load

7

have you tried to use send hotkeys of Fn + PgDn? this will move you to the lowest of the loaded page. this will only work if the data is loaded only when navigating to the bottom of the page

1 Like
8

where i need to place this Fn+pgdn and how i can send both keys at a time.can u help me

9

A simple solution is

  1. If the pagination is at the bottom of the page , pls use set focus activity or ‘Activate’ activity to any page number element
    With this type of approach the page automatically goes down/ scroll down to the bottom of the page

By this all the rows will be displayed by then
Then again if you want , you can add find element or element exist for 25th row