Web Scraping Issue - Shipping Details

Help Studio
, ,
1

I try to perform web scraping on the following web but it always scraps only the last page but the 1st few pages. I do indicate “Later Sailing” when the system asked me to indicate the next page button

From Yantian (Guangdong), China
To Jawaharlal Nehru (MAHARASHTRA), India

2

Hey @WSONG71

It’s scraping 6 results in total is that correct ?

Thanks
#nK

3

no, it should be 15 or more. somehow, it always skips the first 2 to 3 pages.

4

How to confirm this please @WSONG71

As when I tried to use export option from the webpage even that gives me 6results !

5

at the end of each page, there is a “Later Sailing” button which allows you to go to the 2nd page and so on. Only the last page “Later Sailing” button is disabled

1 Like
6

Yes @WSONG71

Exactly, after it was disabled and I counted it was 6 !

Is it hiding some of the elements in the earlier pages ?

7

image
image1620×652 40 KB

per the screen shot, this is considered a single records

1 Like
8

Hello @WSONG71

Did you tried with the Table Extraction feature in the modern designer. if not give it a try, there you can extract the data as a patern(possible in datascrapping also), select all the needed data from each set and then give the navigation button as Later Sailing.

9

Yes @WSONG71 same way I was counting.

But curious to know, is there any identifier for each row in that results ?

10

Hi @WSONG71 ,

If you are using the Modern Design Table Extraction, then could you provide the Delay Between Pages Property value as 3 seconds and Check if it is able to navigate to all pages and extract ?

11

In order for Web scraping to work, the data structure in browser is supposed to be in Table->Row->Column format in html script. However, the data in Maersk searching page may not follow that format. I suggest you to download data from as shown on the web pages and then merge the downloaded CSV files.

image