I try to perform web scraping on the following web but it always scraps only the last page but the 1st few pages. I do indicate “Later Sailing” when the system asked me to indicate the next page button
From Yantian (Guangdong), China
To Jawaharlal Nehru (MAHARASHTRA), India
It’s scraping 6 results in total is that correct ?
no, it should be 15 or more. somehow, it always skips the first 2 to 3 pages.
How to confirm this please @WSONG71
As when I tried to use export option from the webpage even that gives me 6results !
at the end of each page, there is a “Later Sailing” button which allows you to go to the 2nd page and so on. Only the last page “Later Sailing” button is disabled
Exactly, after it was disabled and I counted it was 6 !
Is it hiding some of the elements in the earlier pages ?
per the screen shot, this is considered a single records
Did you tried with the Table Extraction feature in the modern designer. if not give it a try, there you can extract the data as a patern(possible in datascrapping also), select all the needed data from each set and then give the navigation button as Later Sailing.
Yes @WSONG71 same way I was counting.
But curious to know, is there any identifier for each row in that results ?
Hi @WSONG71 ,
If you are using the Modern Design Table Extraction, then could you provide the
Delay Between Pages Property value as 3 seconds and Check if it is able to navigate to all pages and extract ?
In order for Web scraping to work, the data structure in browser is supposed to be in Table->Row->Column format in html script. However, the data in Maersk searching page may not follow that format. I suggest you to download data from as shown on the web pages and then merge the downloaded CSV files.