Data Scraping is not retrieving information

Help Studio
, , ,
1

Hello,

I am trying to Data scrape Home Depot page, although I am having issues retrieving the information.

  1. Data scrape is not retrieving all the information of the products
  2. Data scraping will stop when we have an advertisement
    Data scraping will stop with Adv
    Data scraping will stop with Adv947×509 91 KB
    Data Scraping not retrieving info
    Data Scraping not retrieving info715×641 77.6 KB
    from the webpage

Webpage: https://www.homedepot.com/b/Electrical-Wiring-Devices-Light-Controls-Light-Switches/Rocker/N-5yc1vZc33wZ1z17lxk

2

The HTML of the Ad is different from the data you are scraping. Therefore, the scraper stops when an ad disrupts the pattern. (sometimes I feel that this is a deliberate move to discourage robots from scraping whilst making ad revenue).

One way to get around this is to scrape only the link from each of the grids and then have the Robot navigate to each link and get rest of the information. As a link is a common element between the AD and your data it may be less disruptive.

The other way is to scrape area above the ad region followed by a scrape below the ad region and then merge the two. I am leaning towards this solution because if the AD is showing up in a fixed place on the page, then you have areas that are templated and won’t confuse the Robot.