How to Crawl Product Information from Multiple Websites with Different Layouts Using UiPath?

Help Studio
, ,
1

I need to crawl product information from various websites, but these websites have different layouts and multiple pages containing the required details.

Could anyone guide me on how to fetch these details using UiPath automation? Specifically, I’m looking for best practices and any techniques to handle varying page structures and layouts effectively.

Thanks in advance!

2

Hi @Aman_Tiwari,

  1. Please use sequence to make different workflow for each website that you are looking to capture details from
  2. use application/browser activity to open those websites
  3. use extract table activity or table extraction wizard if the data you are looking to extract is in tabular form.
  4. extraction wizard lets you define how to go to next page to read the data from multiple pages in a website.
  5. once data is read, save it in datatable.
  6. use write range activity to print that data in excel.

Hope this helps.

Regards
Sonali

3

Hey!To scrape product info from different websites in UiPath, create separate workflows for each site since layouts vary. Use dynamic selectors and the Data Scraping Wizard when possible. Handle custom pagination with loops using Click and Element Exists. For pages with CAPTCHA, consider integrating a CAPTCHA solver via API using HTTP Request activities. Wrap all scraping logic in Try-Catch for stability. Standardize extracted data into a consistent format like a DataTable. For better performance, explore headless browsing. Keep workflows modular to handle future site changes easily. Let me know if you need help with selectors or solver integration!

4

Hi @Aman_Tiwari

Use Data Scraping for tables or lists, dynamic selectors for flexible targeting, and automate pagination by clicking “Next” to navigate pages. Use Anchor Base for scraping dynamic content near static elements and OCR for complex or image-based layouts. Implement Regex in selectors for dynamic data extraction, and handle errors with Try-Catch blocks. Use loops to iterate over multiple pages or items and wait for elements with Element Exists to ensure content is loaded. Finally, export data using Write Range or Append to CSV.

If you found helpful, feel free to tick as a solution.
Happy Automation