Data Scraping activity not pulling in all columns and rows

Help
,
1

I’m trying to use the Extract Structured Data activity to scrape data from a webpage into a datatable for my robot to use. I can get the robot to scrape a maximum of 20 rows, and it does not scrape all the columns I selected in the wizard.

The webpage is in Chrome, and I do have the Chrome extension installed in Chrome and Studio

For the rows issue the area I’m scraping the data from needs to be scrolled, but even with scrolling so all data is visible I only get 20 results max.

The columns issue is particularly vexing, as I can see in the XML that it is clearly trying to scrape the data, but no matter what I’ve tried the “Location” column and “Profile URL” columns refuse to scrape and when I save the data to a CSV to check it they don’t even have column headers…

I can’t share the full XML or code, but here are some screenshots I can share. When I look at the full XML I don’t see any noticeable difference between the columns that ARE scraping and the ones that aren’t

image
Partial XML

image
image700×43 2.2 KB

Column headers from the CSV of the scraped data

I’ve searched all over and been unable to find any sort of help for the columns not scraping at all, and everything I’ve found or tried for all the rows failing to scrape has not helped either.

1 Like
2

So all your data is in a single web page, not multiple pages?

3

A single page where scrolling down loaded more data if there were more than 20 results in the search

4

@WitzgaWJ
Buddy can i have the screenshot of the page from where you were data scrapping
That could really help us to resolve this buddy.

I hope the first field is not getting selected properly while data scrapping, no worries buddy we can easily this out…kindly provide the screenshot or even url of the page if it is from a webpage

Cheers

5

I’m sorry, but like I said in the post I can’t share the code, site, or screenshots. There is proprietary and other confidential/sensitive information that I cannot share.

6

did u get the solution for this ?
Data scrapper is not pulling all rows from SAP in my case.

7

Can you please tell me how didi you solve this problem?? I’m facing the same problem.