Web Scraping not returning any data

scraping
studio

#1

Hi,
I’ve created a few workflows, but have now run into this problem trying to scrape data from a web listing.

As an example, I tried to scrape data from a LinkedIn listing, but my data table ends up being empty. Workflow is pretty simple though. I am:

  • Opening a browser with the specified Url (Urlk is stored in a string variable and the opened browser is stored in a browser variable).
  • Doing the Data Scraping, which is configured using the wizard.
  • The scraping picks name and position and places these into a Data Table variable.
  • The scraping is intended to jump from page to page using the next arrow. This is also specified using the wizard.
  • Next Data Table is outputted to a txt file (just to show the result)
  • Finally number of rows is displayed using a message box.

Only a few clicks, and pretty standard, but doesn’t seem to work as per expectations… or am I doing something fundamentally wrong?

Someone, please help if you know how to do this properly. :tired_face:

I’m a new user, so cannot upload my workflow. If someone has an example, it would be great.

Br
Jasper


#2

Hi,
Did you try with IE/Chrome?
Could you do a simple datascraping in amazon site just to check whether its working fine or not?


#3

#4

Hi ddpadil,

First of all, thanks for taking your time.

Yes, I tried both browsers (IE and Chrome) and yes, I can do a simple scraping from ebay.com. But for what ever reason this one from LinkedIn does not work, though it seems pretty simple.

But several problems:

  • it doesnt fetch the data
  • it doesnt traverse the pages

Am beginning to wonder, if LinkedIn actually has done something to actively prevent data scraping.

Br
Jasper


#5

Hi,
are you facing any error.
I have scraped connection ,under my network tab in LinkedIn .
It’s working fine for me.


#6

I have exactly the same issue with data scraping not returning any data from linkedin. Have anyone found any solution for this?


#7

Hi,
Not sure why you guys facing this issue.
What version your using.
Please update package if available.(Irrelevant though :stuck_out_tongue: )
Which browser your using.
Please try to change browser chrome\IE.
Is data scraping works fine for other site (eg: amazon,Ebay…etc.


#8

Hi ddpadil,

Yes, as written above, I have tried both browsers and am able to scrape from other sites, but not LinkedIn.
I cannot upload, since my status is (still) Rookie, but If you have an example, please feel to share. Would be highly appreciated.


#9

@ovi


#10

Hi guys!

I’ve done the scraping on the Jobs page in Linkedin in Chrome browser and it works fine.

  • Indicated the first element to scrape
  • Indicated the last one
  • Click on “See more jobs” button to scrape even more results
  • After the scraping i used Write CSV activity to check the results and also a Message box to display the count of the rows in the outputted datatable.
    The results are correct.

I am attaching my xaml and csv files below.
linkedinScraping.zip (4.7 KB)

.
.
.
.

PS: @Jasper you now have a Basic badge, which means you can upload attachments :slight_smile:


#11

Hello,

I’ve also faced the same issue and I think the problem is that the webpage in linkedin isn’t loaded at the beginning, you have to scroll down to load all page items (results) and buttons in the bottom.
Here is what I made to solve the issue:
1- I zoomed out the page to make all result showed without scrolling down.
2- or you can scroll down before you scrip the data.
these solutions made me scrap data from linkedin,
hope I helped you.
Thanks.


#12

Hello Guys,

I m tried searching the solution in forum but no success.
There is not data getting extracted from web page table, preview is coming as below.


#13

did you tried to zoom out the page in the browser before scraping?


#14

Someone, please explain the logic behind the need for zooming out.
Afterall, it is still the same HTML behind the page so why does the zooming/rendering matter?

Is this in reality a weakness/bug in the tool?


#15

Hey @AhmadSallam thanks for revert, I tired many wasys , it dint happen , as far as I know and zoom in and out should not impact as it is on web page. may be some issue in table structure as first were actual data given is starting from third

row before that 1 row is for header 1 row is blank.

#16

I too have this issue. Tried both IE & Chrome. Can get data if I do not use next page option. If tried to get data from more than one page, no output…no error!


#17

Thanks AhmadSallam. I was facing the same issue. I tried Delay activity and let the page load and the issue got solved. :slight_smile:


#18

hi
I also have same issue. Even tried @ovi flow in my uipath, that also does not work for me. Any other issue you can think of ? Is it a problem with community edition ? I tried scraping with ebay and that worked fine.

Thanks
Guru


#19

Hi @inbox.gurucharan

What website are you trying to scrape?


#20

Apologies , should have written about flow a bit.
So I am trying to scrap linkedin, scrappng is part of a flow as below

  1. I open browser with linkedin.com which land me to my feed page.
  2. I search recruiter in search box and hit enter
  3. i apply filter to search particular country and hit apply
  4. the page which i have now is where i am doing scrapping, i have list of people with recruiter mentioned in there profile
  5. trying to get name and link of the profile of each person and saving it to dataTable using deata scrapping , datatable inturn is gettign written to csv

it works find till step3 but for data scrapping i dont get any error nor do i get any data in csv file.I hope that explain what i am tryign to achieve.