Scraping Social Media data

datatable
csv
scraping
robot
studio

#1

Hi All,

First post within the community. I am fairly new to UI Path. I am trying to currently scrape some data from the site LinkedIn via UIPath and output this into Excel. I am searching for any company (does not matter) and then trying to find people (workers) within that company with a specific i.e. assistant as a job title. I want to output their name, job title, company and email if it exists into an excel file.

I have tried to use UIPath to open the browser and go to the specific section but I am unsure on how to output the data with specific headers into a csv or .xlsx

This should be done for anyone who has the keyword assistant in their title. I would like to attach this via email and send it to myself or anyone encrypted.

Any help with this issue would be a massive help.

Many thanks in advance.


#2

Did you try using Data Scraping wizard to extract all the data from the search results page? Once extracted you can filter the data table to have only the specific assistant title.

You can then use Send SMTP Mail Message/Send Outlook Mail Message/Send Exchange Mail message as your need to send the email.

Let me know if thats not what you are looking for.

Thanks,
Rammohan B.


#3

Hi @Rammohan91 thanks for getting back to me.

I will surely try this out. I am at the moment trying to get past the stage of searching within the intial screen of Linkedin. I have used the recorder and typed a company name. When I try and run the bot it does not seem to type the name of the company I typed (whilst recording). It is very frustrating any idea?

I omitted that specifc parentid as I noticed this changes within linkedin. It worked 2 hrs ago but for some reason I cannot get it to work again :frowning:

Any ideas and suggestions would help.

Note: I have created a variable called UserInput via a input dialog that I get “the user” to fill when the robot runs.

Thanks


#4

With your selector don’t omit just do parentid=‘ember*’ or parentid=’*’ or in the edit selector screen click “attach to live element” and select the search field.


#5

You need to make your selector more dynamic & unique. Try to play around with UiExplorer and look for the attributes that doesn’t change. Something like ‘aaname’ may be.

Note: LinkedIn has a way to detect automation’s on the site and i believe it may block you after few tries while scraping the data.

Thanks,
Rammohan B.


#6

@Rammohan91 you are right LinkedIn is trying to stop me in my tracks but I am persistent…partly your solution and @CBlanchard worked a treat. I used a wild card within the ‘aaname’ this allowed me to input user defined data.

Moving forward I am in dire need of some help. I am now at the data scraping part which to be honest works like a charm. My only problem is when the page reloads (so for instance if I refresh the page or I try and run it the next day) I don’t seem to be able to get the datatable to write into a csv or excel file again. When I datascrape it and write range it works the first time but if the webpage has been reloaded (or its the next day) it stops working…

I have tried using wild cards but unfortunately this is not helping this time. I am running out of ideas. The concept works but I cannot get it to output if I reload the page. The metadata has not changed either.

I am at a crossroads, any help on this matter will be highly appreciated. I feel like I am at the verge of finishing the task but I am unable to get this last part done.

image

image

Thanks


#7

@xkarrox Hey, any good luck? Im trying to do the same but I am stuck as data scrapping returns nothing…

Please share any solution.