NAVIGATE TO LIST OF URLs(RESUMES), SCRAPE RESUME & WRITE TO CSV | PLEASE HELP ME!

I’ve wanted to join the community for a few weeks now, but I’ve been too busy trying to get this process to work.
I’ve also been too embarrassed to admit that I’ve spend endless hours on payroll at work failing to execute the above-mentioned sequence what feels like thousands of times.It’s been weeks now. I now know every single way not to make this work but still can’t get it right. I’ve been staying up late reading programming books, in addition to this forum. I don’t want a hand out, but I must admit I’ve tried my best and failed. Please help.

I will not stop wasting company time until I get it. I’m in medical and technical recruiting, so learning this will literally save me thousands of hours over the next couple years. My boss doesn’t want me wasting time on it because I’m not technically savvy enough. He is quickly catching on and I need to figure this out soon, before I get canned!

Here’s what I did:

  1. Scrape URLs of individual resumes online to csv file, using the click activity to ‘view contact’, so that when I navigate to URLs in the next workflow, the contacts will already be visible.

2.(current) Simply navigate to each URL on my spreadsheet and screen scrape contact information to a new sheet. I thought this was the easy part, but I just can’t get it.

Here is the sequence I somehow ended up with:

Read CSV (with URLs)

For Each Row

Open Browser (row(0).ToString

Screen Scraping (get full text(contact info is from a web page))

Close Tab

Write CSV

Whoever can help me with this will be my hero. I’ll even try to recruit you for a job if you need one!

Thanks for any time anyone kindly spends helping with this. Your efforts will not be forgotten!

can you share your workflow, maybe i can help you fix it.

disclaimer: i too am into recruiting

1 Like

Trying to find a way to get it to you. New users can’t upload attachments

What of your above mentioned workflow is not working and in which errors your flow runs?

1 Like

The first part navigates to each browser just fine. It’s the second part I’m having issues with.
It never writes the scraped data to csv.
The close tab activity isn’t working either, as a new browser is opened for every single URL in my spreadsheet.

When you use debug, can you see the scraped data in UiPath?
What goes wrong, do you get errors or does it work with "unexpected " results?
Are the websites you are scraping in public (If so provide an example)?

1 Like

I can’t see the scraped data in UiPath. When the debug gets to screen scraping, I’m given the following Exception:

Activity Name: Get Visible Text ‘P’
Message: Uninitialized UI node.

We use mainly private databases to serve our specific niche but we can use ziprecruiter.com as an example, as it’s the exact same concept; search for resumes that meet certain criteria, execute data scraping process to list URLs in a CSV(this part is already done) and lastly(the current issue) - create a second workflow to navigate to each URL in a for each loop and screen scrape the phone number and email address(on the resume/url) to a new CSV.

Thanks in advance

Thanks to your question, rost, I actually used debug for the first time. I fixed all of the exceptions except the last one:

Activity name: Write CSV
Message: Object not set to an instance of an object

I defined a new input variable for ‘write csv’ - DT2. The ‘encoding’ text-box I left blank

Debug should always be the first line of defense :wink:
In debug-mode you can use breakpoints and observe variables and datatables at runtime - very helpful, when looking for errors.

I understand your last post, that you not use your datatable with scraped data but a new one - which I don’t understand. The datatable you use in the “write csv”-activity should hold your data to export…

Problem solved! I had to use append csv instead of write csv like you said. I learned all about debug and it will be my best friend from now on.
I scraped 77 contacts - that being said, I also opened 77 browsers. That’s not a huge issue or anything so I’m sure I can figure it out.

Thanks for sharing this profound wisdom!

2 Likes

maybe you can also add a send hotkey alt + F4 in your workflow so that at the end when the loop is completed you dont have to close so many browsers.

Okay, after using the workflow for the first time, I opened it again and subsequently destroyed it. Back to square one. Please note:

-Debug doesn’t throw a single exception
-No exceptions are raised at all

Again, I’m trying to navigate to a list or URLs on a spreadsheet and screen scrape data from the URLs back to the spreadsheet.

Result of below-mentioned workflow:
Data is properly scraped for first url. All additional URLs have nothing scraped from them at
the conclusion my the workflow.

Here is my workflow:

EXCEL APPLICATION SCOPE: (worksheet with URLs as variable ‘dt’)

OPEN BROWSER: ("")

READ RANGE: “Sheet1”, (A1:A73)

FOR EACH ROW: (in dt)

GET ROW ITEM: (row)

NAVIGATE TO: (urlitem)

DELAY: (00:00:03_

SCREEN SCRAPING

ATTACH BROWSER

SEQUENCE

DO

GET FULL TEXT: (output = div7)

GENERATE DATA TABLE: (input = div7, output = dt2)

DELAY: (00:00:03)

EXCEL APPLICATION SCOPE: (origional sheet with URLs)

DO

APPEND RANGE: (“Sheet1”, dt2)

SAVE WORKBOOK

I’d much appreciate any help in finishing this off.

Hi,

Please uncheck the “New Session” property in Open Browser activity. This will use the same browser window again and again to navigate to different resumes. Additionally, try using “Navigate To” activity inside your loop to navigate to each profile.

1 Like

The provided information is not enough. So only guessig would be possible.
So no one can know whats in your for each is included and where it ‘ends’.

Use a breakpoint after scraping an examine the scraped data in debugmode.
Or Run debugmode at slow speed and look what works and what not

My guess, wrong selectors for sceenscraping and or attach Browser… :wink:

1 Like

over here i am guessing that in the excel sheet you are able to only see the data for 1 url where in you want to see the data for 73 url’s

1 Like

That’s exactly what was happening. I just figured out that the above information I gave you guys was incorrect; as the activity in my workflow was ‘write csv’ and not ‘append range’. I was just writing to the same line over and over (i think). Thanks so much for all the help. It works again and I saved a copy for safekeeping this time.

1 Like

Debug along with changing selectors many, many times indeed enabled me to finally finish this for good. Thank you!

Wow, I like that! I now have multiple ways to pull this off and am feeling pretty confident in my ability to recreate this workflow.

1 Like

Let me know if this cleared your problem.