Data Scraping - https://ai.fmcsa.dot.gov/ - company list & company details

Hey guys i need some help. I am a few weeks into Uipath. I successfully scraped data from this webpage;https://ai.fmcsa.dot.gov/hhg/SearchResults.asp?search=5&ads=a&state=TN&Submit=Search. Now i would like to click on each company to get its company details and write in a row with the company name. https://ai.fmcsa.dot.gov/hhg/SearchDetails.asp?ads=a&id=30881602&id2=30518138&f=search%3D5%26ads%3Da%26state%3DTN%26Submit%3DSearch

How do i automate the process to click on each company name, use get text activity to extract company details and write it in each company row with each detail having its own header.

Thanks for the help in advance

You can do like this
Use Data Scrapping for scrapping all company names data
Then add data column for which you want to add data for that company
Then use For each row
In that use click using company name row data then use get text to capture data for that company.

Did this but having a problem finding UI Element corresponding to selector

on which selector

@Ogiki_Collins What are the details you need to extract from the Company Details Page ?

I need to extract all the details on the company details page. then navigate to the safety rating data and extract it as well for each company

i managed to fix the selector i attached a browser before the click activity

This worked but it has wriiten the same company details for every company, i need it get the company details for other companies as well by possibility navigating back to the company list and click on each company get the data using get text and do the same for the other companies

As it is in for each row after scrapping data for one company close that tab at the end

Hey it has extracted an address for a single company. do i have to repeat this process for all the companies

Yup i have told you to do it in a for each row

@Ogiki_Collins Check this Workflow :
Data Scraping.zip (3.9 KB)

I was able to retrieve the data and loop through each one of them and aslo able to add the Company Details to the datatable. I have not added the Safety Details and the Fleet details. You can follow the same methods that I have followed to extract that as well.

I have only checked the execution for about 10 records, it was able to extract those 10 records and it’s company details. Since the time taken for it to extract is too much I just stopped the Execution :sweat_smile: . But it should be able to extract all the 300 records in it.

You can just Execute the Workflow and Check the Output at the end of extraction. Also you can just specify the number of records needed in the Property MaxNumberOfResults of Extract Structured Data Activity to just retrieve those many records only. If you keep it empty it will extract all data.

For Testing Purpose, I suggest you to keep only 10 or 5 records.

1 Like

Thank you so much

1 Like

Thanks

Thank you so so so so so so so so so so so so so much

1 Like

Is it works…?

Most welcome :blush: :blush:

Hey there
How do i deal with the selector that keeps breaking. especially for click activity and attach browser

Try to make then dynamic as much as possible by using * & ?
plus use less attributes for selectors and use static attributes rather than changing one

@Ogiki_Collins Check this Workflow :
It should be able to get you the required details that you need. I couldn’t check it properly because of my Internet Connectivity as the Page was not able to load within the right time. But it should work for you. Let me know if you get any issues.
Data Scraping.zip (6.1 KB)

Hey I will check it out. Thank you again. I will let you know if i have any issues Could we share contacts. I have lots of projects coming up we could work together. +256705457848. Text me on WhatsApp.