Recognize each ui element in chrome and click on it one after the other

Hey there:)

My plan is to automatically add my youtube music songs to my library. I thought of a workflow that would recognize the ui elements (are only visible if i hover over it) and would click on it, select add to library and do that for every song in a playlist.
I found out, that many functions/activities i need are in connection with Computer Vision. I dont quite understand how to get a list of all those elements i need and click one after another. I can extract ui elements but i dont know how to get all of them and not only the first one… the for each ui element activity sounds interesting but like i said i dont know how to get all elements i need…

Can someone explain to me how to do this? Or is there already a workflow existing for this problem?

Thanks in Advance! :wink:

You don’t need Computer Vision for this.

You can get an array of Ui Elements with Find Children. Then loop through the array and click each item.

1 Like

@xda_Wolf

  • Use data scraping to get all those elements and store in a data table
  • Use for each loop, within that use click activity
  • Make sure the click activity has aaname which is the name of the element that you want to click on
  • Pass each element name through a variable in the click activity so that it click on the specific element

refer below docs for data scraping and passing the value dynamically into the selector

1 Like

Thanks so much already :)) i tried it with find children and it kinda works… for one element tho… i think the problem is, that find children only recognizes the (first) element i clicked on, the other elements are not included in the array… is there a way to include all of them although they are not shown? via css or something like that in ui explorer maybe?

1 Like

You don’t give Find Children the first element you want to click. You give it a container element that all the other elements are in, for example a DIV, TABLE, etc.

You could even just give it the entire page, if you can write a filter that gets just the elements you want.

Post a link that has an example of a playlist you’re working with and I’ll do a quick mockup.

1 Like

Thanks so much for your time and help!

This is a test playlist.
My workflow looked like this:



image

Looking at that page, the fact that the three dots menu is hidden except for the one you’re hovering over, it would probably be best just to get all the song titles and right-click each one.

And again, you don’t indicate the element. You indicate the parent container of the elements you want:

Gives us this selector for Find Children:

image

Then using Ui Explorer we find a unique property of each song title:

And put it into the Filter for the Find Children.

image

Also, make sure to set the Find Children activity to Descendants and define an output variable:

image

Now we can just loop through the array and do whatever we want with each item. For demo purposes I just Get Attribute the innertext so we can see that it’s returning what we want. Make sure to change the For Each activity’s TypeArgument to UiPath.Core.UiElement.

And here are the results:

image

A right-click is easy. Just add it inside the For Each, don’t indicate an element, just enter the loop’s variable item as the Input Element of the Click activity:

image

image

And voila, now it right-clicks each song in the playlist. Obviously you’ll want to add some steps after the Right-Click.

Note some tweaks that make it not right-click the first song in the list because it’s the currently playing song and can’t be right-clicked, as well as a keyboard Esc to make the context menu disappear.


2 Likes

Oh I just realized the first one isn’t clickable because something is wrong with that song and it’s not playable. I’ll see if I can find a way to deal with that.

You are a true hero, thank you soo much^-^ i am also thinking of an if activity that checks if it says “add to library” or “remove from library” because some songs appear in different playlists or some songs cant even be added.
Again i thank you so much, this helps me so much. :slight_smile:

OK so detecting if the song menu appeared after right-clicking is pretty easy:

image

So you do your steps for each successful right-click (ie click Add to Playlist or whatever you want to do) inside the Target Appears branch.

1 Like

The Check App State I added that looks for Add to Playlist could be set to look for Add to Library instead. So if Add to Library is there, do the relevant steps inside the Target Appears branch. If not, just press ESC and continue to the next song.

Note that I didn’t use “indicate element” nor provide an Input Element for the Keyboard Shortcuts activity. That way it’ll just send the ESC to the entire window Ui Element, which works fine.

1 Like

alright i just set everything up, the only error i get is in the if clause with “NOT index = 0”
what exactly am i checking just then?

You don’t need that. I thought the first song was not clickable because it’s already playing, but that’s not right. The idea was to always skip the first song (index 0).

But since it’s just because something is wrong with that song and it can’t be right-clicked, I used the Check App State to see if the context menu appeared. If it doesn’t, skip the song.

The answer to your error, though, is you have to define a variable to put the index from the For Each into:

image

Easiest way to do that is just to put your cursor into that field and press CTRL+K then give it a name. I used “index” but you can use whatever you want ie SongIdx, SongRow, whatever makes sense to you.

1 Like