Automatically run bots on new queue item

On the other note - here is how we will address this issue - hope it helps someone

  1. There has to be something that adds items to queue (in my case one robot is screen scrapping a page to add data) - ADD API call once done screen scrapping to start a job (https://platform.uipath.com/swagger/ui/index#!/Jobs/Jobs_StartJobs)
  2. On File Change activity - if you have a process that changes file - you can use it to wake up robots

Workarounds, yes.

2 Likes

You have the right idea @Kemal and I’m sure that will come in time but keep in mind that there may be things that UiPath doesn’t currently do that may get higher priority (like Git integration :wink: ) so I understand their voting system. Not saying that it’s not important but which things to tackle first.

Thankfully there are workarounds for this issue.

3 Likes

Voting, especially community voting, in my mind means - it is not as important to us (UiPath), so we will let you decide. What is the percent of users that use this forum? UiPath should recognize basic functionality need and put that on their road-map. Think of licensing cost as well … if I was to go with suggestion that means I have to buy a “coordinator bot” license to coordinate work, not to mention overhead and another bot that needs to be monitored and supported (support cost)

The category should be:

  1. Nice to have - vote
  2. Must have - on road map

Follow up, who determines the “must have” - they do, and should be directly correlated to cost and maintenance and competitor features.

Anyways, I love the product - don’t get me wrong - I hate workarounds :slight_smile:

2 Likes

According to @badita they have it planned so it’s not about if it’s on the road map or not but where it lies in regards to priority. Since there is a workaround I’m guessing priority won’t be too high unless the community would really like a more direct method (where the votes come in).

1 Like

Fair, I’ve stated it as such in my response to @badita. To me, “in plan” still means we are thinking about it … because I’m sure UiPath goal is absolutely to have everything and anything added to their product - it should be in their plans.

Anyways - I think after all this we are on the same page - just can’t wait to have it as an option.

2 Likes

Point 1. Sometimes not everything is in our mind and, hopefully, we are not one mind. Many opinions here :slight_smile: . Many good things that we did not imagine came out via community (see the queue improvements). We don’t pretend to know everything and we try to challenge our ideas constantly. Priorities are permanently changing too.

Point2. Coming back to the point that is already on the roadmap and generated a long debate :). - https://forum.uipath.com/t/queue-based-best-scheduler/25. The only question is when it will be delivered. Voting and debating is a clue that we should prioritize it against other features like

  • job input arguments
  • decouple the machine/user for easier provisioning
  • licensing, to define and be able to run smoothly with different credentials on the same machine
  • process library
  • dependencies per project
  • host licensing
  • queues per environment
4 Likes

Thanks for response back @badita. Post on Queue Based Scheduler is since 2016 - so I guess I see the benefit of voting. :slight_smile:

Since someone will read this and see an interesting debate, I will throw another idea that we decided to go with vs API

Check out “Start Job” Activity and go vote for this feature! :slight_smile:

1 Like

@Kemal you are right. this feature was always in our mind and there are some other reasons for delaying it. it needs process-queue link, process has environment and how environment looks we’ll be probably changed, so it needs to come come after that.

3 Likes

I resolved the same problem using API calls. One issue I had though was that StartJobs won’t add another pending job if there is already one pending. Consider if 3 new queue items were submitted and only one bot was processing them. The first StartJob call would create a job and begin processing the first queue item, the second StartJob would create a new job and set it to Pending state, and the third StartJob WOULD NOT create a new job because one was already Pending. Therefore the third queue item wouldn’t be processed.

My workaround was to get the “State” of each queue item and continuously run StartJob every few seconds until the “State” of all queue items changed from “New”.

1 Like

I think this will be solved with the new coming (18.3) queues per environment. As of now the jobs are queued per a specific bot when a schedule is triggered. In 18.3 they will be queued per environment and assigned to any bot only when it becomes available.

5 Likes

Hi,

Since 2018.4 adds support for WebHooks, this can now be implemented outside of Orchestrator.

I have created a sample node.js application that does just that. It will use a settings file that links a queue to a process.

It is still work in progress, so feedback is welcome’d:

8 Likes

Nice stuff Bogdan :slight_smile:

2 Likes

Hi,

Current queuing mechanism in UiPath needs processes be constantly checking (pulling) a queue in order to know if there is a new item to process. This seems less than optimal as if there are no items in the queue, the process is just waiting for a new queue item to appear, instead of servicing other processes or queues.

My proposal is to have a new kind of queue, a “push queue”. In this kind of queues, the queue would be linked to an environment (a group of robots) and a process, and as soon as a new item enters the queue, the orchestrator checks which robot is available in the environment and effectively “pushes” the item to the robot, starting the designated process in the robot to process the item. This would be done while there are items in the queue. The orchestrator would follow a round-robin sequence, so all the robots would have the same average load, and all the push queues would have the same chance to push work items to an available robot.

This would optimize robot availability, as different push queues (different processes) could work with the same environment (same set of robots). This would also allow executing a set of processes in a sequential order, just having process A adding a queue item to the push queue associated with process B and so on.

Regards,
Jose.

7 Likes

Overall, I believe this is the best long term solution - “Push Queues”, if it can be implemented by UiPath. The only other good workaround listed above would be that of “Coordinator Robot” that is continually polling queues via Orchestrator API.
I think most of the other workaround solutions, including the new WebHooks, are not efficient methods of achieving the initial goal of automatically running Bots when a new Queue Item is added. Problem, for example, with WebHooks is that you can only setup 1 single Hooks for adding Queue Items, therefore your Web API POST Method is triggered for All Queues in Orchestrator. You cannot limit or designate Adding of Queue Items to a given, single Queue. If you are adding hundreds or thousands of Items across your Queues, then you are going to swamp your Web Service. Not only that, but then if you are Triggering Start Job, then mentioned above, First Job runs, then 2nd goes to Pending and anything after that will not initiate a Job - problem!
Overall, just not a good design to try and trigger Jobs per Queue Item, better off running a Job that can loop and pull/process all Queue Items and then stop. Maybe use the WebHook to initial the Job, but then as others have stated, on subsequent triggers, check the Job Status first, if it is already running then ignore the POST API Call, so you are not processing each POST at a queue item level, but just processing the Queue as a whole. Initiate the Job and let it pull and process all queue items while it runs. Once it completes, then the next new item placed on the Queue will trigger another run of the Job.

1 Like

For this scenario, as a workaround we finally implemented this “Coordination Robot” (we called it “Master Scheduler”), that enables or disables certain configured schedules depending on the queues having items or not, just using the Orchestrator API. We use schedules because that way we can configure which process and how many robots we need consuming items from the queue (the same concept as robots+process from my original post).

It’s working quite well for us. Obviously, it would be much better if this functionality came integrated into Orchestrator, as suggested.

2 Likes

Will be implemented soon with these high level specs:

A queue will have associated no more than one “ consumer” process

At the Consumer Process level we may enable “activation”.

Once the activation is enabled we set the following properties:

**trigger on new queue item

  • **max number of simultaneous jobs - default 1
  • **min number of queue items to trigger a job (don’t count deferred ones) - default 1
  • **ramp-up - default 1 - start 1 bot per x queue items
  • **Non Working Days Calendar (optional)
13 Likes

That will be awesome - thanks!

1 Like

That’s great to hear, I look forward to it!

1 Like

+1 from me.

1 Like

Great New!

~Diego Turati

1 Like