Workflow execution Very Slow

[phyomt (Phyo)]

Hello,

Could someone point me what would be the problem for it? Our bot process running normally in the development machine perfectly fine without any delay/slowness whereas the execution takes ages for each step in the production machine. I tried deleting browser cache, history, local temp file but non-working.

Production and development machines have the same system spec, same studio/orchestrator version with same window updates.

I compared the logs file between production and development machines and found a big difference in execution workflow.

For example in the development machine, the process could finish 5000 steps but in the production machine, it can only finish 1500 steps. I also look at the trace logs of orchestrator and found significant delays between workflow. sometimes it could take up to 2 minutes to execute the workflow. As a result, we have to stop all our processes as it’s taking longer than humans to finish the jobs.

I did email/call Uipath support and they can’t provide accurate answers for it yet. Our only option is to reformat the production machine and reinstall the uipath but we want to avoid doing that.

Thank You

[phyomt (Phyo)]

anyone have a suggestion about it ? @ClaytonM , @Palaniyappan , @loginerror

Thanks

[ClaytonM]

Just out of curiosity, you are Not running it in Debug mode are you? If you run from Studio, Debug is the default run option, so “Run File” would need to be selected.

[sumitd (Sumit)]

@phyomt - what does the process look like? What applications does it touch? Web? Firewall in Production?

Also, is there any other bot running in production? Is it as slow as this?
If there is no other bot in production, could you do a quick test bot, may be touching one of the applications and doing a simple step (login maybe?) and note if this second bot is slow as well?

[phyomt (Phyo)]

I tried running it on both Debug Mode/ Run File to analyst the execution steps. Sometimes it will take forever to invoke the xaml file but sometimes it will just go normal. Even closing the sequence would take forever in some cases.

None of those happen in development machine so i am at lost to what 's causing it.

[phyomt (Phyo)]

We use it for web application. and yes we currently have 9 processes with 2 processes being the big one.
Every single processes is running slow even IF we execute it from orchestrator (schedule mode).

Just to give an example, there’s a one process just do send an email and it would take more than 3 minutes to finish one transaction. ( even had time out error on set transaction step due to performance issues)

We cleared our orchestrator databases log files as well. None working so far. the last option is to reinstall the studio/orchestrator but would rather not to do it ( not knowing if it will solve the problem or not).

Thanks everyone

[ClaytonM]

So would you say that it’s a Studio performance issue, or just a Run-time issue? Sometimes if Studio is slow, it’s because the file is too big, however, you said it works on another machine. I’m assuming there is enough RAM available (check Task Manager for CPU and RAM usage).

If it’s a Run-time issue, I am not sure entirely, but it would depend on what activities and code causing the delays. Sometimes, you need to set the WaitForReady on all element interactions to “None”. Some applications need this or they get some weird delays.

Hopefully, those ideas help.

Regards.

[sumitd (Sumit)]

@phyomt - how different is the config of the Production vs the non-Production server? GPU? If connecting to a database, size of the database and query results? API response?
These can be very different in test/dev environment vs production (real data).

[phyomt (Phyo)]

It would be run time issues as it’s working perfectly fine in development machine without any delay/slowness. here is an example of our trace log

As you see it’s taking nearly 1 minute to execute it. That’s not all some of the workflow are just simple IF else statement and it’s doing the same things.

I did change some of the activities to None just to see if it will work but not

UiPath support is not giving us accurate answer yet for why it’s causing it.

[phyomt (Phyo)]

Everything is same as Development machine. the only different would be in production we get the new data everyday whereas in development it would be same cache. we don’t use any database/query nor api response. it’s a normal web scraping and checking the conditions to do the action.

[sumire (mitsuko kurata)]

when doing web scraping,
the time required to get element change whether we get element from cache or from the latest web page
if we get element by its id or its name.
this is just a opinion from my personal experience, excuse me if I misunderstand

[ClaytonM]

It’s doing it for basic activities like Assign or IF activities right? All I can think of is the execution log or database where the log is being sent to for the production machine is having performance issues. If it is sending the logs to the database, then there could be a network problem. The execution logs are stored locally, so likely that’s not an issue, but you can look at them in the %localAppData% folder for the user running the job, in case those files aren’t like super large or something.

Another thing to check is that the local machine has enough space on the C drive partition so it can create the execution logs. But typically, this causes an error message and not just slows the bot.

[phyomt (Phyo)]

Hi If there’s a way to turn off the logs to the database ? we are storing both locally and into database at the moment.

And yes it’s doing for basic activities as well. I tried running with step by step to see what is causing the problem. Sometimes it will take up to 1 minute to close multi assign,sequence,IF activities etc.

We have enough disk space to store the execution logs and we just cleared out all old logs file as well.

Thanks

[ClaytonM]

@phyomt

If there’s a way to turn off the logs to the database ?

I am not actually sure how to do that. You could check the NLog config file (I think it’s somewhere in the program files UiPath Studio folder). You could simply change the logging level too. Typically, you wouldn’t want Tracing level, and instead “Information” or higher which only captures the Log Message activities or Exceptions.

[phyomt (Phyo)]

It’s currently set to only “Information” but when we looked up the uipath database , we can see all level of logging messages. So i am guessing it will save all logs type into database ?

[ClaytonM]

There could be another setting for logging level to database. For example, in Orchestrator, you can set the Robot settings to have different logging level. I have not worked with the logging settings for the database, so I’m not very much help on where that setting is.

[phyomt (Phyo)]

We are following this link to update it https://docs.uipath.com/orchestrator/docs/orchestrator-logs and https://docs.uipath.com/orchestrator/v2019.0.0-fastTrack/docs/logging-configuration

but it’s not helping. I can see that everything is still writing into database even we set the value to ‘false’. That’s really a big problems for us as even a single process is having a time out issues. ( just to set the transaction status of Queue item from In progress to success, our bot couldn’t do it due to time out).

The last option is to reinstall everything and hope it will go back to normal…

Here is the another example of how slow is it just to execute delay activites

[phyomt (Phyo)]

Just updating about this problem. We disabled the anti virus and everything is back to normal, Unsure why it’s only happening with the Production environment.
Obviously, we can’t disable the anti virus so we are looking for a way to bypass unnecessary logs/exe/dat files etc from real time scanning.

Thanks Everyone for the input.