Question 1: Performance degradation over time when executing terminal activities in a loop. In workflows that run for a long time (over 30 minutes), the terminal activities start to take a long time to complete.
Question 2:
A loop containing 2 terminal activities starts at around 400ms for the first iteration and reaches 3-4 seconds after 1 hour.
Question 3:
For Each Row activity seems to be 60x slower than calling a For Each statement in code.
Multiple issues customers raised for this Issue:
Issue #1: Made an automation that navigates through different views of tables (SQL panel) in db2i in an AS/400 using UiPath terminal emulator. The automation works pretty well, it navigates all screen views, get the data and make some transformations to generate an Excel file. But the problem is occurring from the beginning, Terminal Activities get slower and slower. The tables are not that big (~1000-5000 rows) and the total amount of screens to be saved per table could be around the same number.
Upon checking the log messages right before and after Terminal Activities and, if the first minute it takes 200-300 milliseconds/activity, after 1 hour it takes more than 10 seconds/activity.
The above issue observed using UiPath.Terminal.Activities v2.6.1
Issue #2: A couple of times UiPath takes significantly longer to process a loop through the activities vs code. For example, the following For Each Row activity processes about 1,500 rows per second.
The following VB.Net 1 processes 100,000+ rows in less than a second:
For Each row In in_DT.Rows
r = dtOut.NewRow()
r(0) =rw(0).ToString
r(1) = rw(1).ToString
r(2) =rw(2).ToString
dtOut.Rows.Add(r)
Next rw
Root Cause: The increased execution time is due to telemetry, a feature that was added in 2.6.1.
Work Around:
No execution time increases when running with 2.6.0.
The telemetry for the Terminal Session activity collects some information like “activity name” and “execution duration” for terminal activities in the body of the session and sends all the collected data when the Terminal Session ends its execution. Since thousands of such activities it takes an ever increasing time to process the data and an ever increasing memory to store the data.
Permanent Fix:
The issue is fixed in Telemetry client 1.6 but our product team can not update the version in Studio and packages yet since it will break compatibility between older versions of Studio and latest packages ( System, Excel, Presentations etc.)
Product team will generate a new package to fix this compatibility issue. Changing Fix Version to 23.10.
Tested the following scenarios:
- Provided example with withForEachSheet parameter set to false and the execution time was totalExecutionTime :"00:08:04"
- Provided example with withForEachSheet parameter set to true and observed the behavior described in the issue. Meaning that before we canceled the execution , 33 minutes in , the process managed to read 1229 lines and the time it took to read one line was increasing message": "After read cell 1229 A1229,2023.412388,0.378621500000008"
- Provided example with withForEachSheet parameter set to false and the latest excel package available("UiPath.Excel.Activities": "[2.22.1-preview]") the execution time was "totalExecutionTime":"00:25:03". The last line logged before the execution ended was "message": "After read cell 2001 H2001,1502.6696168,0.0788852999999108"
- All the above tests were done using Studio 2023.10.0-beta.14205.