How To Identify Performance Issues With Insights 21.10+ ?

News Knowledge Base
, , , , , , , ,
1

How to identify performance issues with Insights 21.10+?

Issue Description: Insights is having performance issues and they need to be diagnosed.

Root Cause: While the frontend portal could have performance issues, most likely it is the backend end. This article will mainly outline how to check the performance of backend components.

Diagnosing:

  1. If the issue is related to PDFs, go to the section Troubleshooting PDF Rendering.
  2. Before doing anything, make sure to check the hardware requirements. Most issues will be caused by having a SQL Database that is undersized or by a Linux machine that is undersized.
    1. For SQL, the following query can be used to evaluate the size of the dataset. This can be ran on the Insights SQL serve database: 
      • Select
    S.name + '.' + T.name As TableName ,
    SUM( P.rows ) As RowCont 

        From sys.tables As T
        
Inner Join sys.partitions As P On ( P.OBJECT_ID = T.OBJECT_ID )
        
Inner Join sys.schemas As S On ( T.schema_id = S.schema_id )
        
Where
        
( T.is_ms_shipped = 0 )
        
AND
        
( P.index_id IN (1,0) )
        
And
        
( T.type = ‘U’ )
        

        Group By S.name , T.name
        

        Order By SUM( P.rows ) Desc

    2. If a ticket is opened with UiPath for a performance issue, make sure to share the output of the above query along with the hardware specs of the SQL instance. If the SQL database is co-located with other databases, make sure to mention that.
    3. SQL requirements can be found at Insights Hardware And Software Requirements - SQL Requirements
  3. Next, identify which components are performing slowly.
    1. Pay attention to where the slowness is. Is it during login and the initial page loading? Or is it when accessing a dashboard?
    2. Generate a HAR trace. See How To Generate A HAR File?
    3. For this step, the network tab is needed to be opened. (So right click anywhere, select inspect and then go to the 'Network' tab in the debug window)
    4. Reproduce the behavior
    5. Once the behavior has occurred, sort the network trace by time. An example is below
      1. In the example, sort the requests by time.
      2. We can see that the two longest running requests were 'queries' and 'home'
        • image.png
      3. When clicked on a specific requests (lets say queries in this example), the specific service or component that was being invoked can be seen. (If the request is already highlighted, click another request and then re-click the originally highlighted request).
      4. See the full details of the queries request. It takes 21s and was to the API: https://looker.local:9999/api/internal/querymanager/queries. This indicates that it is the looker component that is slow.
        • image.png
      5. When clicked on the 'home' request (which took 11s), it was directed to: https://insights.local:9443/insights/home. This is the frontend component that sits on the windows machine. This request was also somewhat slow, but can be expected when the site has not been accessed for some time.
        • image.png
      6. If a ticket is to be opened, capture the har request and share it with UiPath support.
  4. Based on the above, zone in on specific components. If a windows component is slow, login to the Insights Windows server and check the current CPU usage and Memory utilization using Task Manager. Perfmon can also be used to see exactly what is causing the performance issue.
    1. For using perfmon see more details at Windows-Commands - perfmon
    2. If it seems the windows component is slow, its recommended that a ticket is opened with UiPath. In the ticket include: A har trace, screenshot of the utilization in Task Manager.
  5. If it seems that its the looker component (which is seen in the case of the 'queries' API call) do the following:
    1. Login to the Looker back end. See more at Accessing The Backend UI Of Insights
    2. After logging in go to the admin section:
      1. In looker 22.4
        • image.png
      2. In looker 21.10
        • image.png
    3. In the Admin section, find the section called 'System Activity'. Within this section of options will be the option 'Database Performance'
      1. Go to 'Database Performance' first
      2. In the dashboard on the top right corner is a 3-dotted icon. Click that and select Download
      3. Go through the download options. The two options for format will be PDF and CSV. Try PDF but sometimes due to performance issues, this might take too long. If this is the case try CSV.
      4. We can also look at 'Instance Performance' but usually this info is not as helpful.
      5. Share the Database Performance information with UiPath.
  6. Finally check the Linux system itself.
    1. The command 'uptime' will display how much CPU the Linux machine is using.
    2. Additionally running 'top' will show the top consumers of CPU.
    3. And ps aux will also show process information.
    4. Capture the output of the above commands can help determine if the bottle neck is in the Linux machine.

Troubleshooting PDF Rendering

  1. Get the looker password and login to the back end UI. See: Accessing The Backend UI Of Insights
  2. In the backend UI, go to the Admin section (see steps in the Diagnosing section, step 5)
    1. Once the Admin section is select, go to the section 'Database' and select 'Queries'
    2. Keep this page open.
  3. In another browser session (we can use incognito or a different browser.) login to Insights and reproduce the issue.
  4. Go back to the backend UI.
  5. Queries will be listed. Either the queries will be in a rendering state or some will have completed. What we are interested in is queries that are timeout or not completing. Also, the render queries will always have the source 'Renderer'. Here is an example
  • image.png
  1. If a query is timing out, go to the query and click 'Details'
    • image.png
  2. Capture screenshot(s) of the Query Details.
    1. Make sure to capture both the info on the 'Info' toggle and the query on the 'SQL' toggle.
    2. What we are interested in is
      1. Which queries are timing out.
      2. How long they take to run
      3. And what the backend query is in SQL.
  3. Additionally if the Insights portal displays any errors, share them as well.