Troubleshooting Guide: Pipeline Failures In AI Center

News Knowledge Base
1

How to resolve failures with pipelines runs?

Follow below guide to troubleshoot failed pipelines in AI Center

  1. Open AIC and check the status of pipeline. Follow steps below according to the status
  • Waiting for Licenses - If the pipeline is stuck in waiting for licenses state, check if the corresponding licenses are available.
Cloud AIC
  • If AI Robots (legacy) licenses are being used
  1. Open License Usage in AI Center
  1. Check if sufficient licenses are available or not. Running a pipeline on CPU requires at least 1 available AI robot license and Running a pipeline on GPU requires at least 1 available AI robot pro (or) 1 AI Robot + 1 GPU license.
  2. If there are no AI Robot/AI Robot pro/GPU licenses available for the tenant, assign them to the tenant by following this link (Allocating Licenses To Tenants ) or contact our sales(https://www.uipath.com/company/contact-us) for procuring licenses
  • If AI units are being used
  1. Open Automation Cloud -> Admin -> Licenses and check if AI units are available or not. If AI units are not available or consumed, contact our sales(https://www.uipath.com/company/contact-us) for procurement.
On-Prem AIC
  • If the installation is online one, follow steps above as licensing & metering of AIC happens in Automation Cloud
  • If installation is airgapped, ensure there are sufficient licenses in orchestrator/Automation Suite
  • Running - If the pipeline is stuck in a running state for a long time,
  1. Open the pipeline and check for logs section
  1. If the logs are recent and are streaming, the pipeline is in progress.
  2. If the last log is generated long back, download the logs using download button present below the logs section and share it with the support engineer. If the download button is not present or disabled, copy the logs from logs section and share it with support engineer
  • Failed - If a pipeline is in failed state, open the logs and check for below error messages
    • Check that document type data is in dataset folder and follows folder structure - The format of the folder provided for training needs to be in dataset format. Ensure that either the path provided is right or that provided dataset is exported from Document Manager. Refer to Document Manager - Export Documents : Dataset Format
      • Incase of scheduled pipelines for auto retraining loop, select the folder containing the exports from datalabelling sessions & latest.txt.
    • Images/ directory does not exist / is empty for invoices dataset - The dataset path provided for either training dataset or evaluation dataset is empty. Update the dataset path for evaluation/training according to the pipeline
    • Unschedulable 0/n nodes are available : Insufficient CPU/memory/GPU -
      • Cloud AIC - Get the details as mentioned below in section Cloud AIC Details and share the details with support engineer.
      • On-prem AIC - Check the CPU/Memory/GPU consumption on the node(s) by using "kubectl describe node <nodename>" command in each AIC server. Check "Allocated Resources" at the bottom of the output and see if any of the resources exceed 90%. If it exceeds more than 90%, there is no enough hardware resource available to provision the pipeline. Either remove existing pipelines/ml skills to make room for new pipeline or increase the hardware resources.
    • No space left on device -
      • Cloud AIC - Get the details as mentioned below in section Cloud AIC Details and share the details with support engineer along with the error message.
      • On-prem AIC - Check the storage disks for free space and take steps to either cleanup or extend the storage accordingly
  • Killed - Pipeline is normally shows this state if the pipeline has been killed by user. Ref - Managing Pipelines . But if the pipeline status is showing Killed without action from user, the pipeline might have been killed automatically after exceeding 7 days. Reference - About Pipelines - Pipeline Status
  1. If pipeline is failing due to reasons not mentioned above, share details mentioned below with UiPath Support Engineer as per the AIC deployment type.


Cloud AIC Details

In the Cloud Tenant where this issue is occurring, gather the following information:

  • Support ID: (This can be found by navigating to Cloud -> Admin -> Settings. The Support ID is located at the top right of this screen)
  • URL: (This can be found by navigating to Cloud -> Admin -> Settings)
  • Account ID: (This can be found by navigating to Cloud -> AICenter -> In the top right of the screen, click the 3 vertical dots -> View Profile)
  • Tenant ID: (This can be found by navigating to Cloud -> AICenter -> In the top right of the screen, click the 3 vertical dots -> View Profile)
  • AI Center Project Name:
  • Share a screenshot of the ML Pipeline Page
  • Pipeline Details: (Click on the pipeline and share a screenshot of the top of the page)
  • Pipeline Logs: (Click on the pipeline and share a screenshot of the pipeline logs before scrolling down the logs. There could be an error message listed here, that would not be visible when exporting the full logs. After gathering the screenshot, copy the logs and share them by pasting in an email. If there is an option at the bottom of the logs stating partial logs are being displayed, please download the full log and share them.)
  • What was the base model used for training the pipeline? (For example: ML Packages/Out of the box Packages/UiPath Document Understanding/Invoices version 22.10.1.0).

On-Prem AIC Details

  • Version of AIC/AS (including minor version ex: 2022.10.1)
  • If the installation is standalone AIC or Automation Suite
  • If the installation is single node or multi node
  • If the installation is airgapped or online
  • Support Bundle
  • Diagnostic Logs
  • Share a screenshot of the ML Pipeline Page
  • Pipeline Details: (Click on the pipeline and share a screenshot of the top of the page)
  • Pipeline Logs: (Click on the pipeline and share a screenshot of the pipeline logs before scrolling down the logs. There could be an error message listed here, that would not be visible when exporting the full logs. After gathering the screenshot, copy the logs and share them by pasting in an email. If there is an option at the bottom of the logs stating partial logs are being displayed, please download the full log and share them.)
  • What was the base model used for training the pipeline? (For example: ML Packages/Out of the box Packages/UiPath Document Understanding/Invoices version 22.10.1.0).