AiCenter Pipeline giving this Error, While retraining the Object Detection ML Package

Help AI Center
1

Hi Everyone, Can anyone help me to understand what this error is why it is coming. My objective is retrain the object detection ML package in Ai center.

Full training of Object_Detection 1.0 launched - Run 26bd2907-5118-4b9e-a2e5-5077a2b5154d
Full training of Object_Detection 1.0 scheduled - Run 26bd2907-5118-4b9e-a2e5-5077a2b5154d
Full training of Object_Detection 1.0 started - Run 26bd2907-5118-4b9e-a2e5-5077a2b5154d
Full training of Object_Detection 1.0 failed - Run 26bd2907-5118-4b9e-a2e5-5077a2b5154d

Error Details : Pipeline failed due to ML Package Issue

zation_95
2022-08-08 07:23:43,618 - root:load_yolo_weights:83 - INFO: conv2d_99
2022-08-08 07:23:43,618 - root:load_yolo_weights:87 - INFO: batch_normalization_96
2022-08-08 07:23:43,622 - root:load_yolo_weights:83 - INFO: conv2d_100
2022-08-08 07:23:43,622 - root:load_yolo_weights:87 - INFO: batch_normalization_97
2022-08-08 07:23:43,627 - root:load_yolo_weights:83 - INFO: conv2d_101
2022-08-08 07:23:43,628 - root:load_yolo_weights:87 - INFO: batch_normalization_98
2022-08-08 07:23:43,639 - root:load_yolo_weights:83 - INFO: conv2d_102
2022-08-08 07:23:43,639 - root:load_yolo_weights:87 - INFO: batch_normalization_99
2022-08-08 07:23:43,647 - root:load_yolo_weights:83 - INFO: conv2d_103
2022-08-08 07:23:43,648 - root:load_yolo_weights:87 - INFO: batch_normalization_100
2022-08-08 07:23:43,657 - root:load_yolo_weights:83 - INFO: conv2d_104
2022-08-08 07:23:43,657 - root:load_yolo_weights:87 - INFO: batch_normalization_101
2022-08-08 07:23:43,662 - root:load_yolo_weights:83 - INFO: conv2d_105
2022-08-08 07:23:43,662 - root:load_yolo_weights:87 - INFO: batch_normalization_102
2022-08-08 07:23:43,672 - root:load_yolo_weights:83 - INFO: conv2d_106
2022-08-08 07:23:43,672 - root:load_yolo_weights:87 - INFO: batch_normalization_103
2022-08-08 07:23:43,676 - root:load_yolo_weights:83 - INFO: conv2d_107
2022-08-08 07:23:43,677 - root:load_yolo_weights:87 - INFO: batch_normalization_104
2022-08-08 07:23:43,685 - root:load_yolo_weights:83 - INFO: conv2d_108
2022-08-08 07:23:43,686 - root:load_yolo_weights:87 - INFO: batch_normalization_105
2022-08-08 07:23:43,692 - root:load_yolo_weights:83 - INFO: conv2d_109
2022-08-08 07:23:43,692 - root:load_yolo_weights:87 - INFO: batch_normalization_106
2022-08-08 07:23:43,704 - root:load_yolo_weights:83 - INFO: conv2d_110
2022-08-08 07:23:43,705 - root:load_yolo_weights:87 - INFO: batch_normalization_107
2022-08-08 07:23:43,709 - root:load_yolo_weights:83 - INFO: conv2d_111
2022-08-08 07:23:43,710 - root:load_yolo_weights:87 - INFO: batch_normalization_108
2022-08-08 07:23:43,719 - root:load_yolo_weights:83 - INFO: conv2d_112
2022-08-08 07:23:43,719 - root:load_yolo_weights:87 - INFO: batch_normalization_109
2022-08-08 07:23:43,723 - root:load_yolo_weights:83 - INFO: conv2d_113
2022-08-08 07:23:43,724 - root:load_yolo_weights:87 - INFO: batch_normalization_110
2022-08-08 07:23:43,733 - root:load_yolo_weights:83 - INFO: conv2d_114
2022-08-08 07:23:43,733 - root:load_yolo_weights:87 - INFO: batch_normalization_111
2022-08-08 07:23:43,737 - root:load_yolo_weights:83 - INFO: conv2d_115
2022-08-08 07:23:43,737 - root:load_yolo_weights:87 - INFO: batch_normalization_112
2022-08-08 07:23:43,746 - root:load_yolo_weights:83 - INFO: conv2d_116
2022-08-08 07:23:43,746 - root:load_yolo_weights:87 - INFO: batch_normalization_113
2022-08-08 07:23:43,751 - root:load_yolo_weights:83 - INFO: conv2d_117
2022-08-08 07:23:43,751 - root:load_yolo_weights:87 - INFO: batch_normalization_114
2022-08-08 07:23:43,761 - root:load_yolo_weights:83 - INFO: conv2d_118
2022-08-08 07:23:43,761 - root:load_yolo_weights:87 - INFO: batch_normalization_115
2022-08-08 07:23:43,797 - root:load_yolo_weights:83 - INFO: conv2d_119
2022-08-08 07:23:43,798 - root:load_yolo_weights:87 - INFO: batch_normalization_116
2022-08-08 07:23:43,805 - root:load_yolo_weights:83 - INFO: conv2d_120
2022-08-08 07:23:43,805 - root:load_yolo_weights:87 - INFO: batch_normalization_117
2022-08-08 07:23:43,833 - root:load_yolo_weights:83 - INFO: conv2d_121
2022-08-08 07:23:43,833 - root:load_yolo_weights:87 - INFO: batch_normalization_118
2022-08-08 07:23:43,840 - root:load_yolo_weights:83 - INFO: conv2d_122
2022-08-08 07:23:43,841 - root:load_yolo_weights:87 - INFO: batch_normalization_119
2022-08-08 07:23:43,880 - root:load_yolo_weights:83 - INFO: conv2d_123
2022-08-08 07:23:43,880 - root:load_yolo_weights:87 - INFO: batch_normalization_120
2022-08-08 07:23:43,887 - root:load_yolo_weights:83 - INFO: conv2d_124
2022-08-08 07:23:43,887 - root:load_yolo_weights:87 - INFO: batch_normalization_121
2022-08-08 07:23:43,917 - root:load_yolo_weights:83 - INFO: conv2d_125
2022-08-08 07:23:43,918 - root:load_yolo_weights:87 - INFO: batch_normalization_122
2022-08-08 07:23:43,924 - root:load_yolo_weights:83 - INFO: conv2d_126
2022-08-08 07:23:43,924 - root:load_yolo_weights:87 - INFO: batch_normalization_123
2022-08-08 07:23:43,957 - root:load_yolo_weights:83 - INFO: conv2d_127
2022-08-08 07:23:43,957 - root:load_yolo_weights:87 - INFO: batch_normalization_124
2022-08-08 07:23:43,965 - root:load_yolo_weights:83 - INFO: conv2d_128
2022-08-08 07:23:43,965 - root:load_yolo_weights:87 - INFO: batch_normalization_125
2022-08-08 07:23:43,995 - root:load_yolo_weights:83 - INFO: conv2d_129
2022-08-08 07:23:43,995 - root:load_yolo_weights:87 - INFO: batch_normalization_126
2022-08-08 07:23:44,002 - root:load_yolo_weights:83 - INFO: conv2d_130
2022-08-08 07:23:44,002 - root:load_yolo_weights:87 - INFO: batch_normalization_127
2022-08-08 07:23:44,032 - root:load_yolo_weights:83 - INFO: conv2d_131
2022-08-08 07:23:44,032 - root:load_yolo_weights:87 - INFO: batch_normalization_128
2022-08-08 07:23:44,038 - root:load_yolo_weights:83 - INFO: conv2d_132
2022-08-08 07:23:44,039 - root:load_yolo_weights:87 - INFO: batch_normalization_129
2022-08-08 07:23:44,068 - root:load_yolo_weights:83 - INFO: conv2d_133
2022-08-08 07:23:44,069 - root:load_yolo_weights:87 - INFO: batch_normalization_130
2022-08-08 07:23:44,072 - root:load_yolo_weights:83 - INFO: conv2d_134
2022-08-08 07:23:44,072 - root:load_yolo_weights:87 - INFO: batch_normalization_130
2022-08-08 07:23:44,077 - root:load_yolo_weights:83 - INFO: conv2d_135
2022-08-08 07:23:44,077 - root:load_yolo_weights:87 - INFO: batch_normalization_131
2022-08-08 07:23:44,081 - root:load_yolo_weights:83 - INFO: conv2d_136
2022-08-08 07:23:44,082 - root:load_yolo_weights:87 - INFO: batch_normalization_132
2022-08-08 07:23:44,091 - root:load_yolo_weights:83 - INFO: conv2d_137
2022-08-08 07:23:44,091 - root:load_yolo_weights:87 - INFO: batch_normalization_133
2022-08-08 07:23:44,096 - root:load_yolo_weights:83 - INFO: conv2d_138
2022-08-08 07:23:44,096 - root:load_yolo_weights:87 - INFO: batch_normalization_134
2022-08-08 07:23:44,106 - root:load_yolo_weights:83 - INFO: conv2d_139
2022-08-08 07:23:44,106 - root:load_yolo_weights:87 - INFO: batch_normalization_135
2022-08-08 07:23:44,110 - root:load_yolo_weights:83 - INFO: conv2d_140
2022-08-08 07:23:44,110 - root:load_yolo_weights:87 - INFO: batch_normalization_136
2022-08-08 07:23:44,119 - root:load_yolo_weights:83 - INFO: conv2d_141
2022-08-08 07:23:44,119 - root:load_yolo_weights:87 - INFO: batch_normalization_137
2022-08-08 07:23:44,122 - root:load_yolo_weights:83 - INFO: conv2d_142
2022-08-08 07:23:44,122 - root:load_yolo_weights:87 - INFO: batch_normalization_137
2022-08-08 07:23:44,127 - root:load_yolo_weights:83 - INFO: conv2d_143
2022-08-08 07:23:44,127 - root:load_yolo_weights:87 - INFO: batch_normalization_138
2022-08-08 07:23:44,131 - root:load_yolo_weights:83 - INFO: conv2d_144
2022-08-08 07:23:44,131 - root:load_yolo_weights:87 - INFO: batch_normalization_139
2022-08-08 07:23:44,136 - root:load_yolo_weights:83 - INFO: conv2d_145
2022-08-08 07:23:44,136 - root:load_yolo_weights:87 - INFO: batch_normalization_140
2022-08-08 07:23:44,141 - root:load_yolo_weights:83 - INFO: conv2d_146
2022-08-08 07:23:44,141 - root:load_yolo_weights:87 - INFO: batch_normalization_141
2022-08-08 07:23:44,147 - root:load_yolo_weights:83 - INFO: conv2d_147
2022-08-08 07:23:44,148 - root:load_yolo_weights:87 - INFO: batch_normalization_142
2022-08-08 07:23:44,151 - root:load_yolo_weights:83 - INFO: conv2d_148
2022-08-08 07:23:44,152 - root:load_yolo_weights:87 - INFO: batch_normalization_143
2022-08-08 07:23:44,157 - root:load_yolo_weights:83 - INFO: conv2d_149
2022-08-08 07:23:44,157 - root:load_yolo_weights:87 - INFO: batch_normalization_144
2022-08-08 07:23:51,262 - uipath_core.training_plugin:trigger_full_training_and_publish_model:479 - ERROR: Failed to trigger full training and publish data, error: list index out of range
2022-08-08 07:23:51,263 - uipath_core.training_plugin:model_run:150 - ERROR: Training failed for pipeline type: FULL_TRAINING, error: list index out of range
2022-08-08 07:23:51,264 - uipath_core.trainer_run:main:90 - ERROR: Training Job failed, error: list index out of range
Traceback (most recent call last):
File “/home/aicenter/.local/lib/python3.6/site-packages/uipath_core/trainer_run.py”, line 85, in main
wrapper.run()
File “/microservice/training_wrapper.py”, line 57, in run
return self.training_plugin.model_run()
File “/home/aicenter/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 151, in model_run
raise e
File “/home/aicenter/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 139, in model_run
self.run_full_training()
File “/home/aicenter/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 172, in run_full_training
self.trigger_full_training_and_publish_model()
File “/home/aicenter/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 480, in trigger_full_training_and_publish_model
raise e
File “/home/aicenter/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 456, in trigger_full_training_and_publish_model
self.train_model(self.training_data_directory)
File “/home/aicenter/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 116, in train_model
self.model.train(directory)
File “/microservice/train.py”, line 52, in train
training.main(self.input_lr)
File “/microservice/training.py”, line 145, in main
for image_data, target in testset:
File “/microservice/yolov3/dataset.py”, line 108, in next
annotation = self.annotations[index]
IndexError: list index out of range

Any Ideas @supermanPunch

2

We had some issues today with AI Center UiPath Status - ML Skill Deployment and Prediction Failures in Europe Region

image
image800×409 16.1 KB

Maybe it is related.

It was working before?

3

Thanks @marian.platonov
Not sure this is related or Not. I implemented today for the first time.

4

Hi @Rohan_Goswami1 ,

Have you Checked out the Documentation for the Object Detection Model ? It maybe also related to the Datasets provided.

5

Thanks and Appreciated @supermanPunch.
Yes I have gone through these docs. Still Not able to find what is the exact.
Regards