Error while Training Object Detection ML Package-AI Center

Hi, I am unable to retrain the Object Detection Package in AI Center.
I have Followed the Documentation: Object Detection

I am always getting an error like…

2021-09-23 10:48:02,158 - uipath_core.trainer_run:main:66 - INFO: Starting training job…
2021-09-23 10:48:04,233 - uipath_core.logs.upload_log_service:upload_logs_file:63 - INFO: Retry Training Triggered:
2021-09-23 10:48:06,256 - uipath_core.storage.azure_storage_client:download:96 - INFO: Dataset from bucket folder training-22fb6f81-1795-4dfb-8432-d673f6651982/44799962-7d75-48ee-b45e-9ef53051d4ee/2101daf7-d51e-4bda-bde8-70a026d18e30 with size 218 downloaded successfully
2021-09-23 10:48:06,257 - uipath_core.training_plugin:train_model:109 - INFO: Start model training…
2021-09-23 10:48:06,257 - uipath_core.training_plugin:initialize_model:103 - INFO: Start model initialization…
2021-09-23 10:48:06,259 - root:init:37 - INFO: Finshed - Function Init : Partitioning done
2021-09-23 10:48:06,259 - uipath_core.training_plugin:initialize_model:106 - INFO: Model initialized successfully
2021-09-23 10:48:11,708 - root:load_yolo_weights:83 - INFO: conv2d_75
2021-09-23 10:48:11,708 - root:load_yolo_weights:90 - INFO: batch_normalization_72
2021-09-23 10:48:11,709 - uipath_core.training_plugin:model_run:145 - ERROR: Training failed for pipeline type: TRAIN_ONLY, error: No such layer: conv2d_75
2021-09-23 10:48:11,717 - uipath_core.trainer_run:main:81 - ERROR: Training Job failed, error: No such layer: conv2d_75
Traceback (most recent call last):
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/trainer_run.py”, line 76, in main
wrapper.run()
File “/microservice/training_wrapper.py”, line 57, in run
return self.training_plugin.model_run()
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 146, in model_run
raise e
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 138, in model_run
self.run_train_only()
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 207, in run_train_only
self.train_model(self.local_dataset_directory)
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 111, in train_model
self.model.train(directory)
File “/microservice/train.py”, line 52, in train
training.main(self.input_lr)
File “/microservice/training.py”, line 47, in main
load_yolo_weights(Darknet, Darknet_weights) # use darknet weights
File “/microservice/yolov3/utils.py”, line 104, in load_yolo_weights
conv_layer = model.get_layer(conv_layer_name)
File “/home/aifabric/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py”, line 563, in get_layer
raise ValueError('No such layer: ’ + name)
ValueError: No such layer: conv2d_75

Does Anyone have Idea About this?

Thanks&Regards
Tejesh

Hello @Tejesh_kumar_Singamsetti!

It seems that you have trouble getting an answer to your question in the first 24 hours.
Let us give you a few hints and helpful links.

First, make sure you browsed through our Forum FAQ Beginner’s Guide. It will teach you what should be included in your topic.

You can check out some of our resources directly, see below:

  1. Always search first. It is the best way to quickly find your answer. Check out the image icon for that.
    Clicking the options button will let you set more specific topic search filters, i.e. only the ones with a solution.

  2. Topic that contains most common solutions with example project files can be found here.

  3. Read our official documentation where you can find a lot of information and instructions about each of our products:

  4. Watch the videos on our official YouTube channel for more visual tutorials.

  5. Meet us and our users on our Community Slack and ask your question there.

Hopefully this will let you easily find the solution/information you need. Once you have it, we would be happy if you could share your findings here and mark it as a solution. This will help other users find it in the future.

Thank you for helping us build our UiPath Community!

Cheers from your friendly
Forum_Staff

Hi @Tejesh_kumar_Singamsetti
Are you using a GPU for this pipeline? Also is it full or train only pipeline?

Jeremy

Hi @Jeremy_Tederry ,
I am not using GPU for this pipeline and i am using Train Only Pipeline.

Hi @Jeremy_Tederry and @Tejesh_kumar_Singamsetti ,
I also tried to retrain the Object Detection ML package and had the same problem as you. I think the documentation is a bit poor on this matter. And I am very interested if someone knows how to help us with this problem.

Hi @jrmp,

I am Also Eagerly waiting to know What’s the Problem in it. Waiting for Response…!!

1 Like

Hi

Sorry for the late answer could you try using full pipeline? We saw some issues on train only pipelines.

Jeremy

Hi @Jeremy_Tederry,

Thanks for your reply @Jeremy_Tederry .

I have tried to train with Full pipeline and it got failed.

Getting Logs like:

Full training of ObjectDetector 1.0 launched - Run 44bbe37f-aaf2-4c7b-a9c8-252b026401c4
Full training of ObjectDetector 1.0 scheduled - Run 44bbe37f-aaf2-4c7b-a9c8-252b026401c4
Full training of ObjectDetector 1.0 started - Run 44bbe37f-aaf2-4c7b-a9c8-252b026401c4
Full training of ObjectDetector 1.0 failed - Run 44bbe37f-aaf2-4c7b-a9c8-252b026401c4

Error Details : Pipeline failed due to ML Package Issue

yolo_weights:83 - INFO: conv2d_106
2021-10-07 06:19:46,753 - root:load_yolo_weights:87 - INFO: batch_normalization_103
2021-10-07 06:19:46,757 - root:load_yolo_weights:83 - INFO: conv2d_107
2021-10-07 06:19:46,758 - root:load_yolo_weights:87 - INFO: batch_normalization_104
2021-10-07 06:19:46,767 - root:load_yolo_weights:83 - INFO: conv2d_108
2021-10-07 06:19:46,768 - root:load_yolo_weights:87 - INFO: batch_normalization_105
2021-10-07 06:19:46,771 - root:load_yolo_weights:83 - INFO: conv2d_109
2021-10-07 06:19:46,772 - root:load_yolo_weights:87 - INFO: batch_normalization_106
2021-10-07 06:19:46,781 - root:load_yolo_weights:83 - INFO: conv2d_110
2021-10-07 06:19:46,781 - root:load_yolo_weights:87 - INFO: batch_normalization_107
2021-10-07 06:19:46,785 - root:load_yolo_weights:83 - INFO: conv2d_111
2021-10-07 06:19:46,785 - root:load_yolo_weights:87 - INFO: batch_normalization_108
2021-10-07 06:19:46,794 - root:load_yolo_weights:83 - INFO: conv2d_112
2021-10-07 06:19:46,794 - root:load_yolo_weights:87 - INFO: batch_normalization_109
2021-10-07 06:19:46,798 - root:load_yolo_weights:83 - INFO: conv2d_113
2021-10-07 06:19:46,798 - root:load_yolo_weights:87 - INFO: batch_normalization_110
2021-10-07 06:19:46,807 - root:load_yolo_weights:83 - INFO: conv2d_114
2021-10-07 06:19:46,808 - root:load_yolo_weights:87 - INFO: batch_normalization_111
2021-10-07 06:19:46,812 - root:load_yolo_weights:83 - INFO: conv2d_115
2021-10-07 06:19:46,812 - root:load_yolo_weights:87 - INFO: batch_normalization_112
2021-10-07 06:19:46,821 - root:load_yolo_weights:83 - INFO: conv2d_116
2021-10-07 06:19:46,821 - root:load_yolo_weights:87 - INFO: batch_normalization_113
2021-10-07 06:19:46,826 - root:load_yolo_weights:83 - INFO: conv2d_117
2021-10-07 06:19:46,826 - root:load_yolo_weights:87 - INFO: batch_normalization_114
2021-10-07 06:19:46,835 - root:load_yolo_weights:83 - INFO: conv2d_118
2021-10-07 06:19:46,835 - root:load_yolo_weights:87 - INFO: batch_normalization_115
2021-10-07 06:19:46,869 - root:load_yolo_weights:83 - INFO: conv2d_119
2021-10-07 06:19:46,870 - root:load_yolo_weights:87 - INFO: batch_normalization_116
2021-10-07 06:19:46,877 - root:load_yolo_weights:83 - INFO: conv2d_120
2021-10-07 06:19:46,878 - root:load_yolo_weights:87 - INFO: batch_normalization_117
2021-10-07 06:19:46,906 - root:load_yolo_weights:83 - INFO: conv2d_121
2021-10-07 06:19:46,907 - root:load_yolo_weights:87 - INFO: batch_normalization_118
2021-10-07 06:19:46,914 - root:load_yolo_weights:83 - INFO: conv2d_122
2021-10-07 06:19:46,914 - root:load_yolo_weights:87 - INFO: batch_normalization_119
2021-10-07 06:19:46,942 - root:load_yolo_weights:83 - INFO: conv2d_123
2021-10-07 06:19:46,942 - root:load_yolo_weights:87 - INFO: batch_normalization_120
2021-10-07 06:19:46,949 - root:load_yolo_weights:83 - INFO: conv2d_124
2021-10-07 06:19:46,949 - root:load_yolo_weights:87 - INFO: batch_normalization_121
2021-10-07 06:19:46,979 - root:load_yolo_weights:83 - INFO: conv2d_125
2021-10-07 06:19:46,979 - root:load_yolo_weights:87 - INFO: batch_normalization_122
2021-10-07 06:19:46,986 - root:load_yolo_weights:83 - INFO: conv2d_126
2021-10-07 06:19:46,986 - root:load_yolo_weights:87 - INFO: batch_normalization_123
2021-10-07 06:19:47,014 - root:load_yolo_weights:83 - INFO: conv2d_127
2021-10-07 06:19:47,015 - root:load_yolo_weights:87 - INFO: batch_normalization_124
2021-10-07 06:19:47,021 - root:load_yolo_weights:83 - INFO: conv2d_128
2021-10-07 06:19:47,021 - root:load_yolo_weights:87 - INFO: batch_normalization_125
2021-10-07 06:19:47,049 - root:load_yolo_weights:83 - INFO: conv2d_129
2021-10-07 06:19:47,050 - root:load_yolo_weights:87 - INFO: batch_normalization_126
2021-10-07 06:19:47,056 - root:load_yolo_weights:83 - INFO: conv2d_130
2021-10-07 06:19:47,057 - root:load_yolo_weights:87 - INFO: batch_normalization_127
2021-10-07 06:19:47,089 - root:load_yolo_weights:83 - INFO: conv2d_131
2021-10-07 06:19:47,090 - root:load_yolo_weights:87 - INFO: batch_normalization_128
2021-10-07 06:19:47,096 - root:load_yolo_weights:83 - INFO: conv2d_132
2021-10-07 06:19:47,096 - root:load_yolo_weights:87 - INFO: batch_normalization_129
2021-10-07 06:19:47,125 - root:load_yolo_weights:83 - INFO: conv2d_133
2021-10-07 06:19:47,125 - root:load_yolo_weights:87 - INFO: batch_normalization_130
2021-10-07 06:19:47,129 - root:load_yolo_weights:83 - INFO: conv2d_134
2021-10-07 06:19:47,129 - root:load_yolo_weights:87 - INFO: batch_normalization_130
2021-10-07 06:19:47,133 - root:load_yolo_weights:83 - INFO: conv2d_135
2021-10-07 06:19:47,133 - root:load_yolo_weights:87 - INFO: batch_normalization_131
2021-10-07 06:19:47,137 - root:load_yolo_weights:83 - INFO: conv2d_136
2021-10-07 06:19:47,137 - root:load_yolo_weights:87 - INFO: batch_normalization_132
2021-10-07 06:19:47,147 - root:load_yolo_weights:83 - INFO: conv2d_137
2021-10-07 06:19:47,148 - root:load_yolo_weights:87 - INFO: batch_normalization_133
2021-10-07 06:19:47,152 - root:load_yolo_weights:83 - INFO: conv2d_138
2021-10-07 06:19:47,152 - root:load_yolo_weights:87 - INFO: batch_normalization_134
2021-10-07 06:19:47,161 - root:load_yolo_weights:83 - INFO: conv2d_139
2021-10-07 06:19:47,162 - root:load_yolo_weights:87 - INFO: batch_normalization_135
2021-10-07 06:19:47,167 - root:load_yolo_weights:83 - INFO: conv2d_140
2021-10-07 06:19:47,167 - root:load_yolo_weights:87 - INFO: batch_normalization_136
2021-10-07 06:19:47,176 - root:load_yolo_weights:83 - INFO: conv2d_141
2021-10-07 06:19:47,176 - root:load_yolo_weights:87 - INFO: batch_normalization_137
2021-10-07 06:19:47,179 - root:load_yolo_weights:83 - INFO: conv2d_142
2021-10-07 06:19:47,179 - root:load_yolo_weights:87 - INFO: batch_normalization_137
2021-10-07 06:19:47,183 - root:load_yolo_weights:83 - INFO: conv2d_143
2021-10-07 06:19:47,183 - root:load_yolo_weights:87 - INFO: batch_normalization_138
2021-10-07 06:19:47,187 - root:load_yolo_weights:83 - INFO: conv2d_144
2021-10-07 06:19:47,187 - root:load_yolo_weights:87 - INFO: batch_normalization_139
2021-10-07 06:19:47,192 - root:load_yolo_weights:83 - INFO: conv2d_145
2021-10-07 06:19:47,192 - root:load_yolo_weights:87 - INFO: batch_normalization_140
2021-10-07 06:19:47,196 - root:load_yolo_weights:83 - INFO: conv2d_146
2021-10-07 06:19:47,196 - root:load_yolo_weights:87 - INFO: batch_normalization_141
2021-10-07 06:19:47,201 - root:load_yolo_weights:83 - INFO: conv2d_147
2021-10-07 06:19:47,201 - root:load_yolo_weights:87 - INFO: batch_normalization_142
2021-10-07 06:19:47,205 - root:load_yolo_weights:83 - INFO: conv2d_148
2021-10-07 06:19:47,205 - root:load_yolo_weights:87 - INFO: batch_normalization_143
2021-10-07 06:19:47,210 - root:load_yolo_weights:83 - INFO: conv2d_149
2021-10-07 06:19:47,210 - root:load_yolo_weights:87 - INFO: batch_normalization_144
2021-10-07 06:20:13,374 - uipath_core.training_plugin:trigger_full_training_and_publish_model:466 - ERROR: Failed to trigger full training and publish data, error: cannot compute Mul as input #1(zero-based) was expected to be a double tensor but is a string tensor [Op:Mul]
2021-10-07 06:20:13,375 - uipath_core.training_plugin:model_run:145 - ERROR: Training failed for pipeline type: FULL_TRAINING, error: cannot compute Mul as input #1(zero-based) was expected to be a double tensor but is a string tensor [Op:Mul]
2021-10-07 06:20:13,393 - uipath_core.trainer_run:main:81 - ERROR: Training Job failed, error: cannot compute Mul as input #1(zero-based) was expected to be a double tensor but is a string tensor [Op:Mul]
Traceback (most recent call last):
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/trainer_run.py”, line 76, in main
wrapper.run()
File “/microservice/training_wrapper.py”, line 57, in run
return self.training_plugin.model_run()
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 146, in model_run
raise e
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 134, in model_run
self.run_full_training()
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 167, in run_full_training
self.trigger_full_training_and_publish_model()
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 467, in trigger_full_training_and_publish_model
raise e
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 443, in trigger_full_training_and_publish_model
self.train_model(self.training_data_directory)
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 111, in train_model
self.model.train(directory)
File “/microservice/train.py”, line 52, in train
training.main(self.input_lr)
File “/microservice/training.py”, line 134, in main
results = train_step(image_data, target)
File “/microservice/training.py”, line 94, in train_step
lr = global_steps / warmup_steps * train_learningRate
File “/home/aifabric/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py”, line 997, in binary_op_wrapper
return func(x, y, name=name)
File “/home/aifabric/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py”, line 1283, in _mul_dispatch
return gen_math_ops.mul(x, y, name=name)
File “/home/aifabric/.local/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py”, line 6089, in mul
_ops.raise_from_not_ok_status(e, name)
File “/home/aifabric/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py”, line 6653, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File “”, line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: cannot compute Mul as input #1(zero-based) was expected to be a double tensor but is a string tensor [Op:Mul]

Hi

I got the below error

Full training of SNOW_Incident_Classification 2.0 launched - Run 762b39d7-7ff8-492c-93a4-e6d9fd90fac8
Full training of SNOW_Incident_Classification 2.0 scheduled - Run 762b39d7-7ff8-492c-93a4-e6d9fd90fac8
Full training of SNOW_Incident_Classification 2.0 started - Run 762b39d7-7ff8-492c-93a4-e6d9fd90fac8
Full training of SNOW_Incident_Classification 2.0 failed - Run 762b39d7-7ff8-492c-93a4-e6d9fd90fac8

Error Details : Pipeline failed due to ML Package Issue

2021-10-11 19:09:36,006 - uipath_core.trainer_run:main:66 - INFO: Starting training job…
2021-10-11 19:09:37,184 - transformers.file_utils::38 - INFO: PyTorch version 1.5.1+cu101 available.
2021-10-11 19:09:38,677 - matplotlib.font_manager:_rebuild:1423 - INFO: Generating new fontManager, this may take some time…
2021-10-11 19:09:59,087 - uipath_core.trainer_run:main:66 - INFO: Starting training job…
2021-10-11 19:10:00,251 - transformers.file_utils::38 - INFO: PyTorch version 1.5.1+cu101 available.
2021-10-11 19:10:01,734 - matplotlib.font_manager:_rebuild:1423 - INFO: Generating new fontManager, this may take some time…
2021-10-11 19:10:04,622 - uipath_core.logs.upload_log_service:upload_logs_file:63 - INFO: Retry Training Triggered:
2021-10-11 19:10:04,693 - uipath_core.storage.azure_storage_client:download:96 - INFO: Dataset from bucket folder training-726b3831-f6b8-4f9a-a037-296e655d67eb/b3a99338-6417-40de-8b8c-049b841d1b9e/dfc36f88-3678-483d-ae7e-cbb6442eeb31 with size 1 downloaded successfully
2021-10-11 19:10:04,695 - uipath_core.training_plugin:download_dataset:409 - WARNING: Deleting already existing folder name: /data/dataset/test
2021-10-11 19:10:04,764 - uipath_core.storage.azure_storage_client:download:96 - INFO: Dataset from bucket folder training-726b3831-f6b8-4f9a-a037-296e655d67eb/b3a99338-6417-40de-8b8c-049b841d1b9e/da264eb5-7b48-4b81-a8e5-9a71ccafe0b7 with size 1 downloaded successfully
2021-10-11 19:10:04,765 - uipath_core.training_plugin:process_data_model:121 - INFO: Start process model data…
2021-10-11 19:10:04,765 - uipath_core.training_plugin:initialize_model:103 - INFO: Start model initialization…
2021-10-11 19:10:04,765 - uipath_core.training_plugin:initialize_model:106 - INFO: Model initialized successfully
2021-10-11 19:10:04,766 - aiflib.data_manager:info:15 - INFO: Loading data from /data/dataset…
2021-10-11 19:10:04,771 - aiflib.data_manager:info:15 - INFO: Failed to read csv [/data/dataset/train.csv] exception:
‘utf-8’ codec can’t decode byte 0x96 in position 679: invalid start byte
2021-10-11 19:10:04,775 - aiflib.data_manager:info:15 - INFO: Failed to read csv [/data/dataset/test/test.csv] exception:
‘utf-8’ codec can’t decode byte 0x96 in position 679: invalid start byte
2021-10-11 19:10:04,776 - aiflib.data_manager:info:15 - INFO: Unable to read any valid data from *.csv files in [/data/dataset]
2021-10-11 19:10:04,776 - aiflib.data_manager:info:15 - INFO: Unable to read any valid data from *.json files in [/data/dataset]
2021-10-11 19:10:04,776 - uipath_core.training_plugin:process_data:425 - ERROR: Failed to process data, error: No valid data to run this pipeline.
2021-10-11 19:10:04,776 - uipath_core.training_plugin:model_run:145 - ERROR: Training failed for pipeline type: FULL_TRAINING, error: No valid data to run this pipeline.
2021-10-11 19:10:04,777 - uipath_core.trainer_run:main:81 - ERROR: Training Job failed, error: No valid data to run this pipeline.
Traceback (most recent call last):
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/trainer_run.py”, line 76, in main
wrapper.run()
File “/microservice/training_wrapper.py”, line 57, in run
return self.training_plugin.model_run()
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 146, in model_run
raise e
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 134, in model_run
self.run_full_training()
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 164, in run_full_training
self.process_data()
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 426, in process_data
raise e
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 423, in process_data
self.process_data_model(self.local_dataset_directory)
File “/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py”, line 122, in process_data_model
self.model.process_data(directory)
File “/microservice/train.py”, line 17, in process_data
self.model.process_data(directory)
File “/microservice/aiflib/model.py”, line 166, in process_data
raise UiPathUsageException(‘No valid data to run this pipeline.’)
aiflib.logger.UiPathUsageException: No valid data to run this pipeline.