2021-10-27 20:48:54,558 - uipath_core.utils.utils:_retries:191 - WARNING: Function: download execution failed, retry count 1 2021-10-27 20:48:54,597 - uipath_core.storage.azure_storage_client:compose:136 - ERROR: Failed to compose file ['train.log.20211027204849'] to 540a1448-6630-4029-bd26-82140d92a9b6.log for bucket training-fef2150b-2243-439c-aebc-727d11f7fd49, error: [Errno 2] No such file or directory: 'trainer_run_logs/train.log.20211027204849' 2021-10-27 20:48:54,612 - uipath_core.storage.azure_storage_client:download:100 - ERROR: Failed to download blob: ['/'] from bucket: training-fef2150b-2243-439c-aebc-727d11f7fd49, error: Directory at 'd5d9e038-53f1-48e7-a44e-61fe59d7a7c2/adc7becb-2915-4a11-b0a1-375e9e5af603' is empty or not exist 2021-10-27 20:48:59,598 - uipath_core.utils.utils:_retries:191 - WARNING: Function: compose execution failed, retry count 1 2021-10-27 20:48:59,599 - uipath_core.storage.azure_storage_client:compose:136 - ERROR: Failed to compose file ['train.log.20211027204849'] to 540a1448-6630-4029-bd26-82140d92a9b6.log for bucket training-fef2150b-2243-439c-aebc-727d11f7fd49, error: [Errno 2] No such file or directory: 'trainer_run_logs/train.log.20211027204849' 2021-10-27 20:48:59,618 - uipath_core.utils.utils:_retries:191 - WARNING: Function: download execution failed, retry count 2 2021-10-27 20:48:59,680 - uipath_core.storage.azure_storage_client:download:100 - ERROR: Failed to download blob: ['/'] from bucket: training-fef2150b-2243-439c-aebc-727d11f7fd49, error: Directory at 'd5d9e038-53f1-48e7-a44e-61fe59d7a7c2/adc7becb-2915-4a11-b0a1-375e9e5af603' is empty or not exist 2021-10-27 20:49:04,605 - uipath_core.utils.utils:_retries:191 - WARNING: Function: compose execution failed, retry count 2 2021-10-27 20:49:04,607 - uipath_core.storage.azure_storage_client:compose:136 - ERROR: Failed to compose file ['train.log.20211027204849'] to 540a1448-6630-4029-bd26-82140d92a9b6.log for bucket training-fef2150b-2243-439c-aebc-727d11f7fd49, error: [Errno 2] No such file or directory: 'trainer_run_logs/train.log.20211027204849' 2021-10-27 20:49:04,685 - uipath_core.utils.utils:_retries:189 - ERROR: Function: download excecution failed, Max retry limit reached 3 2021-10-27 20:49:04,686 - uipath_core.training_plugin:download_dataset:416 - ERROR: AI_Center_Issue: Failed to download bucket: training-fef2150b-2243-439c-aebc-727d11f7fd49 dataset at dataset, error: Directory at 'd5d9e038-53f1-48e7-a44e-61fe59d7a7c2/adc7becb-2915-4a11-b0a1-375e9e5af603' is empty or not exist 2021-10-27 20:49:04,686 - uipath_core.training_plugin:model_run:145 - ERROR: Training failed for pipeline type: EVALUATE_ONLY, error: Directory at 'd5d9e038-53f1-48e7-a44e-61fe59d7a7c2/adc7becb-2915-4a11-b0a1-375e9e5af603' is empty or not exist 2021-10-27 20:49:04,687 - uipath_core.trainer_run:main:81 - ERROR: Training Job failed, error: Directory at 'd5d9e038-53f1-48e7-a44e-61fe59d7a7c2/adc7becb-2915-4a11-b0a1-375e9e5af603' is empty or not exist Traceback (most recent call last): File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/trainer_run.py", line 76, in main wrapper.run() File "/microservice/training_wrapper.py", line 57, in run return self.training_plugin.model_run() File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py", line 146, in model_run raise e File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py", line 136, in model_run self.run_evaluation_only() File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py", line 173, in run_evaluation_only self.download_dataset(self.EVALUATION_FOLDER, self.local_dataset_directory) File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py", line 418, in download_dataset raise e File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py", line 413, in download_dataset self.storage_client.download(self.BUCKET_NAME, bucket_folder_name, local_directory) File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/utils/utils.py", line 190, in _retries raise e File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/utils/utils.py", line 180, in _retries result = fn(*args, **kwargs) File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/storage/azure_storage_client.py", line 101, in download raise e File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/storage/azure_storage_client.py", line 93, in download raise Exception("Directory at %r is empty or not exist" % (bucket_directory_name)) Exception: Directory at 'd5d9e038-53f1-48e7-a44e-61fe59d7a7c2/adc7becb-2915-4a11-b0a1-375e9e5af603' is empty or not exist 2021-10-27 20:49:12,026 - uipath_core.trainer_run:main:66 - INFO: Starting training job... 2021-10-27 20:49:13,387 - transformers.file_utils::38 - INFO: PyTorch version 1.5.1+cu101 available. 2021-10-27 20:49:15,020 - matplotlib.font_manager:_rebuild:1423 - INFO: Generating new fontManager, this may take some time... 2021-10-27 20:49:32,770 - uipath_core.utils.utils:_retries:189 - ERROR: Function: download excecution failed, Max retry limit reached 3 2021-10-27 20:49:32,770 - uipath_core.training_plugin:download_dataset:416 - ERROR: AI_Center_Issue: Failed to download bucket: training-fef2150b-2243-439c-aebc-727d11f7fd49 dataset at dataset, error: Directory at 'd5d9e038-53f1-48e7-a44e-61fe59d7a7c2/adc7becb-2915-4a11-b0a1-375e9e5af603' is empty or not exist 2021-10-27 20:49:32,770 - uipath_core.training_plugin:model_run:145 - ERROR: Training failed for pipeline type: EVALUATE_ONLY, error: Directory at 'd5d9e038-53f1-48e7-a44e-61fe59d7a7c2/adc7becb-2915-4a11-b0a1-375e9e5af603' is empty or not exist 2021-10-27 20:49:32,771 - uipath_core.trainer_run:main:81 - ERROR: Training Job failed, error: Directory at 'd5d9e038-53f1-48e7-a44e-61fe59d7a7c2/adc7becb-2915-4a11-b0a1-375e9e5af603' is empty or not exist Traceback (most recent call last): File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/trainer_run.py", line 76, in main wrapper.run() File "/microservice/training_wrapper.py", line 57, in run return self.training_plugin.model_run() File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py", line 146, in model_run raise e File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py", line 136, in model_run self.run_evaluation_only() File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py", line 173, in run_evaluation_only self.download_dataset(self.EVALUATION_FOLDER, self.local_dataset_directory) File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py", line 418, in download_dataset raise e File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/training_plugin.py", line 413, in download_dataset self.storage_client.download(self.BUCKET_NAME, bucket_folder_name, local_directory) File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/utils/utils.py", line 190, in _retries raise e File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/utils/utils.py", line 180, in _retries result = fn(*args, **kwargs) File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/storage/azure_storage_client.py", line 101, in download raise e File "/home/aifabric/.local/lib/python3.6/site-packages/uipath_core/storage/azure_storage_client.py", line 93, in download raise Exception("Directory at %r is empty or not exist" % (bucket_directory_name)) Exception: Directory at 'd5d9e038-53f1-48e7-a44e-61fe59d7a7c2/adc7becb-2915-4a11-b0a1-375e9e5af603' is empty or not exist