Pipeline failed due to ML Package Issue

[Jack_Moran]

This was the result of a full pipeline run, where I used two spreadsheets to feed in Training and Evaluation datasets. I am trying to train the MultiLingualTextClassification Model. Logs can be seen below:

2023-08-03 14:15:19,918 - UiPath_core.trainer_run:main:74 - INFO: Starting training job…
2023-08-03 14:15:20,069 - matplotlib:_get_config_or_cache_dir:484 - WARNING: Matplotlib created a temporary config/cache directory at /tmp/matplotlib-tty1em4y because the default path (/home/aicenter/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
2023-08-03 14:15:20,451 - matplotlib.font_manager:_load_fontmanager:1443 - INFO: generated new fontManager
2023-08-03 14:15:23,421 - UiPath_core.storage.azure_storage_client:download:118 - INFO: Dataset from bucket folder training-a0394d57-9d8b-4ee7-9b15-97c89266114e/cdd1a17c-7e4d-46ef-ad79-4c48ab6eac9e/94a4d594-4772-4c95-80ae-7a47186275bb with size 1 downloaded successfully
2023-08-03 14:15:23,421 - UiPath_core.training_plugin:download_dataset:513 - WARNING: Deleting already existing folder name: /data/dataset/test
2023-08-03 14:15:23,438 - UiPath_core.storage.azure_storage_client:download:118 - INFO: Dataset from bucket folder training-a0394d57-9d8b-4ee7-9b15-97c89266114e/cdd1a17c-7e4d-46ef-ad79-4c48ab6eac9e/6bff1d99-16e4-40e3-a719-ae8722ff3960 with size 1 downloaded successfully
2023-08-03 14:15:23,439 - UiPath_core.training_plugin:process_data_model:151 - INFO: Start process model data…
2023-08-03 14:15:23,439 - UiPath_core.training_plugin:initialize_model:124 - INFO: Start model initialization…
2023-08-03 14:15:23,441 - UiPath_core.training_plugin:initialize_model:127 - INFO: Model initialized successfully
2023-08-03 14:15:23,441 - root:init:70 - INFO: Loading data from /microservice/dataset
2023-08-03 14:15:23,442 - root:read_all_directories:269 - INFO: Reading from directory stucture [/microservice/dataset]
2023-08-03 14:15:23,443 - root:read_all_data:123 - INFO: Unable to read any valid data from *.csv files in [/microservice/dataset]
2023-08-03 14:15:23,443 - root:read_all_data:128 - INFO: Unable to read any valid data from *.json files in [/microservice/dataset]
2023-08-03 14:15:23,443 - root:read_all_data:133 - INFO: Unable to read any valid data from directory structure [/microservice/dataset]
2023-08-03 14:15:23,443 - root:init:84 - INFO: Done read [0] points with [0] classes
2023-08-03 14:15:23,444 - root:validate:300 - INFO: Series(, Name: Subcategory, dtype: int64)
2023-08-03 14:15:23,445 - root:init:93 - INFO: Loading test data from /microservice/dataset/test
2023-08-03 14:15:23,445 - root:read_all_directories:269 - INFO: Reading from directory stucture [/microservice/dataset/test]
2023-08-03 14:15:23,446 - root:read_all_data:123 - INFO: Unable to read any valid data from *.csv files in [/microservice/dataset/test]
2023-08-03 14:15:23,446 - root:read_all_data:128 - INFO: Unable to read any valid data from *.json files in [/microservice/dataset/test]
2023-08-03 14:15:23,446 - root:read_all_data:133 - INFO: Unable to read any valid data from directory structure [/microservice/dataset/test]
2023-08-03 14:15:23,447 - root:init:104 - INFO: Train: 0, Test: 0
2023-08-03 14:15:23,447 - UiPath_core.training_plugin:process_data_model:154 - INFO: Model process data successfully with response None
2023-08-03 14:15:23,447 - UiPath_core.training_plugin:evaluate_model:140 - INFO: Start model evaluation…
2023-08-03 14:15:23,447 - root:eval_BERT:67 - INFO: Evaluating using trained BERT model
2023-08-03 14:15:23,447 - root:eval_BERT:68 - INFO: Evaluation dataset size 0
2023-08-03 14:15:23,447 - root:load:212 - INFO: Loading model…
2023-08-03 14:15:23,447 - UiPath_core.training_plugin:trigger_full_training_and_publish_model:543 - ERROR: Failed to evaluate untrained model with error: [Errno 2] No such file or directory: ‘/microservice/models/test/model.p’
2023-08-03 14:15:23,447 - UiPath_core.training_plugin:train_model:130 - INFO: Start model training…
2023-08-03 14:15:23,448 - UiPath_core.training_plugin:trigger_full_training_and_publish_model:580 - ERROR: Failed to trigger full training and publish data, error: single positional indexer is out-of-bounds
2023-08-03 14:15:23,449 - UiPath_core.training_plugin:model_run:179 - ERROR: Training failed for pipeline type: FULL_TRAINING, error: single positional indexer is out-of-bounds
2023-08-03 14:15:23,450 - UiPath_core.trainer_run:main:91 - ERROR: Training Job failed, error: single positional indexer is out-of-bounds
Traceback (most recent call last):
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/trainer_run.py”, line 86, in main
wrapper.run()
File “/microservice/training_wrapper.py”, line 58, in run
return self.training_plugin.model_run()
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 195, in model_run
raise ex
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 167, in model_run
self.run_full_training()
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 215, in run_full_training
self.trigger_full_training_and_publish_model()
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 581, in trigger_full_training_and_publish_model
raise e
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 554, in trigger_full_training_and_publish_model
self.train_model(self.training_data_directory)
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 132, in train_model
response = self.model.train(directory)
File “/microservice/train.py”, line 53, in train
train.train(self.opt, self.df_train, self.df_test)
File “”, line 66, in train
File “”, line 24, in train_bert
File “”, line 93, in create_from_dataframes
File “”, line 28, in init
File “/home/aicenter/.local/lib/python3.8/site-packages/pandas/core/indexing.py”, line 967, in getitem
return self._getitem_axis(maybe_callable, axis=axis)
File “/home/aicenter/.local/lib/python3.8/site-packages/pandas/core/indexing.py”, line 1520, in _getitem_axis
self._validate_integer(key, axis)
File “/home/aicenter/.local/lib/python3.8/site-packages/pandas/core/indexing.py”, line 1452, in _validate_integer
raise IndexError(“single positional indexer is out-of-bounds”)
IndexError: single positional indexer is out-of-bounds
2023-08-03 14:15:23,450 - UiPath_core.trainer_run:main:98 - INFO: Job run stopped.

[Anil_G (Anil Gorthi)]

@Jack_Moran

As per error we can see the evaluationd ataset size is zero

Also can you try running the training pipeline alone…and check please…is the file created as per the gicen doc? Or does it have any issues?

Cheers

[Jack_Moran]

I am running the training pipeline now. I set the format environment variable to auto, so I thought it could read the spreadsheet columns in… but apparently not! The evaluation set had several rows of data, and I even double checked to make sure the environment variables for input and target corresponded to the headers.

What is the gicen doc you mentioned? I have never heard of it.

[Anil_G (Anil Gorthi)]

@Jack_Moran

Please check the dataset format to be give

https://docs.uipath.com/ai-center/standalone/2021.10/user-guide/multi-lingual-text-classification#dataset-format

Cheers

[Jack_Moran]

I converted the format to CSV, and re-created the pipeline, but it failed to read the data in once more I made sure the format was set to auto too, and the input and output headers matched perfectly. What else could I be missing from the formatting guidelines?

[Anil_G (Anil Gorthi)]

@Jack_Moran

Is the error same? can you show the pipeline parameters you used and for each type did you give atleast 5 samples?

cheers

[Jack_Moran]

Yes, the error is identical. You can see the parameters below, where Description and Subcategory are the two columns intended for use. Each column has at least 40 samples in both training and validation data.

[Anil_G (Anil Gorthi)]

@Jack_Moran

Can you do only training for now please

And while selecting dataset did you select it properly

Cheers

[Jack_Moran]

Training pipeline failed as well, and I made sure I was using the correct dataset.

[Anil_G (Anil Gorthi)]

@Jack_Moran

Can you please show the detailed exception please

and also how you gave the dataset? a screenshot will help…along with where the file is placed

cheers

[Jack_Moran]

Here is a screenshot of what the Dataset looks like, as well as where it is placed in AI Center. The logs are also included, but they are more or less the same as before.

Logs:

Train only of MLTC_Pack 4.0 launched - Run ad33956d-2c7a-4c35-9e91-c45f301eae7b
Train only of MLTC_Pack 4.0 started - Run ad33956d-2c7a-4c35-9e91-c45f301eae7b
Train only of MLTC_Pack 4.0 scheduled - Run ad33956d-2c7a-4c35-9e91-c45f301eae7b
Train only of MLTC_Pack 4.0 failed - Run ad33956d-2c7a-4c35-9e91-c45f301eae7b

Error Details : Pipeline failed due to ML Package Issue

2023-08-04 13:21:38,220 - UiPath_core.trainer_run:main:74 - INFO: Starting training job…
2023-08-04 13:21:38,613 - matplotlib:_get_config_or_cache_dir:484 - WARNING: Matplotlib created a temporary config/cache directory at /tmp/matplotlib-7zyky9_f because the default path (/home/aicenter/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
2023-08-04 13:21:38,998 - matplotlib.font_manager:_load_fontmanager:1443 - INFO: generated new fontManager
2023-08-04 13:21:42,025 - UiPath_core.storage.azure_storage_client:download:118 - INFO: Dataset from bucket folder training-a0394d57-9d8b-4ee7-9b15-97c89266114e/cdd1a17c-7e4d-46ef-ad79-4c48ab6eac9e/94a4d594-4772-4c95-80ae-7a47186275bb with size 1 downloaded successfully
2023-08-04 13:21:42,025 - UiPath_core.training_plugin:train_model:130 - INFO: Start model training…
2023-08-04 13:21:42,026 - UiPath_core.training_plugin:initialize_model:124 - INFO: Start model initialization…
2023-08-04 13:21:42,030 - UiPath_core.training_plugin:initialize_model:127 - INFO: Model initialized successfully
2023-08-04 13:21:42,030 - root:init:70 - INFO: Loading data from /microservice/dataset
2023-08-04 13:21:42,309 - root:read_all_csv:162 - INFO: Failed to read csv [/microservice/dataset/RogersTrainingData.csv] exception:‘utf-8’ codec can’t decode byte 0x92 in position 202: invalid start byte
2023-08-04 13:21:42,310 - root:read_all_directories:269 - INFO: Reading from directory stucture [/microservice/dataset]
2023-08-04 13:21:42,311 - root:read_all_data:123 - INFO: Unable to read any valid data from *.csv files in [/microservice/dataset]
2023-08-04 13:21:42,311 - root:read_all_data:128 - INFO: Unable to read any valid data from *.json files in [/microservice/dataset]
2023-08-04 13:21:42,311 - root:read_all_data:133 - INFO: Unable to read any valid data from directory structure [/microservice/dataset]
2023-08-04 13:21:42,311 - root:init:84 - INFO: Done read [0] points with [0] classes
2023-08-04 13:21:42,312 - root:validate:300 - INFO: Series(, Name: Subcategory, dtype: int64)
2023-08-04 13:21:42,313 - root:init:97 - INFO: Split test data from train data
2023-08-04 13:21:42,313 - UiPath_core.training_plugin:model_run:179 - ERROR: Training failed for pipeline type: TRAIN_ONLY, error: With n_samples=0, test_size=0.19999999999999996 and train_size=None, the resulting train set will be empty. Adjust any of the aforementioned parameters.
2023-08-04 13:21:42,314 - UiPath_core.trainer_run:main:91 - ERROR: Training Job failed, error: With n_samples=0, test_size=0.19999999999999996 and train_size=None, the resulting train set will be empty. Adjust any of the aforementioned parameters.
Traceback (most recent call last):
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/trainer_run.py”, line 86, in main
wrapper.run()
File “/microservice/training_wrapper.py”, line 58, in run
return self.training_plugin.model_run()
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 195, in model_run
raise ex
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 171, in model_run
self.run_train_only()
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 255, in run_train_only
self.train_model(self.local_dataset_directory)
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 132, in train_model
response = self.model.train(directory)
File “/microservice/train.py”, line 48, in train
self.process_data()
File “/microservice/train.py”, line 177, in process_data
datamanager = DataManager(self.opt, allow_single_file=True, split=split)
File “”, line 99, in init
File “”, line 314, in split
File “/home/aicenter/.local/lib/python3.8/site-packages/sklearn/model_selection/_split.py”, line 2433, in train_test_split
n_train, n_test = _validate_shuffle_split(
File “/home/aicenter/.local/lib/python3.8/site-packages/sklearn/model_selection/_split.py”, line 2111, in _validate_shuffle_split
raise ValueError(
ValueError: With n_samples=0, test_size=0.19999999999999996 and train_size=None, the resulting train set will be empty. Adjust any of the aforementioned parameters.
2023-08-04 13:21:42,315 - UiPath_core.trainer_run:main:98 - INFO: Job run stopped.

[Anil_G (Anil Gorthi)]

@Jack_Moran

Please try creating a different csv file and use…looks like there is some issue with encoding or data in your current csv …the error is about the file not bein gable to be read…to start with I would suggest try with a smaller set may be 10 rows and only two type …mostly as per 0x92 looks like the single quote is creating the issue…try to remove them and see if it works…dont have single and double quotes to start with

Then start adding data once the issue is fixed

INFO: Failed to read csv [/microservice/dataset/RogersTrainingData.csv] exception:‘utf-8’ codec can’t decode byte 0x92 in position 202: invalid start byte

Cheerd

[Jack_Moran]

Unforunately, after running a pipeline with 10 items, I got the error below. I ran another pipeline with another 10 items, and got the same error again. I verified that the data had no single quotes in it, but this happened anyway! What now?

Train only of MLTC_Pack 4.0 launched - Run c217d29a-c2ee-4569-81e8-7f65f4ea2681
Train only of MLTC_Pack 4.0 started - Run c217d29a-c2ee-4569-81e8-7f65f4ea2681
Train only of MLTC_Pack 4.0 scheduled - Run c217d29a-c2ee-4569-81e8-7f65f4ea2681
Train only of MLTC_Pack 4.0 failed - Run c217d29a-c2ee-4569-81e8-7f65f4ea2681

Error Details : Pipeline failed due to ML Package Issue

2023-08-06 23:39:26,859 - UiPath_core.trainer_run:main:74 - INFO: Starting training job…
2023-08-06 23:39:27,013 - matplotlib:_get_config_or_cache_dir:484 - WARNING: Matplotlib created a temporary config/cache directory at /tmp/matplotlib-lwivspq0 because the default path (/home/aicenter/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
2023-08-06 23:39:27,390 - matplotlib.font_manager:_load_fontmanager:1443 - INFO: generated new fontManager
2023-08-06 23:39:30,194 - UiPath_core.storage.azure_storage_client:download:118 - INFO: Dataset from bucket folder training-a0394d57-9d8b-4ee7-9b15-97c89266114e/cdd1a17c-7e4d-46ef-ad79-4c48ab6eac9e/0d6de6ae-95c4-4334-8233-89888c84f39d with size 1 downloaded successfully
2023-08-06 23:39:30,194 - UiPath_core.training_plugin:train_model:130 - INFO: Start model training…
2023-08-06 23:39:30,194 - UiPath_core.training_plugin:initialize_model:124 - INFO: Start model initialization…
2023-08-06 23:39:30,196 - UiPath_core.training_plugin:initialize_model:127 - INFO: Model initialized successfully
2023-08-06 23:39:30,196 - root:init:70 - INFO: Loading data from /microservice/dataset
2023-08-06 23:39:30,486 - root:read_all_csv:179 - INFO: Read [10] data points from [/microservice/dataset/New Microsoft CSV.csv]
2023-08-06 23:39:30,487 - root:read_all_directories:269 - INFO: Reading from directory stucture [/microservice/dataset]
2023-08-06 23:39:30,506 - root:read_all_data:128 - INFO: Unable to read any valid data from *.json files in [/microservice/dataset]
2023-08-06 23:39:30,507 - root:read_all_data:133 - INFO: Unable to read any valid data from directory structure [/microservice/dataset]
2023-08-06 23:39:30,507 - root:init:84 - INFO: Done read [10] points with [4] classes
2023-08-06 23:39:30,508 - root:validate:300 - INFO: Subcategory
ERP 4
Network (Internal/Intranet) 1
Personal Computer Software 4
Production Software 1
Name: Subcategory, dtype: int64
2023-08-06 23:39:30,509 - root:validate:310 - INFO: Provided data does not have enough training examples to train, you must provide at least 1 training examples of each class
2023-08-06 23:39:30,510 - UiPath_core.training_plugin:model_run:179 - ERROR: Training failed for pipeline type: TRAIN_ONLY, error:
2023-08-06 23:39:30,510 - UiPath_core.trainer_run:main:91 - ERROR: Training Job failed, error:
Traceback (most recent call last):
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/trainer_run.py”, line 86, in main
wrapper.run()
File “/microservice/training_wrapper.py”, line 58, in run
return self.training_plugin.model_run()
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 195, in model_run
raise ex
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 171, in model_run
self.run_train_only()
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 255, in run_train_only
self.train_model(self.local_dataset_directory)
File “/home/aicenter/.local/lib/python3.8/site-packages/UiPath_core/training_plugin.py”, line 132, in train_model
response = self.model.train(directory)
File “/microservice/train.py”, line 48, in train
self.process_data()
File “/microservice/train.py”, line 177, in process_data
datamanager = DataManager(self.opt, allow_single_file=True, split=split)
File “”, line 87, in init
AssertionError
2023-08-06 23:39:30,511 - UiPath_core.trainer_run:main:98 - INFO: Job run stopped.

[Anil_G (Anil Gorthi)]

@Jack_Moran

As per error it has read the data now…and it could find different categories as well…but the sames are less…try givigng 5 each…like in 10 rows 5 should be same category and another 5 should be another category…in the current one netwoek and production has only 1 sample which is very less

ERP 4
Network (Internal/Intranet) 1
Personal Computer Software 4
Production Software 1
Name: Subcategory, dtype: int64
2023-08-06 23:39:30,509 - root:validate:310 - INFO: Provided data does not have enough training examples to train, you must provide at least 1 training examples of each class

Hope this helps

Cheers

[Jack_Moran]

I finally figured out the original error log! I didn’t realize that the error log was referring to an ASCII code, so when you said single quotes, I thought you meant the single quotes on my keyboard. Once I used the ASCII code to find the actual single quote in question, I was able to remove it from my data. I kept using this method to remove illegal characters until my data was acceptable.

However, as we discussed above, each category needs at least 5 examples… and sadly, I do not have enough data for each category to achieve this. Is there a way to get around this requirement or shut it off? Or do I have to get more data?

[Anil_G (Anil Gorthi)]

@Jack_Moran

One work around would be to use the same data multiple times if you dont have enough samples

Like repeat the same row 5 times and that also should work

Hope this helps…

If it resolves please close the topic else happy to help

Cheers

[Jack_Moran]

Thanks to @Anil_G for helping me find the solution! There were two things I needed to do:

First, I needed to pay attention to this error message:

INFO: Failed to read csv [/microservice/dataset/RogersTrainingData.csv] exception:‘utf-8’ codec can’t decode byte 0x92 in position 202: invalid start byte

0x92 is an ASCII code, so by Googling ‘0x92 ASCII’, I found the specific special character that was causing the issue. I used some Excel shortcuts to remove the character from my data. I got more errors similar to this, but the solution was the same: look for the ASCII code in the error message, Google it, remove it, and run the pipeline with the updated data.

After that, I had an issue where I did not have enough samples for certain labels in my data. For this, my solution was to manipulate the data to make sure there were at least 5 samples for each label. In my case, I simply duplicated data points within these labels to hit 5 samples.