Error: Pipeline failed due to ML Package Issue

Help AI Center
,
1

Full training of MultilingualTextClassification 4.0 launched - Run 76d43538-f839-4034-9f15-fd0f36052e5f
Full training of MultilingualTextClassification 4.0 scheduled - Run 76d43538-f839-4034-9f15-fd0f36052e5f
Full training of MultilingualTextClassification 4.0 started - Run 76d43538-f839-4034-9f15-fd0f36052e5f
Full training of MultilingualTextClassification 4.0 failed - Run 76d43538-f839-4034-9f15-fd0f36052e5f

Error Details : Pipeline failed due to ML Package Issue

2022-12-03 12:31:45,992 - uipath_core.trainer_run:main:73 - INFO: Starting training job…
2022-12-03 12:31:46,125 - matplotlib:_get_config_or_cache_dir:484 - WARNING: Matplotlib created a temporary config/cache directory at /tmp/matplotlib-5otx498p because the default path (/home/aicenter/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
2022-12-03 12:31:46,418 - matplotlib.font_manager:_load_fontmanager:1443 - INFO: generated new fontManager
2022-12-03 12:31:48,937 - uipath_core.storage.azure_storage_client:download:115 - INFO: Dataset from bucket folder training-884121af-b7a9-465d-b342-3a168d69b4f9/197a50ac-d1a6-4ae8-8488-c6f1d8506d2e/28c2e010-894f-4fb7-8d82-5d714e27bc92 with size 1 downloaded successfully
2022-12-03 12:31:48,938 - uipath_core.training_plugin:download_dataset:503 - WARNING: Deleting already existing folder name: /data/dataset/test
2022-12-03 12:31:48,978 - uipath_core.storage.azure_storage_client:download:115 - INFO: Dataset from bucket folder training-884121af-b7a9-465d-b342-3a168d69b4f9/197a50ac-d1a6-4ae8-8488-c6f1d8506d2e/28c2e010-894f-4fb7-8d82-5d714e27bc92 with size 1 downloaded successfully
2022-12-03 12:31:48,978 - uipath_core.training_plugin:process_data_model:141 - INFO: Start process model data…
2022-12-03 12:31:48,978 - uipath_core.training_plugin:initialize_model:114 - INFO: Start model initialization…
2022-12-03 12:31:48,981 - uipath_core.training_plugin:initialize_model:117 - INFO: Model initialized successfully
2022-12-03 12:31:48,981 - root:read_all_json:20 - INFO: Reading data from /microservice/dataset
2022-12-03 12:31:48,982 - root:read_all_json:20 - INFO: Reading data from /microservice/dataset/test
2022-12-03 12:31:48,983 - root:read_data:55 - INFO: Series(, Name: label, dtype: int64)
2022-12-03 12:31:48,984 - root:read_data:60 - INFO: Train: (0, 2), Test: (0, 2)
2022-12-03 12:31:48,984 - uipath_core.training_plugin:process_data_model:144 - INFO: Model process data successfully with response None
2022-12-03 12:31:48,984 - uipath_core.training_plugin:evaluate_model:130 - INFO: Start model evaluation…
2022-12-03 12:31:48,984 - root:eval_BERT:67 - INFO: Evaluating using trained BERT model
2022-12-03 12:31:48,984 - root:eval_BERT:68 - INFO: Evaluation dataset size 0
2022-12-03 12:31:48,984 - root:load:212 - INFO: Loading model…
2022-12-03 12:31:48,985 - uipath_core.training_plugin:trigger_full_training_and_publish_model:533 - ERROR: Failed to evaluate untrained model with error: [Errno 2] No such file or directory: ‘/microservice/models/test/model.p’
2022-12-03 12:31:48,985 - uipath_core.training_plugin:train_model:120 - INFO: Start model training…
2022-12-03 12:31:48,986 - uipath_core.training_plugin:trigger_full_training_and_publish_model:560 - ERROR: Failed to trigger full training and publish data, error: single positional indexer is out-of-bounds
2022-12-03 12:31:48,986 - uipath_core.training_plugin:model_run:169 - ERROR: Training failed for pipeline type: FULL_TRAINING, error: single positional indexer is out-of-bounds
2022-12-03 12:31:48,988 - uipath_core.trainer_run:main:90 - ERROR: Training Job failed, error: single positional indexer is out-of-bounds
Traceback (most recent call last):
File “/home/aicenter/.local/lib/python3.8/site-packages/uipath_core/trainer_run.py”, line 85, in main
wrapper.run()
File “/microservice/training_wrapper.py”, line 57, in run
return self.training_plugin.model_run()
File “/home/aicenter/.local/lib/python3.8/site-packages/uipath_core/training_plugin.py”, line 185, in model_run
raise ex
File “/home/aicenter/.local/lib/python3.8/site-packages/uipath_core/training_plugin.py”, line 157, in model_run
self.run_full_training()
File “/home/aicenter/.local/lib/python3.8/site-packages/uipath_core/training_plugin.py”, line 205, in run_full_training
self.trigger_full_training_and_publish_model()
File “/home/aicenter/.local/lib/python3.8/site-packages/uipath_core/training_plugin.py”, line 561, in trigger_full_training_and_publish_model
raise e
File “/home/aicenter/.local/lib/python3.8/site-packages/uipath_core/training_plugin.py”, line 537, in trigger_full_training_and_publish_model
self.train_model(self.training_data_directory)
File “/home/aicenter/.local/lib/python3.8/site-packages/uipath_core/training_plugin.py”, line 122, in train_model
response = self.model.train(directory)
File “/microservice/train.py”, line 53, in train
train.train(self.opt, self.df_train, self.df_test)
File “”, line 66, in train
File “”, line 24, in train_bert
File “”, line 93, in create_from_dataframes
File “”, line 28, in init
File “/home/aicenter/.local/lib/python3.8/site-packages/pandas/core/indexing.py”, line 967, in getitem
return self._getitem_axis(maybe_callable, axis=axis)
File “/home/aicenter/.local/lib/python3.8/site-packages/pandas/core/indexing.py”, line 1520, in _getitem_axis
self._validate_integer(key, axis)
File “/home/aicenter/.local/lib/python3.8/site-packages/pandas/core/indexing.py”, line 1452, in _validate_integer
raise IndexError(“single positional indexer is out-of-bounds”)
IndexError: single positional indexer is out-of-bounds

**I’m facing this issue while running the ML package MultilingualTextClassification.
Below are the steps I followed to train a model.

  1. Datasets → create a folder training and upload the csv file to this folder.
  2. ML Package → Create a ML Package for Multilingual model.
  3. Pipeline → After creating ML Package, create a pipeline for ML package and run the pipeline
    After this steps I got the error.
    Please help to resolve this issue**
    image
    image1358×598 55.6 KB
2

Hi @AbarnaKalaiselvam

Did you give atleast 5 samples? looks like the dataset is not enough can you increase the number of samples and try

cheers

3

why are you running a full pipeline with no evaluation dataset?

If you don’t have an evaluation dataset you can only run a train pipeline with your dataset. If you need a full pipeline run, please add more data and create an evaluation dataset.

Hope this helps.

1 Like
4

Train only of MultiLingualTextClassification 4.0 launched - Run da57fe9b-fa27-4fc0-af18-65f1d72ee5a2 Train only of MultiLingualTextClassification 4.0 scheduled - Run da57fe9b-fa27-4fc0-af18-65f1d72ee5a2 Train only of MultiLingualTextClassification 4.0 started - Run da57fe9b-fa27-4fc0-af18-65f1d72ee5a2 Train only of MultiLingualTextClassification 4.0 failed - Run da57fe9b-fa27-4fc0-af18-65f1d72ee5a2 Error Details : Pipeline failed due to ML Package Issue 2022-12-06 09:41:30,509 - uipath_core.trainer_run:main:73 - INFO: Starting training job… 2022-12-06 09:41:30,625 - matplotlib:_get_config_or_cache_dir:484 - WARNING: Matplotlib created a temporary config/cache directory at /tmp/matplotlib-ott_iww6 because the default path (/home/aicenter/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing. 2022-12-06 09:41:30,945 - matplotlib.font_manager:_load_fontmanager:1443 - INFO: generated new fontManager 2022-12-06 09:41:33,585 - uipath_core.storage.azure_storage_client:download:115 - INFO: Dataset from bucket folder training-884121af-b7a9-465d-b342-3a168d69b4f9/3d3526e9-1fb4-4456-bea1-29fc0d510f73/b77821d1-9939-4255-bbec-815d2a0ca3f7 with size 1 downloaded successfully 2022-12-06 09:41:33,585 - uipath_core.training_plugin:train_model:120 - INFO: Start model training… 2022-12-06 09:41:33,585 - uipath_core.training_plugin:initialize_model:114 - INFO: Start model initialization… 2022-12-06 09:41:33,624 - uipath_core.training_plugin:initialize_model:117 - INFO: Model initialized successfully 2022-12-06 09:41:33,624 - root:read_all_json:20 - INFO: Reading data from /microservice/dataset 2022-12-06 09:41:33,625 - root:read_all_json:20 - INFO: Reading data from /microservice/dataset/test 2022-12-06 09:41:33,627 - root:read_data:55 - INFO: Series(, Name: label, dtype: int64) 2022-12-06 09:41:33,628 - root:read_data:60 - INFO: Train: (0, 2), Test: (0, 2) 2022-12-06 09:41:33,629 - uipath_core.training_plugin:model_run:169 - ERROR: Training failed for pipeline type: TRAIN_ONLY, error: single positional indexer is out-of-bounds 2022-12-06 09:41:33,637 - uipath_core.trainer_run:main:90 - ERROR: Training Job failed, error: single positional indexer is out-of-bounds Traceback (most recent call last): File “/home/aicenter/.local/lib/python3.8/site-packages/uipath_core/trainer_run.py”, line 85, in main wrapper.run() File “/microservice/training_wrapper.py”, line 57, in run return self.training_plugin.model_run() File “/home/aicenter/.local/lib/python3.8/site-packages/uipath_core/training_plugin.py”, line 185, in model_run raise ex File “/home/aicenter/.local/lib/python3.8/site-packages/uipath_core/training_plugin.py”, line 161, in model_run self.run_train_only() File “/home/aicenter/.local/lib/python3.8/site-packages/uipath_core/training_plugin.py”, line 245, in run_train_only self.train_model(self.local_dataset_directory) File “/home/aicenter/.local/lib/python3.8/site-packages/uipath_core/training_plugin.py”, line 122, in train_model response = self.model.train(directory) File “/microservice/train.py”, line 53, in train train.train(self.opt, self.df_train, self.df_test) File “”, line 66, in train File “”, line 24, in train_bert File “”, line 93, in create_from_dataframes File “”, line 28, in init File “/home/aicenter/.local/lib/python3.8/site-packages/pandas/core/indexing.py”, line 967, in getitem return self._getitem_axis(maybe_callable, axis=axis) File “/home/aicenter/.local/lib/python3.8/site-packages/pandas/core/indexing.py”, line 1520, in _getitem_axis self._validate_integer(key, axis) File “/home/aicenter/.local/lib/python3.8/site-packages/pandas/core/indexing.py”, line 1452, in _validate_integer raise IndexError(“single positional indexer is out-of-bounds”) IndexError: single positional indexer is out-of-bounds

Creating Pipeline for Multilingual Text Classification

pipline1
pip2
var

Dataset

dataset
dataset1350×654 36.3 KB

Error : Pipeline failed due to ML Package Issue
please help to solve this issue

5

Hi @AbarnaKalaiselvam

I see the target column name is ‘target’ in the csv and ‘targett’ with double t in the variables you provided can you correct it please

and make sure you have minimum of 5 rows for each classification

cheers

6

sry, I mistakenly type it as double T…
In my data set there are 4000 rows

7

Hi @AbarnaKalaiselvam

So is it failing with the right column as well?

Did you place the csv directly under the training dataset only or did you create a sub folder under it?

for each type individually you have more than 5 I believe.

And are you running only training pipeline?

The error says it could not read anything from the input

cheers

8

Hi @Anil_G

image
image1265×404 27.6 KB

image
image948×439 20.6 KB

I created a folder training
Inside this folder csv file is located

My pipeline is train under the train run type

9

@AbarnaKalaiselvam

Please open the csv in a notepad an check if there are any issues

Alternately try saving the csv as utf-8 encoding and check

Let me replicate the steps and see if we are able to train now…We did train these previously and did not face issue like this

cheers

1 Like
10

Thank you @Anil_G
my issue solved, but it take 21h to train…

image
image1279×510 38.7 KB

11

Hi @AbarnaKalaiselvam

That Try enabling gpu that might decrease some time.

Are you on a community plan then it is a shared resource so it might take time

To Know the actual time go to ml logs and check when it started actually and how much time it took

cheers

12

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.