2021-03-23 13:59:16,448 - uipath_core.trainer_run:main:66 - INFO: Starting training job... 2021-03-23 13:59:24,418 - uipath_core.storage.azure_storage_client:download:96 - INFO: Dataset from bucket folder training-7b66d190-b3d4-4b4f-bae9-d12ada9364ba/34524b99-e06d-4ff3-8b53-9574701a7676/bb7f427f-2112-4e2e-a33d-03457383dc1d with size 28 downloaded successfully 2021-03-23 13:59:24,419 - uipath_core.training_plugin:train_model:104 - INFO: Start model training... 2021-03-23 13:59:24,419 - uipath_core.training_plugin:initialize_model:98 - INFO: Start model initialization... 2021-03-23 13:59:24,420 - root:configure_options:109 - INFO: Document type invoices_india language: en 2021-03-23 13:59:24,421 - uipath_core.training_plugin:initialize_model:101 - INFO: Model initialized successfully 2021-03-23 13:59:24,422 - root:configure_options:109 - INFO: Document type invoices_india language: en 2021-03-23 13:59:24,422 - root:preprocess_data:457 - INFO: Creating dataset for document type invoices_india.. 2021-03-23 13:59:24,439 - root:preprocess_data:460 - INFO: Doctype invoices_india Statistics: 2021-03-23 13:59:24,495 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice4.pdf_0.jpg.json 2021-03-23 13:59:24,690 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 13:59:24,691 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 13:59:24,691 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 13:59:24,691 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 13:59:24,691 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 13:59:24,691 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 13:59:25,214 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 13:59:25,214 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 13:59:25,214 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 13:59:25,214 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 13:59:25,214 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 13:59:25,214 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 13:59:25,214 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 13:59:25,348 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 13:59:25,349 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 13:59:25,349 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 13:59:25,349 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 13:59:25,349 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 13:59:25,349 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 13:59:27,294 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_OD220514724105351000 (1).pdf_0.jpg.json 2021-03-23 13:59:27,417 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 13:59:27,418 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 13:59:27,418 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 13:59:27,418 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 13:59:27,418 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 13:59:27,418 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 13:59:27,418 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 13:59:27,545 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 13:59:27,546 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 13:59:27,546 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 13:59:27,546 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 13:59:27,546 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 13:59:27,546 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 13:59:27,546 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 13:59:27,546 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 13:59:27,546 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 13:59:27,633 - root:preprocess_data:523 - INFO: train: (11, 12) 2021-03-23 13:59:27,634 - root:preprocess_data:524 - INFO: train: (2, 12) 2021-03-23 13:59:27,634 - root:preprocess_data:529 - INFO: ratio: 0.153846 2021-03-23 13:59:27,636 - root:load_glove_vocabulary:90 - INFO: Loading offsets for en 2021-03-23 13:59:29,665 - root:build_vocabulary:129 - INFO: Building vocabulary... 2021-03-23 13:59:29,668 - root:build_embeddings:218 - INFO: Building embeddings... 2021-03-23 13:59:29,720 - root:save:71 - INFO: Saving invoices_india doctype object at /microservice/dataset/doctype.p... 2021-03-23 13:59:29,750 - root:save:241 - INFO: Saving dataset object at /microservice/dataset/dataset.p... 2021-03-23 13:59:29,754 - root:__init__:147 - INFO: Dataset creation done. Contains doctypes: ['invoices_india']. 2021-03-23 13:59:29,754 - root:train:147 - INFO: Training extraction model... 2021-03-23 13:59:29,754 - root:__init__:61 - INFO: Finetuning from base model: multi_task... 2021-03-23 13:59:29,755 - root:load:230 - INFO: Loading model multi_task... 2021-03-23 13:59:29,758 - root:create_network:127 - INFO: Building network ... 2021-03-23 13:59:29,758 - root:__init__:28 - INFO: Creating new network ... 2021-03-23 13:59:30,035 - root:load_network:265 - INFO: Loading network ... 2021-03-23 13:59:30,111 - root:create_network:127 - INFO: Building network ... 2021-03-23 13:59:30,112 - root:__init__:28 - INFO: Creating new network ... 2021-03-23 13:59:30,370 - root:create_network:130 - INFO: Cloning heads from base model multi_task... 2021-03-23 13:59:30,415 - root:__init__:91 - INFO: Model creation done. Contains tasks ['invoices_india', 'all'] 2021-03-23 13:59:30,416 - root:train:381 - INFO: Creating torch datasets ... 2021-03-23 13:59:30,579 - root:train_first_stage:400 - INFO: First stage training started... 2021-03-23 13:59:30,582 - root:train_first_stage:404 - INFO: Prep first stage done 2021-03-23 13:59:30,582 - root:_train:451 - INFO: Training all tasks for 100 epochs from epoch 0... 2021-03-23 13:59:30,582 - root:create_optimizer:183 - INFO: Building first_stage optimizer for task all... 2021-03-23 13:59:30,586 - root:create_scheduler:202 - INFO: Building first_stage scheduler for task all... 2021-03-23 13:59:30,586 - root:_train:460 - INFO: Starting Training Loop ... 2021-03-23 13:59:49,166 - root:save:212 - INFO: Saving model... 2021-03-23 13:59:49,687 - root:save:216 - INFO: Model saved 2021-03-23 13:59:49,688 - root:_log_to_console:1299 - INFO: [Task: all][Epoch: 001][tag_f1.invoices_india: 0.3616/0.9332][tag_f1.all: 0.3616/0.9332][lr.all: 0.001000] 2021-03-23 13:59:49,688 - root:_log_to_console:1303 - INFO: [Found best model][tag_f1.all: 0.9332][tag_f1.invoices_india: 0.9332] 2021-03-23 13:59:50,338 - root:train:170 - ERROR: multi_task Extraction Model Training Failed Traceback (most recent call last): File "", line 154, in train File "", line 38, in train_tag File "", line 385, in train File "", line 404, in train_first_stage File "", line 470, in _train File "", line 1237, in _track_aux_metrics File "", line 797, in calculate_extraction_scores File "", line 1426, in calculate_extraction_F1 File "/opt/conda/lib/python3.7/site-packages/pandas/core/frame.py", line 7548, in apply return op.get_result() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 180, in get_result return self.apply_standard() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 271, in apply_standard results, res_index = self.apply_series_generator() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 300, in apply_series_generator results[i] = self.f(v) File "", line 1425, in File "", line 769, in calculate_row_F1 File "", line 736, in prepare_row_for_scoring AttributeError: 'list' object has no attribute 'get_value' 2021-03-23 13:59:50,341 - uipath_core.training_plugin:model_run:140 - ERROR: Training failed for pipeline type: TRAIN_ONLY, error: multi_task Extraction Model Training Failed 2021-03-23 13:59:50,345 - uipath_core.trainer_run:main:81 - ERROR: Training Job failed, error: multi_task Extraction Model Training Failed Traceback (most recent call last): File "", line 154, in train File "", line 38, in train_tag File "", line 385, in train File "", line 404, in train_first_stage File "", line 470, in _train File "", line 1237, in _track_aux_metrics File "", line 797, in calculate_extraction_scores File "", line 1426, in calculate_extraction_F1 File "/opt/conda/lib/python3.7/site-packages/pandas/core/frame.py", line 7548, in apply return op.get_result() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 180, in get_result return self.apply_standard() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 271, in apply_standard results, res_index = self.apply_series_generator() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 300, in apply_series_generator results[i] = self.f(v) File "", line 1425, in File "", line 769, in calculate_row_F1 File "", line 736, in prepare_row_for_scoring AttributeError: 'list' object has no attribute 'get_value' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/trainer_run.py", line 76, in main wrapper.run() File "/microservice/training_wrapper.py", line 57, in run return self.training_plugin.model_run() File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 141, in model_run raise e File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 133, in model_run self.run_train_only() File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 202, in run_train_only self.train_model(self.local_dataset_directory) File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 106, in train_model self.model.train(directory) File "/microservice/train.py", line 25, in train train.train(self.opt, self.dataset) File "", line 172, in train Exception: multi_task Extraction Model Training Failed 2021-03-23 13:59:59,260 - uipath_core.trainer_run:main:66 - INFO: Starting training job... 2021-03-23 14:00:06,649 - uipath_core.logs.upload_log_service:upload_logs_file:56 - INFO: Retry Training Triggered: 2021-03-23 14:00:07,330 - uipath_core.storage.azure_storage_client:download:96 - INFO: Dataset from bucket folder training-7b66d190-b3d4-4b4f-bae9-d12ada9364ba/34524b99-e06d-4ff3-8b53-9574701a7676/bb7f427f-2112-4e2e-a33d-03457383dc1d with size 28 downloaded successfully 2021-03-23 14:00:07,330 - uipath_core.training_plugin:train_model:104 - INFO: Start model training... 2021-03-23 14:00:07,330 - uipath_core.training_plugin:initialize_model:98 - INFO: Start model initialization... 2021-03-23 14:00:07,331 - root:configure_options:109 - INFO: Document type invoices_india language: en 2021-03-23 14:00:07,332 - uipath_core.training_plugin:initialize_model:101 - INFO: Model initialized successfully 2021-03-23 14:00:07,333 - root:configure_options:109 - INFO: Document type invoices_india language: en 2021-03-23 14:00:07,333 - root:preprocess_data:457 - INFO: Creating dataset for document type invoices_india.. 2021-03-23 14:00:07,351 - root:preprocess_data:460 - INFO: Doctype invoices_india Statistics: 2021-03-23 14:00:07,409 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice4.pdf_0.jpg.json 2021-03-23 14:00:07,620 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:00:07,620 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:00:07,620 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:00:07,620 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:00:07,620 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:00:07,620 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:00:08,183 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:00:08,183 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:00:08,183 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:00:08,183 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:00:08,183 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:00:08,184 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:00:08,184 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:00:08,323 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:00:08,324 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:00:08,324 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:00:08,324 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:00:08,324 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:00:08,324 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:00:10,467 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_OD220514724105351000 (1).pdf_0.jpg.json 2021-03-23 14:00:10,599 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:00:10,599 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:00:10,599 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:00:10,599 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:00:10,599 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:00:10,599 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:00:10,599 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:00:10,738 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:00:10,739 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:00:10,739 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:00:10,739 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:00:10,739 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:00:10,739 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:00:10,739 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:00:10,739 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:00:10,739 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:00:10,842 - root:preprocess_data:523 - INFO: train: (11, 12) 2021-03-23 14:00:10,843 - root:preprocess_data:524 - INFO: train: (2, 12) 2021-03-23 14:00:10,843 - root:preprocess_data:529 - INFO: ratio: 0.153846 2021-03-23 14:00:10,845 - root:load_glove_vocabulary:90 - INFO: Loading offsets for en 2021-03-23 14:00:13,025 - root:build_vocabulary:129 - INFO: Building vocabulary... 2021-03-23 14:00:13,027 - root:build_embeddings:218 - INFO: Building embeddings... 2021-03-23 14:00:13,085 - root:save:71 - INFO: Saving invoices_india doctype object at /microservice/dataset/doctype.p... 2021-03-23 14:00:13,118 - root:save:241 - INFO: Saving dataset object at /microservice/dataset/dataset.p... 2021-03-23 14:00:13,122 - root:__init__:147 - INFO: Dataset creation done. Contains doctypes: ['invoices_india']. 2021-03-23 14:00:13,122 - root:train:147 - INFO: Training extraction model... 2021-03-23 14:00:13,122 - root:__init__:61 - INFO: Finetuning from base model: multi_task... 2021-03-23 14:00:13,122 - root:load:230 - INFO: Loading model multi_task... 2021-03-23 14:00:13,126 - root:create_network:127 - INFO: Building network ... 2021-03-23 14:00:13,126 - root:__init__:28 - INFO: Creating new network ... 2021-03-23 14:00:13,423 - root:load_network:265 - INFO: Loading network ... 2021-03-23 14:00:13,498 - root:create_network:127 - INFO: Building network ... 2021-03-23 14:00:13,498 - root:__init__:28 - INFO: Creating new network ... 2021-03-23 14:00:13,759 - root:create_network:130 - INFO: Cloning heads from base model multi_task... 2021-03-23 14:00:13,805 - root:__init__:91 - INFO: Model creation done. Contains tasks ['invoices_india', 'all'] 2021-03-23 14:00:13,807 - root:train:381 - INFO: Creating torch datasets ... 2021-03-23 14:00:13,973 - root:train_first_stage:400 - INFO: First stage training started... 2021-03-23 14:00:13,976 - root:train_first_stage:404 - INFO: Prep first stage done 2021-03-23 14:00:13,976 - root:_train:451 - INFO: Training all tasks for 100 epochs from epoch 0... 2021-03-23 14:00:13,976 - root:create_optimizer:183 - INFO: Building first_stage optimizer for task all... 2021-03-23 14:00:13,980 - root:create_scheduler:202 - INFO: Building first_stage scheduler for task all... 2021-03-23 14:00:13,980 - root:_train:460 - INFO: Starting Training Loop ... 2021-03-23 14:00:32,308 - root:save:212 - INFO: Saving model... 2021-03-23 14:00:32,405 - root:save:216 - INFO: Model saved 2021-03-23 14:00:32,405 - root:_log_to_console:1299 - INFO: [Task: all][Epoch: 001][tag_f1.invoices_india: 0.3616/0.9332][tag_f1.all: 0.3616/0.9332][lr.all: 0.001000] 2021-03-23 14:00:32,405 - root:_log_to_console:1303 - INFO: [Found best model][tag_f1.all: 0.9332][tag_f1.invoices_india: 0.9332] 2021-03-23 14:00:33,092 - root:train:170 - ERROR: multi_task Extraction Model Training Failed Traceback (most recent call last): File "", line 154, in train File "", line 38, in train_tag File "", line 385, in train File "", line 404, in train_first_stage File "", line 470, in _train File "", line 1237, in _track_aux_metrics File "", line 797, in calculate_extraction_scores File "", line 1426, in calculate_extraction_F1 File "/opt/conda/lib/python3.7/site-packages/pandas/core/frame.py", line 7548, in apply return op.get_result() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 180, in get_result return self.apply_standard() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 271, in apply_standard results, res_index = self.apply_series_generator() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 300, in apply_series_generator results[i] = self.f(v) File "", line 1425, in File "", line 769, in calculate_row_F1 File "", line 736, in prepare_row_for_scoring AttributeError: 'list' object has no attribute 'get_value' 2021-03-23 14:00:33,094 - uipath_core.training_plugin:model_run:140 - ERROR: Training failed for pipeline type: TRAIN_ONLY, error: multi_task Extraction Model Training Failed 2021-03-23 14:00:33,098 - uipath_core.trainer_run:main:81 - ERROR: Training Job failed, error: multi_task Extraction Model Training Failed Traceback (most recent call last): File "", line 154, in train File "", line 38, in train_tag File "", line 385, in train File "", line 404, in train_first_stage File "", line 470, in _train File "", line 1237, in _track_aux_metrics File "", line 797, in calculate_extraction_scores File "", line 1426, in calculate_extraction_F1 File "/opt/conda/lib/python3.7/site-packages/pandas/core/frame.py", line 7548, in apply return op.get_result() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 180, in get_result return self.apply_standard() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 271, in apply_standard results, res_index = self.apply_series_generator() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 300, in apply_series_generator results[i] = self.f(v) File "", line 1425, in File "", line 769, in calculate_row_F1 File "", line 736, in prepare_row_for_scoring AttributeError: 'list' object has no attribute 'get_value' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/trainer_run.py", line 76, in main wrapper.run() File "/microservice/training_wrapper.py", line 57, in run return self.training_plugin.model_run() File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 141, in model_run raise e File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 133, in model_run self.run_train_only() File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 202, in run_train_only self.train_model(self.local_dataset_directory) File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 106, in train_model self.model.train(directory) File "/microservice/train.py", line 25, in train train.train(self.opt, self.dataset) File "", line 172, in train Exception: multi_task Extraction Model Training Failed 2021-03-23 14:01:59,444 - uipath_core.trainer_run:main:66 - INFO: Starting training job... 2021-03-23 14:02:06,797 - uipath_core.logs.upload_log_service:upload_logs_file:56 - INFO: Retry Training Triggered: 2021-03-23 14:02:07,427 - uipath_core.storage.azure_storage_client:download:96 - INFO: Dataset from bucket folder training-7b66d190-b3d4-4b4f-bae9-d12ada9364ba/34524b99-e06d-4ff3-8b53-9574701a7676/bb7f427f-2112-4e2e-a33d-03457383dc1d with size 28 downloaded successfully 2021-03-23 14:02:07,427 - uipath_core.training_plugin:train_model:104 - INFO: Start model training... 2021-03-23 14:02:07,427 - uipath_core.training_plugin:initialize_model:98 - INFO: Start model initialization... 2021-03-23 14:02:07,428 - root:configure_options:109 - INFO: Document type invoices_india language: en 2021-03-23 14:02:07,429 - uipath_core.training_plugin:initialize_model:101 - INFO: Model initialized successfully 2021-03-23 14:02:07,430 - root:configure_options:109 - INFO: Document type invoices_india language: en 2021-03-23 14:02:07,430 - root:preprocess_data:457 - INFO: Creating dataset for document type invoices_india.. 2021-03-23 14:02:07,448 - root:preprocess_data:460 - INFO: Doctype invoices_india Statistics: 2021-03-23 14:02:07,506 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice4.pdf_0.jpg.json 2021-03-23 14:02:07,700 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:02:07,701 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:02:07,701 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:02:07,701 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:02:07,701 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:02:07,701 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:02:08,211 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:02:08,211 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:02:08,211 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:02:08,211 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:02:08,211 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:02:08,211 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:02:08,211 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:02:08,347 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:02:08,347 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:02:08,347 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:02:08,347 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:02:08,347 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:02:08,347 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:02:10,264 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_OD220514724105351000 (1).pdf_0.jpg.json 2021-03-23 14:02:10,389 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:02:10,389 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:02:10,389 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:02:10,389 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:02:10,389 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:02:10,389 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:02:10,389 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:02:10,517 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:02:10,517 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:02:10,517 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:02:10,517 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:02:10,517 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:02:10,518 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:02:10,518 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:02:10,518 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:02:10,518 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:02:10,608 - root:preprocess_data:523 - INFO: train: (11, 12) 2021-03-23 14:02:10,608 - root:preprocess_data:524 - INFO: train: (2, 12) 2021-03-23 14:02:10,608 - root:preprocess_data:529 - INFO: ratio: 0.153846 2021-03-23 14:02:10,611 - root:load_glove_vocabulary:90 - INFO: Loading offsets for en 2021-03-23 14:02:12,569 - root:build_vocabulary:129 - INFO: Building vocabulary... 2021-03-23 14:02:12,572 - root:build_embeddings:218 - INFO: Building embeddings... 2021-03-23 14:02:12,620 - root:save:71 - INFO: Saving invoices_india doctype object at /microservice/dataset/doctype.p... 2021-03-23 14:02:12,649 - root:save:241 - INFO: Saving dataset object at /microservice/dataset/dataset.p... 2021-03-23 14:02:12,652 - root:__init__:147 - INFO: Dataset creation done. Contains doctypes: ['invoices_india']. 2021-03-23 14:02:12,652 - root:train:147 - INFO: Training extraction model... 2021-03-23 14:02:12,652 - root:__init__:61 - INFO: Finetuning from base model: multi_task... 2021-03-23 14:02:12,652 - root:load:230 - INFO: Loading model multi_task... 2021-03-23 14:02:12,655 - root:create_network:127 - INFO: Building network ... 2021-03-23 14:02:12,656 - root:__init__:28 - INFO: Creating new network ... 2021-03-23 14:02:12,911 - root:load_network:265 - INFO: Loading network ... 2021-03-23 14:02:12,983 - root:create_network:127 - INFO: Building network ... 2021-03-23 14:02:12,983 - root:__init__:28 - INFO: Creating new network ... 2021-03-23 14:02:13,230 - root:create_network:130 - INFO: Cloning heads from base model multi_task... 2021-03-23 14:02:13,272 - root:__init__:91 - INFO: Model creation done. Contains tasks ['invoices_india', 'all'] 2021-03-23 14:02:13,273 - root:train:381 - INFO: Creating torch datasets ... 2021-03-23 14:02:13,425 - root:train_first_stage:400 - INFO: First stage training started... 2021-03-23 14:02:13,427 - root:train_first_stage:404 - INFO: Prep first stage done 2021-03-23 14:02:13,428 - root:_train:451 - INFO: Training all tasks for 100 epochs from epoch 0... 2021-03-23 14:02:13,428 - root:create_optimizer:183 - INFO: Building first_stage optimizer for task all... 2021-03-23 14:02:13,431 - root:create_scheduler:202 - INFO: Building first_stage scheduler for task all... 2021-03-23 14:02:13,431 - root:_train:460 - INFO: Starting Training Loop ... 2021-03-23 14:02:32,167 - root:save:212 - INFO: Saving model... 2021-03-23 14:02:32,488 - root:save:216 - INFO: Model saved 2021-03-23 14:02:32,488 - root:_log_to_console:1299 - INFO: [Task: all][Epoch: 001][tag_f1.invoices_india: 0.3616/0.9332][tag_f1.all: 0.3616/0.9332][lr.all: 0.001000] 2021-03-23 14:02:32,488 - root:_log_to_console:1303 - INFO: [Found best model][tag_f1.all: 0.9332][tag_f1.invoices_india: 0.9332] 2021-03-23 14:02:33,117 - root:train:170 - ERROR: multi_task Extraction Model Training Failed Traceback (most recent call last): File "", line 154, in train File "", line 38, in train_tag File "", line 385, in train File "", line 404, in train_first_stage File "", line 470, in _train File "", line 1237, in _track_aux_metrics File "", line 797, in calculate_extraction_scores File "", line 1426, in calculate_extraction_F1 File "/opt/conda/lib/python3.7/site-packages/pandas/core/frame.py", line 7548, in apply return op.get_result() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 180, in get_result return self.apply_standard() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 271, in apply_standard results, res_index = self.apply_series_generator() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 300, in apply_series_generator results[i] = self.f(v) File "", line 1425, in File "", line 769, in calculate_row_F1 File "", line 736, in prepare_row_for_scoring AttributeError: 'list' object has no attribute 'get_value' 2021-03-23 14:02:33,120 - uipath_core.training_plugin:model_run:140 - ERROR: Training failed for pipeline type: TRAIN_ONLY, error: multi_task Extraction Model Training Failed 2021-03-23 14:02:33,123 - uipath_core.trainer_run:main:81 - ERROR: Training Job failed, error: multi_task Extraction Model Training Failed Traceback (most recent call last): File "", line 154, in train File "", line 38, in train_tag File "", line 385, in train File "", line 404, in train_first_stage File "", line 470, in _train File "", line 1237, in _track_aux_metrics File "", line 797, in calculate_extraction_scores File "", line 1426, in calculate_extraction_F1 File "/opt/conda/lib/python3.7/site-packages/pandas/core/frame.py", line 7548, in apply return op.get_result() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 180, in get_result return self.apply_standard() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 271, in apply_standard results, res_index = self.apply_series_generator() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 300, in apply_series_generator results[i] = self.f(v) File "", line 1425, in File "", line 769, in calculate_row_F1 File "", line 736, in prepare_row_for_scoring AttributeError: 'list' object has no attribute 'get_value' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/trainer_run.py", line 76, in main wrapper.run() File "/microservice/training_wrapper.py", line 57, in run return self.training_plugin.model_run() File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 141, in model_run raise e File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 133, in model_run self.run_train_only() File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 202, in run_train_only self.train_model(self.local_dataset_directory) File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 106, in train_model self.model.train(directory) File "/microservice/train.py", line 25, in train train.train(self.opt, self.dataset) File "", line 172, in train Exception: multi_task Extraction Model Training Failed 2021-03-23 14:03:38,301 - uipath_core.trainer_run:main:66 - INFO: Starting training job... 2021-03-23 14:03:45,614 - uipath_core.logs.upload_log_service:upload_logs_file:56 - INFO: Retry Training Triggered: 2021-03-23 14:03:46,240 - uipath_core.storage.azure_storage_client:download:96 - INFO: Dataset from bucket folder training-7b66d190-b3d4-4b4f-bae9-d12ada9364ba/34524b99-e06d-4ff3-8b53-9574701a7676/bb7f427f-2112-4e2e-a33d-03457383dc1d with size 28 downloaded successfully 2021-03-23 14:03:46,240 - uipath_core.training_plugin:train_model:104 - INFO: Start model training... 2021-03-23 14:03:46,240 - uipath_core.training_plugin:initialize_model:98 - INFO: Start model initialization... 2021-03-23 14:03:46,241 - root:configure_options:109 - INFO: Document type invoices_india language: en 2021-03-23 14:03:46,242 - uipath_core.training_plugin:initialize_model:101 - INFO: Model initialized successfully 2021-03-23 14:03:46,243 - root:configure_options:109 - INFO: Document type invoices_india language: en 2021-03-23 14:03:46,243 - root:preprocess_data:457 - INFO: Creating dataset for document type invoices_india.. 2021-03-23 14:03:46,262 - root:preprocess_data:460 - INFO: Doctype invoices_india Statistics: 2021-03-23 14:03:46,322 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice4.pdf_0.jpg.json 2021-03-23 14:03:46,549 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:03:46,549 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:03:46,549 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:03:46,549 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:03:46,549 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:03:46,549 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice1 (1).pdf_0.jpg.json 2021-03-23 14:03:47,092 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:03:47,092 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:03:47,092 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:03:47,092 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:03:47,092 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:03:47,092 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:03:47,092 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice13 (1).pdf_0.jpg.json 2021-03-23 14:03:47,221 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:03:47,221 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:03:47,221 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:03:47,222 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:03:47,222 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:03:47,222 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice2 (1).pdf_0.jpg.json 2021-03-23 14:03:49,204 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_OD220514724105351000 (1).pdf_0.jpg.json 2021-03-23 14:03:49,324 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:03:49,324 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:03:49,324 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:03:49,324 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:03:49,324 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:03:49,324 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:03:49,324 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice3 (1).pdf_0.jpg.json 2021-03-23 14:03:49,455 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:03:49,455 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:03:49,455 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:03:49,455 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:03:49,455 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:03:49,456 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:03:49,456 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:03:49,456 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:03:49,456 - root:create_datapoint:255 - INFO: Box with zero/negative height on line 0, file: invoicenewlist_invoice15 (2).pdf_0.jpg.json 2021-03-23 14:03:49,541 - root:preprocess_data:523 - INFO: train: (11, 12) 2021-03-23 14:03:49,541 - root:preprocess_data:524 - INFO: train: (2, 12) 2021-03-23 14:03:49,541 - root:preprocess_data:529 - INFO: ratio: 0.153846 2021-03-23 14:03:49,544 - root:load_glove_vocabulary:90 - INFO: Loading offsets for en 2021-03-23 14:03:51,471 - root:build_vocabulary:129 - INFO: Building vocabulary... 2021-03-23 14:03:51,473 - root:build_embeddings:218 - INFO: Building embeddings... 2021-03-23 14:03:51,524 - root:save:71 - INFO: Saving invoices_india doctype object at /microservice/dataset/doctype.p... 2021-03-23 14:03:51,555 - root:save:241 - INFO: Saving dataset object at /microservice/dataset/dataset.p... 2021-03-23 14:03:51,559 - root:__init__:147 - INFO: Dataset creation done. Contains doctypes: ['invoices_india']. 2021-03-23 14:03:51,559 - root:train:147 - INFO: Training extraction model... 2021-03-23 14:03:51,559 - root:__init__:61 - INFO: Finetuning from base model: multi_task... 2021-03-23 14:03:51,559 - root:load:230 - INFO: Loading model multi_task... 2021-03-23 14:03:51,563 - root:create_network:127 - INFO: Building network ... 2021-03-23 14:03:51,563 - root:__init__:28 - INFO: Creating new network ... 2021-03-23 14:03:51,830 - root:load_network:265 - INFO: Loading network ... 2021-03-23 14:03:51,903 - root:create_network:127 - INFO: Building network ... 2021-03-23 14:03:51,903 - root:__init__:28 - INFO: Creating new network ... 2021-03-23 14:03:52,164 - root:create_network:130 - INFO: Cloning heads from base model multi_task... 2021-03-23 14:03:52,207 - root:__init__:91 - INFO: Model creation done. Contains tasks ['invoices_india', 'all'] 2021-03-23 14:03:52,209 - root:train:381 - INFO: Creating torch datasets ... 2021-03-23 14:03:52,364 - root:train_first_stage:400 - INFO: First stage training started... 2021-03-23 14:03:52,367 - root:train_first_stage:404 - INFO: Prep first stage done 2021-03-23 14:03:52,367 - root:_train:451 - INFO: Training all tasks for 100 epochs from epoch 0... 2021-03-23 14:03:52,367 - root:create_optimizer:183 - INFO: Building first_stage optimizer for task all... 2021-03-23 14:03:52,370 - root:create_scheduler:202 - INFO: Building first_stage scheduler for task all... 2021-03-23 14:03:52,371 - root:_train:460 - INFO: Starting Training Loop ... 2021-03-23 14:04:11,317 - root:save:212 - INFO: Saving model... 2021-03-23 14:04:11,635 - root:save:216 - INFO: Model saved 2021-03-23 14:04:11,635 - root:_log_to_console:1299 - INFO: [Task: all][Epoch: 001][tag_f1.invoices_india: 0.3616/0.9332][tag_f1.all: 0.3616/0.9332][lr.all: 0.001000] 2021-03-23 14:04:11,636 - root:_log_to_console:1303 - INFO: [Found best model][tag_f1.all: 0.9332][tag_f1.invoices_india: 0.9332] 2021-03-23 14:04:12,306 - root:train:170 - ERROR: multi_task Extraction Model Training Failed Traceback (most recent call last): File "", line 154, in train File "", line 38, in train_tag File "", line 385, in train File "", line 404, in train_first_stage File "", line 470, in _train File "", line 1237, in _track_aux_metrics File "", line 797, in calculate_extraction_scores File "", line 1426, in calculate_extraction_F1 File "/opt/conda/lib/python3.7/site-packages/pandas/core/frame.py", line 7548, in apply return op.get_result() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 180, in get_result return self.apply_standard() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 271, in apply_standard results, res_index = self.apply_series_generator() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 300, in apply_series_generator results[i] = self.f(v) File "", line 1425, in File "", line 769, in calculate_row_F1 File "", line 736, in prepare_row_for_scoring AttributeError: 'list' object has no attribute 'get_value' 2021-03-23 14:04:12,309 - uipath_core.training_plugin:model_run:140 - ERROR: Training failed for pipeline type: TRAIN_ONLY, error: multi_task Extraction Model Training Failed 2021-03-23 14:04:12,312 - uipath_core.trainer_run:main:81 - ERROR: Training Job failed, error: multi_task Extraction Model Training Failed Traceback (most recent call last): File "", line 154, in train File "", line 38, in train_tag File "", line 385, in train File "", line 404, in train_first_stage File "", line 470, in _train File "", line 1237, in _track_aux_metrics File "", line 797, in calculate_extraction_scores File "", line 1426, in calculate_extraction_F1 File "/opt/conda/lib/python3.7/site-packages/pandas/core/frame.py", line 7548, in apply return op.get_result() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 180, in get_result return self.apply_standard() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 271, in apply_standard results, res_index = self.apply_series_generator() File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 300, in apply_series_generator results[i] = self.f(v) File "", line 1425, in File "", line 769, in calculate_row_F1 File "", line 736, in prepare_row_for_scoring AttributeError: 'list' object has no attribute 'get_value' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/trainer_run.py", line 76, in main wrapper.run() File "/microservice/training_wrapper.py", line 57, in run return self.training_plugin.model_run() File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 141, in model_run raise e File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 133, in model_run self.run_train_only() File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 202, in run_train_only self.train_model(self.local_dataset_directory) File "/home/aifabric/.local/lib/python3.7/site-packages/uipath_core/training_plugin.py", line 106, in train_model self.model.train(directory) File "/microservice/train.py", line 25, in train train.train(self.opt, self.dataset) File "", line 172, in train Exception: multi_task Extraction Model Training Failed