Ai fabric deployment error

Hi there,

I am facing an issue here, I have a zip file containing a ML model, and I can upload it whithout any problems, but when I try to create an MLSkill a series os warnings show up before an error mensage and the deployment fails, can anyone help?

The Error mensage: Failed to create deployment AIFabric_ML_Demo_OM

The warnings mensage:
File “/microservice/main.py”, line 19, in
testeiro = Main()
File “/microservice/main.py”, line 10, in init
self.model = joblib.load(’./model/ClassProbl.joblib’)
File “/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py”, line 585, in load
obj = _unpickle(fobj, filename, mmap_mode)
File “/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py”, line 504, in _unpickle
obj = unpickler.load()
File “/usr/local/lib/python3.8/pickle.py”, line 1210, in load
dispatchkey[0]
File “/usr/local/lib/python3.8/pickle.py”, line 1587, in load_reduce
stack[-1] = func(*args)
File “sklearn/tree/_tree.pyx”, line 607, in sklearn.tree._tree.Tree.cinit
ValueError: Buffer dtype mismatch, expected ‘SIZE_t’ but got ‘int’

File “/microservice/main.py”, line 19, in
testeiro = Main()
File “/microservice/main.py”, line 10, in init
self.model = joblib.load(’./model/ClassProbl.joblib’)
File “/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py”, line 585, in load
obj = _unpickle(fobj, filename, mmap_mode)
File “/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py”, line 504, in _unpickle
obj = unpickler.load()
File “/usr/local/lib/python3.8/pickle.py”, line 1210, in load
dispatchkey[0]
File “/usr/local/lib/python3.8/pickle.py”, line 1587, in load_reduce
stack[-1] = func(*args)
File “sklearn/tree/_tree.pyx”, line 607, in sklearn.tree._tree.Tree.cinit
ValueError: Buffer dtype mismatch, expected ‘SIZE_t’ but got ‘int’

Hi Bruno

Looking quickly on the internet this may be caused by two things:

  • different types of OS 32 bits/64 bits
  • different librairies version
    Let’s check first librairies version can you check what version you used locally to train the model and if you specify exact same version in requirements.txt file?

Jeremy

Hello,
the libraries used on the joblibe file:

!pip install --user pandas==0.24.2
!pip install --user pandas_ml==0.6.1
#downgrade matplotlib to bypass issue with confusion matrix being chopped out
!pip install --user matplotlib==3.1.0
!pip install --user scikit-learn==0.21.3
!pip install -q --user scikit-plot
!pip install --user scipy
!pip install pydotplus

on the requirements.txt:

scipy>=0.17.0
scikit-learn>=0.21.3
joblib>=0.14.1
pandas>=0.24.2
matplotlib>=3.1.0

Hello again,
I checked the libraries and updated the requirements.txt file, but now I’m getting a diferent error mensage, can you help me?

The new error mensage:
Error while creating docker image for image gcr.io/ai-fabrics/host/e2703282_2221_4ea0_b3eb_236e1bed9fde/4d2e4bfe_76b2_4cce_842f_90cd0812312f/aifabric_ml_demo_om:4 with details ERROR: Command errored out with exit status 1: ERROR: Failed building wheel for scikit-learn ERROR: Command errored out with exit status 1:ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c ‘import sys, setuptools, tokenize; sys.argv[0] = ‘"’"’/tmp/pip-install-k83b7uy8/scikit-learn/setup.py’"’"’; file=’"’"’/tmp/pip-install-k83b7uy8/scikit-learn/setup.py’"’"’;f=getattr(tokenize, ‘"’"‘open’"’"’, open)(file);code=f.read().replace(’"’"’\r\n’"’"’, ‘"’"’\n’"’"’);f.close();exec(compile(code, file, ‘"’"‘exec’"’"’))’ install --record /tmp/pip-record-q65_h3ai/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/scikit-learn Check the logs for full command output.ERROR: An error occurred: non-zero (13) exit code from uipath/uipath-core-s2i-python38:0.1-AIF

And thank you for your atention

Hey @Bruno
I’m not sure I understand this part:

the libraries used on the joblibe file:
!pip install --user pandas==0.24.2
!pip install --user pandas_ml==0.6.1
#downgrade matplotlib to bypass issue with confusion matrix being chopped out
!pip install --user matplotlib==3.1.0
!pip install --user scikit-learn==0.21.3
!pip install -q --user scikit-plot
!pip install --user scipy
!pip install pydotplus

Is that what you ran locally or is this in one of the file you uploaded?

You should use == in you requirements.txt to be sure that you keep same version as the one you used locally. The following should work:

scipy==0.17.0
scikit-learn==0.21.3
joblib==0.14.1
pandas==0.24.2
matplotlib==3.1.0

Thank you,
That solved my problem… But there is somethink else wrong with my project because a diferent error mensage showed up… this time is this one:
Error while creating docker image for image gcr.io/ai-fabrics/host/e2703282_2221_4ea0_b3eb_236e1bed9fde/4d2e4bfe_76b2_4cce_842f_90cd0812312f/aifabric_ml_demo_om:4 with details ERROR: Command errored out with exit status 1: ERROR: Failed building wheel for scikit-learn ERROR: Command errored out with exit status 1:ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c ‘import sys, setuptools, tokenize; sys.argv[0] = ‘"’"’/tmp/pip-install-k83b7uy8/scikit-learn/setup.py’"’"’; file =’"’"’/tmp/pip-install-k83b7uy8/scikit-learn/setup.py’"’"’;f=getattr(tokenize, ‘"’"‘open’"’"’, open)( file );code=f.read().replace(’"’"’\r\n’"’"’, ‘"’"’\n’"’"’);f.close();exec(compile(code, file , ‘"’"‘exec’"’"’))’ install --record /tmp/pip-record-q65_h3ai/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/scikit-learn Check the logs for full command output.ERROR: An error occurred: non-zero (13) exit code from uipath/uipath-core-s2i-python38:0.1-AIF

Thank you again!

Would you mind sharing your code here? (remove the model itself)

import joblib
import json
import numpy as np
import pandas as pd

class Main:

def __init__(self):
    self.model = joblib.load('./model/ClassProbl.joblib')
    
def predict(self, x):
    example = json.loads(x)
    df = pd.DataFrame([example])
    result = self.model.predict(df)
    return json.dumps(result.tolist())

testeiro = Main()

Ok looks like an issue with librairies and versions. Are you sure that you are using python38 locally?
If I have to bet I would say that you are on 3.5 or something.
If you check https://scikit-learn.org/dev/install.html#:~:text=Scikit-learn%200.22%20supported%20Python,requires%20Python%203.6%20or%20newer. sklearn 21.3 is not supported on python 3.8.
And scipy 0.17 is not supported above Python 3.5: https://docs.scipy.org/doc/scipy/reference/release.0.17.0.html

So we don’t have right python language for this.
One thing yo can try is changing Scipy version to 0.19.1 and use Python36 when you upload the ML package.
If that doesn’t work I would say that the easiest would be to retrain your models with more recent version of python librairies.

Hello again,

So… I re trained the model using more recent version of the libraries as you said, and the error message came back to the frst one, with the warnings
(File “/microservice/main.py”, line 9, in init
self.model = joblib.load(’./model/ClassificaProbl.joblib’)
File “/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py”, line 605, in load
obj = _unpickle(fobj, filename, mmap_mode)
File “/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py”, line 529, in _unpickle
obj = unpickler.load()
File “/usr/local/lib/python3.8/pickle.py”, line 1210, in load
dispatchkey[0]
File “/usr/local/lib/python3.8/pickle.py”, line 1587, in load_reduce
stack[-1] = func(*args)
File “sklearn/tree/_tree.pyx”, line 607, in sklearn.tree._tree.Tree.cinit
ValueError: Buffer dtype mismatch, expected ‘SIZE_t’ but got ‘int’

File “/microservice/main.py”, line 9, in init
self.model = joblib.load(’./model/ClassificaProbl.joblib’)
File “/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py”, line 605, in load
obj = _unpickle(fobj, filename, mmap_mode)
File “/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py”, line 529, in _unpickle
obj = unpickler.load()
File “/usr/local/lib/python3.8/pickle.py”, line 1210, in load
dispatchkey[0]
File “/usr/local/lib/python3.8/pickle.py”, line 1587, in load_reduce
stack[-1] = func(*args)
File “sklearn/tree/_tree.pyx”, line 607, in sklearn.tree._tree.Tree.cinit
ValueError: Buffer dtype mismatch, expected ‘SIZE_t’ but got ‘int’

File “/microservice/main.py”, line 9, in init
self.model = joblib.load(’./model/ClassificaProbl.joblib’)
File “/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py”, line 605, in load
obj = _unpickle(fobj, filename, mmap_mode)
File “/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py”, line 529, in _unpickle
obj = unpickler.load()
File “/usr/local/lib/python3.8/pickle.py”, line 1210, in load
dispatchkey[0]
File “/usr/local/lib/python3.8/pickle.py”, line 1587, in load_reduce
stack[-1] = func(*args)
File “sklearn/tree/_tree.pyx”, line 607, in sklearn.tree._tree.Tree.cinit
ValueError: Buffer dtype mismatch, expected ‘SIZE_t’ but got ‘int’

File “/microservice/main.py”, line 9, in init
self.model = joblib.load(’./model/ClassificaProbl.joblib’)
File “/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py”, line 605, in load
obj = _unpickle(fobj, filename, mmap_mode)
File “/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py”, line 529, in _unpickle
obj = unpickler.load()
File “/usr/local/lib/python3.8/pickle.py”, line 1210, in load
dispatchkey[0]
File “/usr/local/lib/python3.8/pickle.py”, line 1587, in load_reduce
stack[-1] = func(*args)
File “sklearn/tree/_tree.pyx”, line 607, in sklearn.tree._tree.Tree.cinit
ValueError: Buffer dtype mismatch, expected ‘SIZE_t’ but got ‘int’)

I also updated the requirements.txt
scikit-plot==0.3.7
scikit-learn==0.23.2
joblib==0.14.1
pandas==1.1.3
pydotplus==2.0.2
pandas_ml==0.6.1
scipy==1.5.3

What should I do?

Any new information about this? I’m kind of stuck and already late to delivery this…

Thank you in advanced.

Hi Bruno
Can you confirm that libraries and python versions that you use are all same on your local environments and on AI Fabric?

The AI Fabric ones that you mean are the ones that I specified on the requerements.txt? If so yes.

Hello again,

this is still bugging me, this is a project that I converted from other one, this other worked perfectly and I culd deploy it on aifabric. Due to confidentiality issues I can’t use this one, so I converted changing the names, the model is the exactly same, but the one I converted is now showing this error:
Non retryable error occurred while deploying mlskill: AIFabric_Demo_ML_OM, reason: ModuleNotFoundError:
file “/microservice/main.py”, line 21, in
testeiro = main()
file “/microservice/main.py”, line 10, in init
self.model = joblib.load(’./model/classificaproblema.joblib’)
file “/home/aifabric/.local/lib/python3.7/site-packages/joblib/numpy_pickle.py”, line 605, in load
obj = _unpickle(fobj, filename, mmap_mode)
file “/home/aifabric/.local/lib/python3.7/site-packages/joblib/numpy_pickle.py”, line 529, in _unpickle
obj = unpickler.load()
file “/usr/local/lib/python3.7/pickle.py”, line 1088, in load
dispatchkey[0]
file “/usr/local/lib/python3.7/pickle.py”, line 1385, in load_stack_global
self.append(self.find_class(module, name))
file “/usr/local/lib/python3.7/pickle.py”, line 1426, in find_class
import(module, level=0)
modulenotfounderror: no module named ‘sklearn.preprocessing._data’

file “/microservice/main.py”, line 21, in
testeiro = main()
file “/microservice/main.py”, line 10, in init
self.model = joblib.load(’./model/classificaproblema.joblib’)
file “/home/aifabric/.local/lib/python3.7/site-packages/joblib/numpy_pickle.py”, line 605, in load
obj = _unpickle(fobj, filename, mmap_mode)
file “/home/aifabric/.local/lib/python3.7/site-packages/joblib/numpy_pickle.py”, line 529, in _unpickle
obj = unpickler.load()
file “/usr/local/lib/python3.7/pickle.py”, line 1088, in load
dispatchkey[0]
file “/usr/local/lib/python3.7/pickle.py”, line 1385, in load_stack_global
self.append(self.find_class(module, name))
file “/usr/local/lib/python3.7/pickle.py”, line 1426, in find_class
import(module, level=0)
modulenotfounderror: no module named ‘sklearn.preprocessing._data’

And the other deploys perfectly. I’m creating the MLPackege using Python 3.7 on both.
Can you help me?

Thanks in advanced.

Not sure I actually understand what you did any way you can share the ML Packages here on using private message do I can have a look?

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.