File uploaded looks like junk when opened from Storage bucket

[AndyMenon]

I’m trying to upload a file to Storage Bucket via APIs. API call succeeds, and I can see the file in Storage Bucket (1). But opening the file shows it’s junk and appears as a small white square (2) !

The post linked below seems to have solved the problem with Curl + PHP but, I am not sure what the issue is with my Python script. Here for example is the affected code snippet that is posting junk to the storage bucket:

Doing an api request to add a file to bucket with CURL

Hello! Making upload form of a file from web service to orchestrator bucket. (using PHP + CURL) For first i got a dynamicly generated link to upload with GET request: odata/Buckets(1)/UiPath.Server.Configuration.OData.GetWriteUri?path=\&expiryInMinutes=5 So - to upload file to a bucket i need to make a PUT request to link, that i got with previous one. file is in $_FILES, and formated to object with CURLFile function and attached to POSTFIELDS of curl request. The problem → nothing happeni…

cc: @codemonkee

[Shikhar_Tandon]

Hi @AndyMenon,

@loginerror may be can help here.

Thanks

[AndyMenon]

Thanks @Shikhar_Tandon.
Hope it gets his attention.

Even otherwise, I don’t understand why this must be so hard. If something was wrong with the composition of the API, then it must fail!

I’m wondering if I’ve missed any other method of packaging the file up in binary format.

[loginerror (Maciej Kuźmicz)]

Interesting issue!

As a start, I would suggest checking if it works well with a request from Postman.

If yes, I would suggest sending the exact same request to something like https://requestbin.com/ or https://webhook.site to compare the two to see if there are any discrepancies.

[AndyMenon]

Yep. Works with postman perfectly.

Will test the others.

Thank you!

[Vibhor.Shrivastava]

Hi @Ovidiu_Constantin @Corina_Marginas Could you please help or loop in someone who can guide here

[codemonkee (Tim)]

I gave myself a good chuckle as I intended to reply back a few days ago but kept getting busy and then I got hung up the PUT API call as I decided to try External Applications for the first time and wasn’t paying attention when I kept passing the authentication in both my Python and Postman PUT request when I shouldn’t have been … something silly for far too long… I blame it on me not having written Python…

ANYWAYS
Don’t mind the spaghetti code and lack of parametrization and formatting.

The following Python code works for me, tested on a small image and downloading that image returns expected output.

import requests
from oauthlib.oauth2 import BackendApplicationClient
from requests_oauthlib import OAuth2Session

token_url = 'https://cloud.uipath.com/identity_/connect/token'
clientId = 'MAGIC_CLIENT_ID'
clientSecret = 'SUPERDUPERSECRET'
clientScope = 'SOMEOVERTHETOPAPPLICATIONSCOPE'

client = BackendApplicationClient(client_id=clientId)
uipath = OAuth2Session(client=client)

uipath.fetch_token(
    token_url=token_url,
    client_id=clientId,
    client_secret=clientSecret,
    response_type='client_credentials',
    scope=clientScope
)
bucketUrl = 'https://cloud.uipath.com/org/tenant/odata/Buckets(132456)/UiPath.Server.Configuration.OData.GetWriteUri'
headers = {
    'X-UIPATH-OrganizationUnitId': '5454645654645'
}
params = {
    'path': '\\path\\tim-tuque.jpg',
    'contentType': 'image/jpeg'
}
r = uipath.request('GET', bucketUrl, headers=headers, params=params)
jsonResponse = r.json()

data = open('.\\tim-tuque.jpg', 'rb').read()
r = requests.put(jsonResponse['Uri'], data=data, headers={'x-ms-blob-type': 'BlockBlob'})
if r.status_code == 201:
    print('SUCCESS')

Possible the difference with how files and data is treated and looking at the documentation of Requests.Put it doesn’t appear that it takes files as an argument [But I find the formatting of those docs to be awkward and not sure what the kwargs / Key Word Arguments intails].

r = requests.put(BlobUploadUri, files={"src": file_data},headers=headers)

vs

r = requests.put(BlobUploadUri, data=data, headers=headers)

• files – (optional) Dictionary of 'name': file-like-objects (or {'name': file-tuple} ) for multipart encoding upload. file-tuple can be a 2-tuple ('filename', fileobj) , 3-tuple ('filename', fileobj, 'content_type') or a 4-tuple ('filename', fileobj, 'content_type', custom_headers) , where 'content-type' is a string defining the content type of the given file and custom_headers a dict-like object containing additional headers to add for the file.

• data – (optional) Dictionary, list of tuples, bytes, or file-like object to send in the body of the Request .

requests. put ( url , data=None , **kwargs )[source]

Sends a PUT request.

Parameters: * url – URL for the new Request object.

• data – (optional) Dictionary, list of tuples, bytes, or file-like object to send in the body of the Request .
• json – (optional) json data to send in the body of the Request .
• **kwargs – Optional arguments that request takes.
  Returns: Response object
  Return type:

[AndyMenon]

Hello @codemonkee ,

@codemonkee

I gave myself a good chuckle

Indeed! This one deserves it! No worries. I have been paying attention to this in a Round Robin fashion. Thanks for taking the time to elaborate on your response. Truly appreciate it!

My code was based on the request-response cycle of Postman that works without issues. In Postman the request body appears as:
src: filename.ext

You are correct. This is not right, but I modified that later after my initial efforts didn’t work.

@AndyMenon

I’m wondering if I’ve missed any other method of packaging the file up in binary format.

If you notice in the code snippet I posted, I commented out the line at (1) before fixing it and moving it into a separate function call. Seems like I was under code hypnosis and erred on (2).

I will apply the fix and report on the status.

Thanks again!
Andy

[codemonkee (Tim)]

@AndyMenon

My code was based on the request-response cycle of Postman that works without issues. In Postman the request body appears as:
src: filename.ext

Out of curiosity, what approach did you take in postman? I haven’t been able to get it to work using form-data and a File upload… I assume the Keys I’ve tried to be incorrect which I’ve tried files, src, data, and in each of those cases The File is created with a HTTP200, but 0KB in Size.

It’s only when I switch to binary in Postman that I can get a valid file upload

Similar in my python testing it only worked for me when I past the parameter data in the PUT request instead of files. I suspect that part is a me problem, but I’m not 100% sure of others have been successful with form-data PUT.

[AndyMenon]

Yep! Binary works. Form-data does not. I tried that option before switching to binary.

If I recall correctly, the response code is 201 Created. But, I will send you a snapshot of the request in a few.

@codemonkee

It’s only when I switch to binary in Postman that I can get a valid file upload

Also, recalling from the swagger docs, the format of the body to upload the blob file is specified as binary. I recall seeing that and making the switch to binary.

Thanks

[codemonkee (Tim)]

@AndyMenon

If I recall correctly, the response code is 201 Created.

I was just testing against a private 2020.10 instance, so probably just a version difference. I do recall a HTTP201 when I was testing against Cloud last night.

Another difference I’ve noticed is the headers being returned in the Cloud instance from the WriteUri to be included with the PUT

@codemonkee

headers={'x-ms-blob-type': 'BlockBlob'}

The header array in the GetWriteUri is empty in my 2020.10 instance.

{
    "@odata.context": "https://fqdn/odata/$metadata#UiPath.Server.Configuration.OData.BlobFileAccessDto",
    "Uri": "https://fqdn/api/BlobFileAccess/Put?t=23452345-9A51-41AA-2345-D023452345324C54&r=sdfasdfasdf&s=bIwOre3qAeIi-asdfasdFAE$T$#@TFVDSUwuimJ06w0.",
    "Verb": "PUT",
    "Headers": {
        "Keys": [],
        "Values": []
    }
}

So an interesting difference to call out as I haven’t found any documentation on the PUT request yet.

[AndyMenon]

@AndyMenon

I will send you a snapshot of the request in a few.

The Headers came from the preceding request below. I tested this from the Cloud instance. I have to test this with our Resident orchestrator provided our Admin allows me to without a lengthy explanation first.

The uploaded file as seen in the Orchestrator - this request is about 3 minutes old.

And this is where my troubles began with Python. This is the request body of Postman:

@codemonkee

Another difference I’ve noticed is the headers being returned in the Cloud instance from the WriteUri to be included with the PUT

Yes. That is correct. I got that from studying the original Azure blob storage documentation. The URL I referred to is below. And to correct my earlier statement, the switch to binary was a result of going through this page and not swagger.

Put Blob (REST API) - Azure Storage

@codemonkee

The header array in the GetWriteUri is empty in my 2020.10 instance. So an interesting difference to call out as I haven’t found any documentation on the PUT request yet.

The API version in the Cloud is v13.0. Could this be a factor? But yes, I had to work with these APIs by moving between Swagger and the MSFT documentation page above. Adding the GetWriteUri request here so that those who need it may find it helpful.

@codemonkee

Another difference I’ve noticed is the headers being returned in the Cloud instance from the WriteUri to be included with the PUT

Agree. When I saw the response from Postman below, it was kind of suggestive of this fact. At that time, I wanted to doubly confirm this and therefore, I went to the MSFT documentation linked above. The upload wouldn’t work without these headers.

[codemonkee (Tim)]

Awesome! Well let me know if you need a second pair of eyes on anything related to this… Happy to test against both Cloud 2021 and Private 2010.10.

Happy PUTing.

[AndyMenon]

@AndyMenon

I will apply the fix and report on the status.

Thank you @codemonkee! It worked! I just ran the Python client and I can see the file as it should be . To make sure of that, I uploaded a completely different file of type image/png. And here are the results:

[AndyMenon]

@codemonkee

Happy PUTing.

And talking about PUTing, I just changed the name of that Python function from
post_UploadBlobToBucket

to

put_UploadBlobToBucket

Agreed! I will reach out. Thanks for stepping in and helping me with this!

[AndyMenon]

FYI @codemonkee ,

Just finished up on a counterpart C# API client. Again that stupid line {src: filename} was the root cause of all maladies.

Works like a charm now.

thanks