How to fetch more than 5000 row data from dataverse

Help Studio
, , , ,
1

I am using get api, there are more than 10k records , i am getting only 5k records . Can any one help in increasing limit to maximum !

1 Like
2

UiPath’s “Get API” activity typically retrieves data in pages by default, with a limitation on the number of records returned per page. Here’s how to address this and potentially increase the limit to retrieve more or all records when dealing with over 10,000 entries:

1. Check API Documentation:

  • The most reliable approach is to refer to the API documentation provided by the service you’re calling with the “Get API” activity. It should specify:
    • Whether there’s a way to increase the page size or result limit for the API call.
    • How to implement this increase, such as using specific parameters in the request URL or headers.

2. Pagination Parameters (if available):

  • If the API supports pagination, look for parameters like $top, $skip, or similar keywords in the documentation. These parameters allow you to specify the number of records per page ($top) and the starting record for the desired page ($skip).
  • You can construct the URL dynamically in UiPath using string manipulation functions to adjust these parameters based on your needs.

3. Consider Alternative Approaches (if pagination unavailable):

  • If the API doesn’t offer pagination, explore alternative approaches depending on the service:
    • Batching: If the API allows filtering by specific criteria, you can potentially retrieve data in smaller batches using filters to segment the results.
    • Alternative Endpoints: The API might offer alternative endpoints designed for retrieving large datasets.

**4. Schedulers **

  • Ask dev team is any scheduler we can build where we hit the API and it allow to download this file or put this file SFTP folder.

5. Throttling and Rate Limits:

  • Be aware that some APIs might impose throttling or rate limits on requests. Increasing the limit significantly could lead to throttling errors. Check the API documentation for any such limitations.

By following these steps and consulting the API documentation, you should be able to determine the best way to increase the result limit or retrieve all records exceeding 10,000 entries using UiPath’s “Get API” activity. If the API doesn’t offer clear options, consider contacting the service provider for further guidance.

3

Hi @MitheshBolla

I assume you are using a loop to save the entries and that you have an Office 365 Power Apps license. If that is the case then your license limits you to loops no bigger than 5,000. Using pagination you can get all the records, but can only process 5,000 in a single Apply to each loop. You need a medium or high performance profile to process larger loops. That means a paid license.

Check below as well:-

Thanks

4

please share a screenshot of that Activity with us. Thanks for support

5

You need to use the get option from Httpclient Activities

image
image1232Ă—699 66 KB

6

It’s because of the size of the Jsion object or API output, In Server level need to increase the MAX Request Size, which solve the issue and the solution is that you need to use the pagination for getting the data.

7

with the Activity
grafik
grafik

And the mention of

the mention of

looks like a LLM confusion

8

image
image1420Ă—788 162 KB

you need to select the method, Note: before this you just check your API is working in the Postman

9

thanks for the feedback. no explanation is needed as the usage of HTTP Request is clear. Kindly note how we handle LLM Answers here in the forum: