Getting Error While Formatting the string Variable with Huge data

Help Studio
, ,
1

I have a text file data with the size of around 500mb data(around 1045553 rows of data ). Iam reading that text file, storing it in a string variable and while formatting that data like using assign activity and do ReadTextFileString=ReadTextFileString.Replace(“”“”,“”). Iam getting a error “Job stopped with an unexpected exit code: 0xE0434352” . If i try the same with less number of data like 60000 rows of data, it takes time but it is working. If i try to format huge data it is making the error. Can anyone please help me to resolve this issue. Thanks in Advance!

2

Hi

In your case, the most likely cause of the error is that you are trying to process too much data in memory. The UiPath platform has a limit on the amount of data that can be processed in memory at one time. If you try to process more data than the limit, the platform will throw an error.

To resolve this issue, you can try the following:

  1. Split the data into smaller chunks. Where You can use the ReadTextFile activity to read the data in chunks. For example, you could read the first 1000 rows of data, process it, and then read the next 1000 rows of data.

  2. Use a database to store the data

  3. Try to use KILL PROCESS activity where pass your application executable Filename as ProcessName property

Refer these threads for similar issue and resolution

Hope this helps

Cheers @Dinesh_dine

3

The Text File Data I have, will be generated as a single file either 500 MB file or 1000 MB file(around 1045000 lines of data or 3000000 lines of data). I tried with 50000 lines of data alone in UiPath I can able to process the above formatting. So if I need to process this data I need to read the data by 50000 lines store it in a string variable and do a loop till I reach the end. But I’m not sure how to read specific lines of data from that text file( for example if I need to read from line 50001 to line 100000) or split the file in runtime that may result in loss of data. Can you please help on that issue

1 Like
4

@Palaniyappan Can you please help me on the above issue. Thanks in Advance

5

Fine

To process large text files in UiPath efficiently without loading the entire file into memory, you can use a combination of techniques to read specific lines or split the file in runtime without losing data

A general outline of the steps:

  • Use the “Read Text File” activity to read the file line by line.
  • Maintain a line counter variable.
  • Start reading from the beginning of the file.
  • Use a loop to skip lines until you reach the starting line you want to process.
  • Read and process the lines within your desired range.
  • Stop reading when you reach the ending line you want to process or when you reach the end of the file.

Here’s a simplified example in pseudocode:

lineCounter = 0 desiredStartLine = 50001 desiredEndLine = 100000 While Not EndOfFile line = ReadNextLineFromFile lineCounter = lineCounter + 1 If lineCounter >= desiredStartLine And lineCounter <= desiredEndLine ProcessLine(line) If lineCounter > desiredEndLine ExitLoop End While

2. Splitting the File in Runtime: If you prefer to split the file into smaller chunks without losing data, you can create new files with a limited number of lines each. Here’s how you can do it:

  • Use the “Read Text File” activity to read the original file line by line.
  • Maintain a line counter variable.
  • Create a new output file.
  • Start reading from the beginning of the original file.
  • Use a loop to write lines to the output file until you reach a certain line limit.
  • Close the current output file and create a new one if the line limit is reached.
  • Continue until you reach the end of the original file.

This way, you’ll have multiple smaller files with subsets of the original data. You can then process these smaller files individually.

Hope this helps

Cheers @Dinesh_dine

6

You could try to read it line by line like this:
image