Dealing with large Excel files, is it better to filter in Excel or from a DT?

[jonathan.traeger]

Hello all!

I have a project where the dispatcher will need to retrieve an .xlsx from an automated email. The email has an average of 150,000 rows of data, spread over 8 columns.

There is some filtering that needs to be done prior to uploading these items to Orchestrator. My question, is it more effective to:

1. upload the entire file as a data table to Studio, then filter
2. do some/all the filtering on excel file prior to uploading as a DT

Thanks in advance!

[Anil_G (Anil Gorthi)]

@jonathan.traeger

Use Workbook activities read the data and use filtering in Studio

If you used excel activities you will end up with high time(If the data is 8 lakh plus then filtering on excel and using might make sense or use excel as database and use query where filtering is also done using where clause)

Hope this helps

cheers

[Srini84 (Srinivas)]

@jonathan.traeger

It’s better you read the entire file and do the filters
Also keep an eye on the reading time and filtering time

If you feel that reading time / Filtering time is high than expected

Then you can try Excel as a Database and there you can write queries as SQL which is very fast

Thanks,
Srini