I would keep a small track file that includes the files that I downloaded. Most probably an excel would work.
So at the beginning of the process, we check whether the tracking excel do not have any data. If not, that means we are good and we can continue downloading. After each download, I would enter a record to the excel (let’s say the file name for example) so that it will maintain a list of all the files i downloaded. So that if a system error occurs, we can read the excel file and identify which files we downloaded before and contirnue from there.
Check excel : empty
download 1st file
download 2nd file
– System crashes
Restart of the program
Check excel file : not empty
- Read the data in the excel to a datatable and identify which files were downloaded
- Then, if the next downloading file is not available in the excel, download, else, skip to the next
You can have this in a memory as a array or something, However, I wanted to make sure it also supports in case of a complete failure and the process also needs to start again. In that case, having it stored in somewhere like a excel file helps alot.