Download 500k files from SFTP server

Help Studio
, ,
1

Hi,

I am working on creating an automation where I have to download 500k files from Server A and upload it to Server B. I have tried using the FTP activity to download file in For each loop on enumerate objects. However, this is too slow and it would take forever for the files to be downloaded. What avenue should I explore to fasten this process - Multithreading (I have never used this before so might need guidance on how to approach), Parallel for each or something else?

2

I would create a dispatcher automation that creates a queue item for each file, then a performer that works from the queue to do the download/upload…that way you can run multiple jobs of the performer.

1 Like
3

@shrishti.diggikar

  1. Create a dispatcher which will loop through each files and uploads file name to the queue with unique reference. So in case if robot fails because of timeout you can restart the bot and it will not add the previously added queue items. Use retry scope and try catch to avoid timeout errors. In performer you can download the file and upload it. You can use multiple performer bots to do this.

0r

  1. Use the above solution with parallel for each loop
1 Like
4

Hi @shrishti.diggikar

Since you are dealing with large number of files its better to go for invoke code and it works much faster

Sample invoke code to download the files
Import these namespaces
image

Try
    outErrorMessage = Nothing

    Using sftp As New Renci.SshNet.SftpClient(ftpServer, ftpPort, ftpUsername, ftpPassword)
        sftp.Connect()

        If Not sftp.IsConnected Then
            Throw New Exception("Failed to connect to SFTP server.")
        End If

      

        ' List files in the remote directory
        Dim files = sftp.ListDirectory(ftpFolder).Where(Function(f) Not f.IsDirectory)

        For Each file In files
            Dim localFilePath As String = System.IO.Path.Combine(localPath, file.Name)
            Using fs As New System.IO.FileStream(localFilePath, System.IO.FileMode.Create)
                sftp.DownloadFile(file.FullName, fs)
            End Using
        Next

        sftp.Disconnect()
    End Using

Catch ex As Exception
    outErrorMessage = "Download failed: " & ex.Message
End Try

For more refer this link

Try this and hope this helps!

Happy Automation :innocent:

5

Hi @shrishti.diggikar

Try to use parallel for each with batching. break the file list into smaller batches and process multiple files simultaneously by setting degreeofparallelism (eg. 5 to 10). this allows concurrent ftp transfers without overwhelming the servers.

If helpful, mark as solution. Happy automation with UiPath