I’m trying to loop through a folder of large CSV files and load the data into a SQL database. The tricky part is each CSV file has around 250K rows of data.
Can someone please explain the Activity workflow? I understand the basic workflow is Connect to SQL => Read CSV => Insert to SQL.
I believe there is a limit to the number of Rows in the insert activity. I also need to add a loop for each file.
Any information would be greatly appreciated.
“Get Files in Folder” activity to retrieve a list of CSV files from the specified folder.
Use a “For Each” activity to iterate through each CSV file in the folder.
Inside the loop, use the “Read CSV” activity to read the data from the current CSV file into a DataTable.
Use the “Connect” activity to establish a connection to your SQL database. Provide the necessary connection string and credentials.
Insert Data into SQL:
Use the “Execute Non-Query” activity to insert the data from the DataTable into the SQL database. You can use SQL commands to perform bulk inserts.
BULK INSERT YourTableName
WITH (FIELDTERMINATOR = ‘,’, ROWTERMINATOR = ‘\n’, FIRSTROW = 2)
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.