Load CSV files into SQL

[timothy.mullady]

Hello,

I’m trying to loop through a folder of large CSV files and load the data into a SQL database. The tricky part is each CSV file has around 250K rows of data.

Can someone please explain the Activity workflow? I understand the basic workflow is Connect to SQL => Read CSV => Insert to SQL.

I believe there is a limit to the number of Rows in the insert activity. I also need to add a loop for each file.

Any information would be greatly appreciated.

Thanks,

Tim

[Dilli_Reddy]

@timothy.mullady

“Get Files in Folder” activity to retrieve a list of CSV files from the specified folder.

Use a “For Each” activity to iterate through each CSV file in the folder.

Inside the loop, use the “Read CSV” activity to read the data from the current CSV file into a DataTable.

Use the “Connect” activity to establish a connection to your SQL database. Provide the necessary connection string and credentials.

Insert Data into SQL:

Use the “Execute Non-Query” activity to insert the data from the DataTable into the SQL database. You can use SQL commands to perform bulk inserts.

BULK INSERT YourTableName
FROM ‘C:\Path\To\Your\File.csv’
WITH (FIELDTERMINATOR = ‘,’, ROWTERMINATOR = ‘\n’, FIRSTROW = 2)

Cheers…!