SQL Server Bulk Data Insert Using UiPath

Hi Team,

Any idea about below process.

Explain the Process :

Both Excel and DB columns are same, but the data type is different since DB data type is constant and excel data type is a object , need to modify based on DB data in spreadsheet.

From the output file need to take the unmatched column name comparing with SQL server & Excel file to be segregated.

Finally matched column values to be inserted into the SQL server

Incase of any error occurs while importing, need to get notified

Thanks
Shyam

When working with UiPath to handle data between Excel and SQL Server, especially when dealing with mismatched data types, there’s a structured approach you can follow to ensure smooth data transfer and error handling.

1. Read Excel File

  • Use the “Read Range” activity from the Excel activities package to read the data from the Excel file into a DataTable. This will treat all data as type Object, which is the default behavior for Excel data in .NET.

2. Analyze and Modify DataTypes Based on SQL Server Schema

  • Connect to your SQL Server database and fetch the schema of the target table using a “Execute Query” activity with a schema-fetching SQL command like SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = ‘YourTableName’.
  • Iterate over the columns in your DataTable and compare their names and data types with those in the database schema. For each column that matches by name but has a different data type, you will need to modify the data in your DataTable to match the SQL Server data type.
    • For example, if a column in Excel contains numbers as text (e.g., “1234”) and the corresponding column in SQL Server is an integer, you’ll need to convert those text values to integers.

3. Segregate Unmatched Columns

  • Create a list or a separate DataTable to store unmatched columns. As you iterate through the Excel columns and compare them with the database schema, add the names of unmatched columns to this list/DataTable.

4. Insert Matched Column Values into SQL Server

  • Use the “Insert” SQL command within an “Execute Non-Query” activity for inserting data. You may need to build a dynamic SQL query string or use a stored procedure if the column names and numbers are dynamic.
  • Ensure that you only include columns that matched in both name and data type in your INSERT commands.
  • To handle dynamic columns, you might consider generating the column names and values part of your INSERT query dynamically based on the matched columns.

5. Error Handling and Notifications

  • Wrap your activities in a “Try Catch” block to handle exceptions. In the “Catch” section, you can specify actions to take in case of errors, such as logging the error or sending an email notification.
  • To send notifications, you can use the “Send SMTP Mail Message” activity or any other notification activity available in UiPath, configured with your email server settings.

Additional Tips:

  • Data Type Conversion : For data type conversion, use .NET methods like Convert.ToInt32(), Convert.ToDateTime(), etc., depending on the target data type.
  • Logging and Monitoring : Utilize UiPath’s logging capabilities to log successful operations, the number of records inserted, and details of any records or columns that failed to match or convert.
  • Transaction Management : Consider using database transactions if your workflow involves multiple insert/update operations. This way, you can roll back changes in case of an error in any part of the process.

Hi @Muhammad_Nadeem_Abbas

Thanks :slight_smile:

Do you have any code for this ?

In Point 2 sub Point 2 have confusion conversion issue.

Thanks
Shyam

I can guide you through the conceptual structure and pseudo-code to outline how your process could be implemented:

Step 1: Read Excel File

Excel Application Scope (FilePath: "YourExcelFilePath.xlsx")
    Read Range (Range: "", Output: dtExcelData)

Step 2: Analyze and Modify DataTypes Based on SQL Server Schema

Connect to Database (Connection String: "YourConnectionString")
    Execute Query (SQL: "SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 'YourTableName'", Output: dtSchema)

For Each Row in dtSchema (Row: schemaRow)
    For Each Column in dtExcelData.Columns (Column: excelColumn)
        If schemaRow["COLUMN_NAME"].ToString.Equals(excelColumn.ColumnName)
            // Check and convert data type as necessary, e.g., from string to int
            For Each dataRow in dtExcelData.Rows
                // Example conversion: string to int
                If schemaRow["DATA_TYPE"].ToString.Equals("int")
                    dataRow(excelColumn) = Convert.ToInt32(dataRow(excelColumn))

Step 3: Segregate Unmatched Columns

List<String> unmatchedColumns = new List<String>()

For Each excelColumn in dtExcelData.Columns
    If Not dtSchema.AsEnumerable().Any(Function(schemaRow) schemaRow["COLUMN_NAME"].ToString.Equals(excelColumn.ColumnName))
        unmatchedColumns.Add(excelColumn.ColumnName)

// Optionally, create a DataTable to store unmatched columns

Step 4: Insert Matched Column Values into SQL Server

// Building a dynamic SQL INSERT command based on matched columns
String sqlInsertCommand = "INSERT INTO YourTableName (MatchedColumnNames) VALUES (@Value1, @Value2, ...)"

Connect to Database (Connection String: "YourConnectionString")
    For Each dataRow in dtExcelData.Rows
        Execute Non-Query (SQL: sqlInsertCommand, Parameters: {MatchedColumnValues})

Step 5: Error Handling and Notifications

Try
    // Wrap the above steps here
Catch exception as Exception
    Log Message (Message: exception.Message, Level: "Error")
    Send SMTP Mail Message (Subject: "Workflow Error", Body: exception.Message, ...)
1 Like

Hi @Muhammad_Nadeem_Abbas

Thanks :slight_smile:

I will try and update you…

Thanks
Shyam

That would be great :slight_smile:

Doesnt know the columns name of schemaRow (Eveything Dynamic)

image

How to check this place ?

Main.xaml (11.1 KB)

Thanks
Shyam

If just the columns of schemaRow are dynamic, meaning you are dealing with a scenario where the SQL Server table schema (column names and data types) might change over time, but you need to adapt your UiPath workflow to handle these changes gracefully, here’s a refined approach focusing on the dynamic nature of the SQL schema:

Step 1: Read Excel File

First, read the data from the Excel file into a DataTable. This will capture the Excel data that you want to insert into the SQL Server table.

Excel Application Scope
    Read Range (Output: dtExcel)

Step 2: Fetch SQL Server Schema Dynamically

Connect to your SQL Server database and dynamically fetch the schema of the target table. This gives you the current structure of the table without knowing the column names in advance.

Execute Query (SQL: "SELECT TOP 0 * FROM YourTableName", Output: dtSchema)

Here, dtSchema will have the same structure as your target table but without any rows. It allows you to understand the table structure dynamically.

Step 3: Match Excel Columns to SQL Table Columns

Iterate through the columns in dtExcel and match them to the columns in dtSchema by name. This assumes that the column names in Excel match those in the SQL Server table. If they don’t match exactly, you might need a mapping logic or manual mapping table.

For Each excelColumn In dtExcel.Columns
    If dtSchema.Columns.Contains(excelColumn.ColumnName) Then
        // Column exists in SQL table, proceed with data type conversion if necessary
    Else
        // Column does not exist in SQL table, handle accordingly (e.g., log or skip)
    End If

Step 4: Convert Data Types If Necessary

Before inserting data into the SQL Server table, ensure the data types in the DataTable match those expected by SQL Server. This step might require dynamic type conversion based on the dtSchema column data types.

// Pseudocode - you’ll need to implement conversion logic based on your needs
For Each row As DataRow In dtExcel.Rows
    For Each col As DataColumn In dtExcel.Columns
        If dtSchema.Columns.Contains(col.ColumnName) Then
            // Convert row(col.ColumnName) to the type required by dtSchema.Columns(col.ColumnName).DataType
        End If
    Next
Next

Step 5: Dynamically Build and Execute SQL Insert Command

Construct and execute an SQL INSERT command for each row in your DataTable. The command should dynamically include only the columns present in both dtExcel and the SQL Server table.

// Build INSERT SQL command dynamically based on matched columns
String sqlInsertCommand = "INSERT INTO YourTableName (MatchedColumnNames) VALUES (@Value1, @Value2, ...)"

// Use Execute Non-Query activity to insert each row

Remember to parameterize your SQL queries to prevent SQL injection.

Step 6: Implement Error Handling

Use Try-Catch blocks around critical sections, especially database interactions and data conversion, to manage exceptions. Optionally, add logic to notify you in case of errors, for instance, through email notifications using the “Send SMTP Mail Message” activity.

Note

  • Implementing dynamic data type conversion based on the SQL schema requires custom logic. You might need to map .NET data types to SQL data types and handle conversions accordingly.
  • Ensure that your approach to building dynamic SQL commands is secure and guards against SQL injection.
  • This approach assumes that there is some consistency or mapping logic between the Excel data and the SQL table schema that can be leveraged for dynamic matching and conversion.

While doing the Step. 3 I am Stuck and Confused.

Can you help me…