Test manager override parameter ruins dataset

Help Test Cloud
1

With the recent deprecation of Orchestrator Testing, our team has been forced to migrate our entire testing suite to Test Manager. Orchestrator Testing used to work perfectly for our setup, but we have hit a massive roadblock in Test Manager that feels like a regression bug and is currently halting our test automation.

We maintain a large suite of about 500 test cases, which are reused across multiple Test Sets. All of these test cases are data-driven, with test data attached via an Excel sheet in Studio. Since many of these sheets contain multiple rows, the test cases are supposed to run multiple iterations.

Alongside the Excel data, every test case has two standalone ‘In’ arguments (inCompany and inSecurityRoles) that are not mapped to the Excel sheet. We use these arguments to test the cases with different roles and in different environments by altering them at the Test Set level.

In the old Orchestrator Testing, we could easily alter these parameters, and the test would still iterate through all the Excel rows perfectly. But in Test Manager, this logic is broken.

When I have a Test Set in Test Manager and use the “Override parameters” feature to change inSecurityRoles and inCompany, Test Manager completely ignores the attached Excel datasheet. The test case will execute exactly once, skipping all the other rows.

If I don’t alter the parameters, it correctly runs all iterations from the Excel sheet, but obviously with the wrong default arguments for that specific environment.

Because we have around 500 test cases, creating duplicates in Studio or updating 500 Excel sheets manually to hardcode these roles is absolutely not a viable or scalable workaround for us.

Has anyone else encountered this issue? Is there any known, scalable solution to pass Test Set-specific parameters to a data-driven test without Test Manager killing the dataset iterations?

Thanks in advance!

2

The safer workarnd is to avoid Test Set override for values like environment/role and instead retrieve them at runtime from a centralized configuration source such as Orchestrator Assets, JSON config, or another external store. This allows the Excel sheet to remain the sole data driver, so all iterations are preserved

3

So how would this work?
Testset A with testcase 1 will be getting the same asset as Testset B with testcase 1?

4

Not necessarily. The asset doesn’t have to be the same for every Test Set. One approach is to store the values in Orchestrator assets per folder or environment, so Test Set A and Test Set B can read different values even if the asset name is the same. Another option is to use different asset names per Test Set (e.g., TestSetA_Company, TestSetB_Company). This way the Excel sheet remains the iteration driver, while the environment/role values can still vary per Test Set without breaking the dataset execution.

5

Thanks for the suggestions! While those approaches work well in standard scenarios, they unfortunately don’t solve the issue given our specific constraints and the scale of 500 test cases:

  1. Regarding Folder/Environment-specific assets:
    All of our Test Sets are executed within the same Orchestrator folder. We do not have separate folders for Test Set A and Test Set B. Because they run in the exact same folder, we cannot use the same asset name to yield different values.

  2. Regarding different asset names (e.g., TestSetA_Company):
    If we create differently named assets for each Test Set, the underlying .xaml test case needs a way to know which asset to fetch at runtime. Since a UiPath robot does not natively know the name of the Test Set that triggered it, the code inside the reusable test case cannot dynamically decide whether to use a Get Asset activity for “TestSetA_Company” or “TestSetB_Company”.

Because the test case is completely “blind” to which Test Set is currently executing it, the Override Parameters feature in Test Manager is literally the only native way to pass that Test Set-specific context down to the script.

Since the Override feature currently breaks the dataset iterations, we are stuck. We need a way to pass this context without killing the Excel rows, and without having to hardcode 500 individual Excel files.

6

you r right that folder-scoped assets or asset naming conventions won’t help much. Can you try to use wrapper test case ? Instead of Test Manager running the data-driven test directly, it runs a wrapper test that receives inCompany and inSecurityRoles from the Test Set. The wrapper then reads the Excel dataset, loops through each row, and calls the reusable test logic while passing both the row data and the Test Set parameters.

7

That is actually a very clever workaround to bypass the Test Manager bug, and you are completely right that moving the dataset logic inside a wrapper would allow the Overrides to work again.

However, implementing that approach comes with two major drawbacks that make it unviable for our enterprise setup:

1. Loss of Granular Reporting: The main benefit of Test Manager’s native data-driven testing is that it creates separate test execution logs for each iteration. If we move the iteration into a For Each Row loop inside a wrapper workflow, Test Manager will only register it as a single execution. If row 4 fails, the entire test just fails, and we lose the step-level visibility we rely on.

2. The 500 Test Cases Refactoring Nightmare: To implement this, we would essentially have to refactor all 500 of our existing test cases to include Read Range activities, build 500 individual wrappers, or build a highly complex dynamic invoker. That is months of development work just to bypass a regression bug introduced by the deprecation of Orchestrator Testing.

While your wrapper logic is sound and definitely a great technical workaround for a small project, it unfortunately doesn’t scale for an existing suite of our size.

We really need the native “Override Parameters” feature to work alongside attached Test Data, exactly as it did before the migration. Hopefully, someone from the UiPath product team can acknowledge this regression and provide a timeline for a fix.

8

You’re absolutely right that moving the iteration into a wrapper would break the granular reporting per dataset row. And with a suite of ~500 reusable test cases, refactoring everything just to work around this behavior would clearly not be practical. I have tried to provide a possible workaround for this issue. If it helps in any way, please feel free to mark the solution as resolved.

9

Thanks for the help.
I wont be marking this as solved before Test Manager works as intended