I’m new to UiPath, and I’ve been playing with Test Manager. When assigning Test Cases to a Test Set, I see that there is a sequential number appended to the Project Name (for example: Project1:1, Project1:2, Project1:3…). However, there doesn’t appear to be any way (at least I haven’t found it) of sorting the test cases by these, and the order of execution seems to be random. I can’t figure out how to control the order of test case execution within a test set. Given there’s a sequential key, I would think there must be a way of controlling the order of execution, but maybe not?
Just an update on this for anyone having a similar question/issue. After meeting with UiPath, this is not currently possible, but is planned to be added in the near future.
This is an endless discussion. There are even more suggestions from UiPath, such as “Make a test case that runs all other test cases in the desired order” or the statement “UiPath believes that all test cases should run independently of each other.” …
I think UiPath’s solutions are really great, there are a lot of things that make them different from other solutions that are great, but in any other testing tool you can order test cases in a test set, that’s a basic feature.
Your proposed solution is of no use to us at all, because we would have to create a test set for each individual test case, which is not only cumbersome, but also generates quite a bit of extra work and I don’t see any point in it. Among other things, we test the following:
Customer opening
Account opening
Depot opening
Make a Payment
Buy shares
etc
This is neither an END-TO-END test case, as claimed by UiPath, nor can these test cases be started independently of each other, because without a customer there is no account and without an account there is no payment etc.
UiPath always points out that you can run your tests on multiple machines at the same time. I think it’s great, but please give me the opportunity to decide for myself. A machine and in what order.
Hey Michael - totally agree with your points. There’s obviously more tedious ways around this, but not practical in many cases. IMO this was a major oversight when they designed the Test Manager. As I mentioned previously, I have spoken to UiPath about this and they told me that it is on their future development list and the person was hopeful that it would be added soon. I was hoping it would be in this latest release, but it wasn’t :-(.
I also agree this should be basic functionality, needed for a chain test (eg, Vreate customer, create order, create contract, etc.)
All mentioned work-arounds are not gooed enough for reporting the results to the business
For anyone following this post or has the concern in the future, I just wanted to let you know that I have heard back from UiPath on this and, unfortunately, it will not make it into a release this year
Below is their response, and I have asked that they respond to this post, so people can respond directly. Ordering of tests in a test set is basic functionality for any QA software (especially automation), and the idea that you should rename a test or that all tests should not be dependent on another, is unrealistic, especially when working with larger applications (like ERP systems).
With that said this is their response…
Let me get the ‘bad’ news out of the way – as far as I can tell from our internal channels, the effort to provide a more “advanced ordering system” of some kind seems to have been pushed to a potential 2023 release. I’m not sure if this is due to it just not ‘making the cut’ for the upcoming 2022.10 release, if there are additional efforts running alongside it that have elongated time to release, or if the product team does not currently see it as a priority based on current provided guidance, but that’s about as much as I can find at this point.
The still-current guidance I keep running up on is two-fold:
- Current ‘limitations’ for ordering are intentional. I’ve read several [internal and external] back-and-forths that include various scenarios; however, and for multiple reasons, we still don’t really recommend the ordering of test cases due to their very nature – they’re intended to be created as stateless as possible and should be run independently. “Runtime (or orchestrator) is designed to support distributed execution. If TC 2 has a dependency on TC 1, then this means they can't run on two separate robots at the same time.” That said…
• Test cases can always be ordered within a studio workflow. You can invoke test cases within another test case. But this dependency is defined at design-time
• Aside from that, see also this entry on thecodebuzz.com – Ordering on test cases can be achieved by any of the following mechanisms:
• Alphabetical – Order test method based on test name’s Alphabetical order
• DisplayName – Order test method based on test name’s DisplayName order
• Priority– Order test method based on test name’s using priority order set by user providing better control on execution.
Would still appreciate an official response from someone at UiPath on this (other than what I posted above) and why you feel this is ‘not’ basic functionality and why it’s not being prioritized for an official release.
Several points made are invalid, don’t work, or are just misguided, so it would be nice to have an official word on this and if we can ever expect this functionality to be added in the future.
Hey Michael - funny, I didn’t know either, but found it - if you scroll all the way to the top of this post, there’s a ‘Vote’ button next to the title on the Left-hand side.
All feedback post in the forum will have a vote button in the post header. This helps product managers of respective products gauge the need for a fix or feature requests. More the number of votes, better chances that it may be prioritised.
You did not see it before because when you posted this thread, you posted it to the discussions section which does not have such a vote button. I have now re categorised this post to feedback.
I too think this is a fundamental ask for any testing team. Fingers crossed!
I would love to see the specification of test execution order added soon in Orchestrator. Within QA we often encounter the need where certain test cases are dependent upon the successful execution of others; such dependencies make execution order necessary.
I see that the original post i from Feb 2022. I hope that this capability is included in an update very soon. It’s definitely needed.
I have just seen this post and would have liked to contribute earlier.
This is not the way tests work, they don’t run in sequence and I personally don’t think they should.
It would also be a real pain from an infrastructure perspective.
Each test case runs as its own Job, as a result you could have one test case in a test set run on one machine / account and another test case running on another, you could have a production job start on in between test case jobs as they have higher priority if you use the same robot for testing and production.
In my opinion each test case should run completely self contained, and thats how I design mine, this avoids any of the issues mentioned above.
There are many different types of tests - unit tests, regression tests, black-box tests, End-to-End tests, Performance tests, etc. Although, this statement is true for some tests (such as unit tests), it does not work for most Regression/End-to-End testing. I’ve described this before, but for an ERP system (for example), you usually have to…
Create a Customer
Create a Quote/Requisition
Create an Order
Ship the Order
Create an Invoice
Send Invoice to Customer
Each can be a separate test, to test individual functionality, but if testing E2E, then you have to use the Customer created in Step1, to use on the Quote for step2, and that Quote # is used to create the Order, and so on. In otherwards, the data from 1 test has to be used in subsequent tests. This is where UiPath comes up short because there is no way to handle that in Test Manager currently without doing funky naming (e.g. putting an ‘A’ or ‘1’ for the first step etc.) OR make a huge process to do all those steps (which means maintaining more processes, jobs, etc.) - not efficient.
UiPath is the ONLY automation software that I’ve seen (and I’ve used a lot) that doesn’t allow ordering of test cases - it’s a huge oversight imo, but this has been going on for 2 years now and it hasn’t been made a priority, so this is probably a mute discussion. I just wanted to address the argument that it’s not the way tests work, because not all tests are created equal
When I say thats not how it works, I am referring to the tests in UiPath based on the infrastucture, as I mentioned, each test case runs as a job and as such can be allocated dynamically. So just like you wouldnt chain together jobs in a sequence, I am suggesting don’t do that in tests.
To do a regression test that validates if components work in sequence, then I say make a test case that does that. Thats your regression test.
The test set should not have a fixed order or be dependent on each other. I feel like this is a really simple thing to handle and having ‘ordered’ test sets will introduce needless complications and reduce flexibility.
We’re not talking about ordered test sets here. Instead, we’re talking about being able to order test cases within a test set. I should be able to use the same automated test case in any test set, and in any order I wish by simply adding them in a certain sequence, and they should run in that sequence added. Of course I can create a separate test case that runs tests in a sequence (this is what I have to do now), but I have to then maintain more test cases, unnecessarily - and, if I ever want another one with a slight different order, then I have to create yet another test case vs just reordering them in a new test case. This requires extra coding, maintenance, support, processes, etc., which is not very efficient. I disagree that it would introduce needless complications and reduce flexibility - on the contrary - it only adds flexibility (ability to run test cases in any order I want), which is why every other testing suite I’ve used provides that ability.
Anil_G
ashokkarale
jinal.shah
Gautham_Pattabiraman
postwick
chandreshsinh.jadeja
vrdabberu
Ajay_Mishra
sven.wullum1
Vyshnavi_Nalumachu