Executing Tests

You can execute tests either manually or automatically. Manual test executions are invoked through Test Manager, whereas automated executions are invoked through Test Manager and executed via Orchestrator. To execute your test cases, you must first create a test set, which will be later executed.

Conditions

You can use both Execute Manually and Execute Automated for test sets created in Test Manager as well as the linked test sets from Orchestrator
Only the test case versions included in the test set are executed. If you want to execute new test cases, you need to assign new versions to your test set. For Orchestrator test sets, you need to make sure that the latest packages are published.

Automated execution

Automated execution refers to test sets that have been linked from Orchestrator or test sets created in Test Manager that have test cases linked from Studio.

Before you execute automated tests in Test Manager, make sure you create test sets in Orchestrator.

Manual execution

Manual execution refers to test sets created in Test Manager, including test cases that have manual steps. Manual execution can be done for test sets created in Test Manager as well as the linked test sets from Orchestrator.

Executing manual tests

Execute Test Sets Manually

Both Test Manager test sets and test sets that have been linked from Orchestrator can be executed manually. if manual steps have been defined for the test cases. For more information, see Executing Tests.

Open your project in Test Manager and then navigate to Test Sets.
Select a test set and click the vertical ellipsis.
Select Execute Manually. The Manual execution assistant opens in a new window. Make sure your popup blocker allows the window to be opened.

For more information on test results, see Analyzing Test Results.

Note:

Test cases that do not contain any manual steps will be skipped during execution without producing any result.
Automated test case can have manual steps defined and can also be executed manually.

Alternatively, you can open your test set, click Execute and select Manually from the dropdown.

Manual execution assistant

The manual execution assistant lets you perform a set of manual tests one after each other. You need to set a result on each manual step to advance to the next test case in the test set.

Through the Manual Execution Assistant, you need to treat and mark steps in each case either as passed or failed.

Use Set Failed or Set Passed to mark a step in each case. Add comments, especially in case of a failure, to provide details about the particular result. You can set several consecutive steps as passed by clicking Set Passed Till Here.

Once the last step of a test case has a result assigned, you can click Next. This enables you to go to the next test case from the test set. Once you are finished with the last test case, clicking Done finishes the execution entirely.

Executing automated tests

You can execute all test sets using Execute Automated. Orchestrator is invoked for test cases that have been linked to Test Manager. To execute test sets linked from Orchestrator, make sure you create your test sets in Orchestrator.

You can also execute automated test if the test set includes test cases linked from Studio. For more information, see Executing Tests.

Conditions

Important: To use the Execute Automated option, you need at least one automated test case in a test set.

Test execution for test sets linked from Orchestrator includes only the test cases that have been published to the Orchestrator Tenant Process Feed in the latest package version. If you want to execute new test cases, you need to publish a new package version in Orchestrator, and then link the test set to Test Manager.
If your Test Manager test set includes test cases from multiple Studio projects, then the project that has the highest number of test cases is considered for execution. Therefore, only the test cases from this project will be executed. In this case, you will receive the following message within Test Manager: "Test cases from multiple packages have been selected. Some of the test cases will not execute while running this test set."
You can execute test sets including test cases from multiple Studio projects, only if you have a test set linked from Orchestrator.
If your test set includes test cases that are duplicate in multiple Studio projects, the sorting is done by the project name in descending order. Only the test cases from the project that is placed first in the list will be executed.

Execute Automated Test Sets

Open your project in Test Manager and then navigate to Test Sets.
Select a test set and click the vertical ellipsis.
Select Execute Automated.
For more information on test results, see Analyzing Test Results.
Note:
- Test cases that cannot be executed are skipped during execution through Orchestrator.
- Test cases that are not executed generate test case logs with no result.

Alternatively, you can open your test set, click Execute and select Automated from the dropdown.

Troubleshooting automated executions

Automated execution could not start since the package was not found on Orchestrator. Only test cases from packages which have been published to Orchestrator can be executed.

Solution

You need to publish the Studio package to Orchestrator. This issue occurs when a test case is linked from Studio to Test Manager but the package is not yet published to Orchestrator.

Test cases could not be executed since no executable versions could be found. Only test cases published with the latest package version to Orchestrator can be executed.

Solution

This issue occurs when test cases in a test set are not published in the latest Studio package to Orchestrator. You might get the same error when there is no process/release found on Orchestrator for the studio package.