test-suite

2023.4

false

Test Suite User Guide

DELIVERY:

Automation Cloud Automation Cloud Public Sector Automation Suite Standalone

Last updated Aug 13, 2024

Executing Tests

You can execute tests either manually or automatically. Manual test executions are invoked through Test Manager, whereas automated executions are invoked through Test Manager and executed via Orchestrator. To execute your test cases, you must first create a test set, which will be later executed.

Conditions

You can use both Execute Manually and Execute Automated for test sets created in Test Manager as well as the linked test sets from Orchestrator
Only the test case versions included in the test set are executed. If you want to execute new test cases, you need to assign new versions to your test set. For Orchestrator test sets, you need to make sure that the latest packages are published.

Automated execution

Automated execution refers to test sets that have been linked from Orchestrator or test sets created in Test Manager that have test cases linked from Studio.

Before you execute automated tests in Test Manager, make sure you create test sets in Orchestrator.

Manual execution

Manual execution refers to test sets created in Test Manager, including test cases that have manual steps. Manual execution can be done for test sets created in Test Manager as well as the linked test sets from Orchestrator.

Executing manual tests

Execute Test Sets Manually

Both Test Manager test sets and test sets that have been linked from Orchestrator can be executed manually. if manual steps have been defined for the test cases. For more information, see Executing Tests.

Open your project in Test Manager and then navigate to Test Sets.
Select a test set and click the vertical ellipsis.
Select Execute Manually. The Manual execution assistant opens in a new window. Make sure your popup blocker allows the window to be opened.

For more information on test results, see Analyzing Test Results.

Note:

Test cases that do not contain any manual steps will be skipped during execution without producing any result.
Automated test case can have manual steps defined and can also be executed manually.

Alternatively, you can open your test set, click Execute and select Manually from the dropdown.

Manual execution assistant

The manual execution assistant lets you perform a set of manual tests one after each other. You need to set a result on each manual step to advance to the next test case in the test set.

Through the Manual Execution Assistant, you need to treat and mark steps in each case either as passed or failed.

Use Set Failed or Set Passed to mark a step in each case. Add comments, especially in case of a failure, to provide details about the particular result. You can set several consecutive steps as passed by clicking Set Passed Till Here.

Once the last step of a test case has a result assigned, you can click Next. This enables you to go to the next test case from the test set. Once you are finished with the last test case, clicking Done finishes the execution entirely.

Executing automated tests

You can execute all test sets using Execute Automated. Orchestrator is invoked for test cases that have been linked to Test Manager. To execute test sets linked from Orchestrator, make sure you create your test sets in Orchestrator.

You can also execute automated test if the test set includes test cases linked from Studio. For more information, see Executing Tests.

Conditions

Important: To use the Execute Automated option, you need at least one automated test case in a test set.

Test execution for test sets linked from Orchestrator includes only the test cases that have been published to the Orchestrator Tenant Process Feed in the latest package version. If you want to execute new test cases, you need to publish a new package version in Orchestrator, and then link the test set to Test Manager.
If your Test Manager test set includes test cases from multiple Studio projects, then the project that has the highest number of test cases is considered for execution. Therefore, only the test cases from this project will be executed. In this case, you will receive the following message within Test Manager: "Test cases from multiple packages have been selected. Some of the test cases will not execute while running this test set."
You can execute test sets including test cases from multiple Studio projects, only if you have a test set linked from Orchestrator.
If your test set includes test cases that are duplicate in multiple Studio projects, the sorting is done by the project name in descending order. Only the test cases from the project that is placed first in the list will be executed.

Execute Automated Test Sets

Open your project in Test Manager and then navigate to Test Sets.
Select a test set and click the vertical ellipsis.
Select Execute Automated.
For more information on test results, see Analyzing Test Results.
Note:
- Test cases that cannot be executed are skipped during execution through Orchestrator.
- Test cases that are not executed generate test case logs with no result.

Alternatively, you can open your test set, click Execute and select Automated from the dropdown.

Re-executing test executions

If you re-execute a test execution the old execution results are deleted and replaced with the new results. If you don't want to replace a test exection result, then run the test set again. This will generate new test execution results, independent from the previous ones.

The pre-defined user roles that can perform re-executions are: Test Manager, Test Executor and Manual Tester.

Re-executing test set executions

Go to Test Results and choose one of the following:
- Select More Options for the test execution that you want to redo.
- Open the test execution and select Tasks.
Select one of the following:
- Re-execute manually (available only for manual test executions)
- Re-execute automated (available only for automated test executions)
The Re-execute dialog appears.
Select Confirm and execute the manual steps or wait for the automated test set to finish.

Note:
Re-executing test set executions doesn't display the activity coverage anymore.

Re-executing test case executions

You can re-execute one or more test case executions. For data-driven test cases, you can only re-execute one test case execution at a time. You can't re-execute data-driven test case executions in bulk.

Note: Even if you publish a newer version of the test case, the version that you re-execute is the same version that is recorded in the test execution.

Go to Test Results and open a test set execution.
Select the checkboxes for one or more test case executions and then select Re-execute selection.
Select one of the following:
- Re-execute manually (available only for manual test executions)
- Re-execute automated (available only for automated test executions)
The Re-execute dialog appears.
Select Confirm and execute the manual steps or wait for the automated test case to finish.

Troubleshooting automated executions

Automation already linked to another test case in Test Manager

Issue description

If you attempt to link a Studio test case that's already linked to a different Test Manager instance than the one you're currently working in, the following message is displayed: "Automation already linked to another test case in Test Manager".

Solution

You can't directly unlink the test case. However, you can work around this by converting the test case to a workflow and then converting it back to a test case. This process changes the test case ID, allowing it to be linked to your current Test Manager instance, but without modifying the underlying logic of the test case.

Visit Convert workflow to test case for information about converting a workflow into a test case.

Automated execution could not start since the package was not found on Orchestrator. Only test cases from packages which have been published to Orchestrator can be executed.

Solution

You need to publish the Studio package to Orchestrator. This issue occurs when a test case is linked from Studio to Test Manager but the package is not yet published to Orchestrator.

Test cases could not be executed since no executable versions could be found. Only test cases published with the latest package version to Orchestrator can be executed.

Solution

This issue occurs when test cases in a test set are not published in the latest Studio package to Orchestrator. You might get the same error when there is no process/release found on Orchestrator for the Studio package.