Test Results

Test results in Test Manager provide design, execution, update, or deletion information for the test sets that have been executed.

When you start executing a test set, Test Manager goes through the following process:

A Test Execution is created in the Test Results section of Test Manager. This serves as the container for the test results.
For every test case within the executed test set, an empty Test Case Log is generated. This log is attached to the Test Execution and will hold its associated test results.
As the execution continues, the Test Manager fills each Test Case Log with results and related log information.

Note: Every time a test execution is created, Test Case Logs for all associated test cases from the original test set are added. This means existing Test Executions remain consistent, regardless of updates or deletions made to the original test set.

The chart below illustrates the process of executing a test set in Test Manager.

As a consequence, after a test set has been executed, any changes to the test set or its associated test cases do not affect the results from previous executions. Even when a test set is deleted, all the test executions remain unchanged.

Analyze Test Results

To view test executions, open Test Results in Test Manager. The execution of each test is listed as a test execution entry. To understand how the test executions work behind the scenes, see Test Results.

By analyzing your test execution, you can take the following actions:

Find test results that have been executed manually or automatically through Orchestrator.
Check the progress on running test executions.
Examine logs and attachments.
Create defects in your defect management system directly from Test Manager, if you have an Application Lifecycle Management tool integration.

By default, the test executions are sorted based on the date on which the execution was finished. Currently running and pending executions are placed at the top.

You can identify the status through the color codes assigned to each test execution, as follows.:

Green:- Test cases that passed.
Red: Test cases that failed.
Grey: Test cases without a definitive results, such as test cases that have not been executed yet but are part of a test set that is currently being executed.

Results from automated tests

The results from automated test executions are imported from Orchestrator automatically. To have your automated tests imported to Test Manager, you need to meet the following condition: The automated test needs to be part of a test set on your Test Manager project. For more information, see Automated Tests.

Note: If a Test Execution from Orchestrator holds results for test cases from several projects in Test Manager, the test execution is split in Test Manager. The results appear in the projects where the test cases are.

Working with test execution results

To get detailed results from a test execution, go to Test Results and click an entry to open the detailed view. A typical test execution detailed view shows information on when the executions started, duration, and test case logs.

You can use the Reporting Date filter to set a single date for all the results from the test execution. This is useful when you run test overnight, where some test cases are executed before and after midnight. Otherwise, the results are going to be split between different dates in the reports.

In the Results section you can examine all the test cases that were executed within the test set. You can take actions for each test case. For more info, see Test case logs

Test case logs

The test case logs hold information such as steps description, error messages, and screenshots that were collected during a test set execution. To open a test case log, go to Test Results, open a test execution and then click a test case Key.

Note: The Screenshots tab is available only if screenshots were taken during the test case execution. To make sure that Orchestrator takes screenshots during executions, check for Test Case Execution Artifacts permissions.

You can examine the test case details by clicking the Open Test Case icon.

Create defects from results

You can create defect reports including the execution log to your external defect management system, if you already have it integrated with Test Manager. For more information, see Defect synchronization.

Defect Synchronization

You can synchronize execution results with external tools, as part of the Application Lifecycle Management tool integration. Information that is gathered during execution, such as results, logs, timestamps, and other details is synchronized with the tool that you have integrated with Test Manager.

Please note that at the moment there can be only one connection which is enabled for defect synchronization overall per project.

Note:

To synchronize defects, you need to configure a connector in Test Manager. See available connections in Test Manager.
You need to have executed a test set first.

Create defect

You can create defects when you access test case logs in the Test Results page.

Navigate to Test Results
Open a test result and the click the test case key to open the logs.
Click Tasks and select Create Defect.

The defect is created and synchronized with your external tool. You can open the defect directly in the tool (e.g., Atlassian Jira) by navigating to the test execution result that has a synchronized defect.

Unlink defect

When you unlink defects from an external tool, the entry created in the tool remains unchanged. In Test Manager, the test execution result will not be linked with an external tool.