activities

latest

false

UI Automation Activities

Last updated Oct 28, 2024

Advanced descriptor configuration

For more information on using the Selection Screen's more advanced features, check out the video below.

Duplicates

Indicating a target may produce duplicates, which are other UI elements that are perceived as extremely similar to the target UI element and might cause the automation to fail at runtime. When duplicates are found, they are highlighted in yellow and have a label above them containing a letter which corresponds to the targeting method that found each duplicate:

One of the best ways to avoid duplicates is indicating an anchor.

Anchors

To create a reliable descriptor, it is recommended to indicate an anchor after the target is indicated.

An anchor is another nearby element that helps to uniquely identify the target. When you indicate a target, UiPath tries to identify and automatically select an anchor. You can also manually add up to three anchors.

If only Strict selector is enabled as a targeting method, anchors are not used.

When to use an anchor

Anchors are useful when a target element is not unique. For example, a form in a desktop application typically contains multiple similar text boxes into which to enter various data. The way you identify what data needs to go into each one is by looking at the labels next to them. Similarly, when the label of a text box is added as its anchor, the robot uses the anchor to identify the text box to type into.

Sometimes multiple anchors are required to distinguish a target from other similar elements called duplicates. For example, if a form contains billing address and shipping address sections, the text boxes in them appear twice with the same label, so you need to add both the labels (e.g. "Zip code") and the title of the section (e.g. "Billing address") as anchors to identify the correct one.

Anchors can be found in several scenarios, as described below:

Anchor automatically found

After indicating a target, an attempt is made to auto-identify an anchor for it.

Important: If an anchor is not found with 100% certainty, or if duplicates are found, it is not considered as an auto-anchor.

If an auto-anchor is successfully found, it is highlighted in blue, and the Selection Options Helper confirms that the target and its anchor are OK.

Anchor identification method

In this section, you can find out more technical details about the method in which this search is performed.

Once a target is indicated, the type of the indicated UI element is identified. The following elements and directions are searched for:

Button - inside of the element.
Check box - to the right of the element.
Drop-down - to the left and above the element.
Input box - to the left and above the element. In the case of browser automation, the label is also searched inside the input box, looking for the AreaLabeledBy attribute. If this label is not identified, an element with the tag=label and for='id of the input box' is searched for.
Radio button - to the right of the element.
Image - to the right and below the element.

Depending on the type of element that is identified, a corresponding anchor is searched for the element automatically.

An automatically identified anchor is in the form of a text element that is always present near the target element and thus can be used to identify the element at runtime with more precision.

If none of the above directions provide results, a generic case is used, which searches in the element hierarchy for a suitable anchor.

Manually adding an anchor

In case an anchor is not automatically found and added to the specified target, you are notified by the Selection Options helper, as shown in the following screenshot:

In this situation, you can manually select an anchor for your target element. When hovering another element, it is highlighted blue and connected with a line to the target you have selected. To select the anchor you want, you can simply click the text you want to use as an anchor.

Hover menu

You can also edit the a specific element (be it target or anchor) by accessing the menu that is displayed when hovering over the target element. This menu has multiple options:

The button closes the indicate process and saves the changes you have made.

The button enables you to target elements regardless of their specific text content. This can be helpful when trying to select the first element in a search list when you don't know exactly what the list is going to be populated with, or when interacting with a dynamic element that changes its label due to various reasons. Using this feature when indicating a target uses only the Fuzzy selector method for identification, and strips down the Fuzzy selector to make sure the element is still targeted regardless of label changes.

By clicking the button, you can select an adjacent UI element that acts as an additional anchor. You can indicate up to three anchors for any target.

The button expands the target section in the Selection Options, enabling you to configure the targeting methods used for the current descriptor.

The button deletes the target you have indicated and lets you choose another one.

Configuring the descriptor

Options

The Options section can be expanded to reveal the following options:

Auto-apply changes - When selected, auto-applies changes to target and anchor elements.
Responsive websites - When selected, enables the anchor to automatically move from left to the top of the target, or from top to the left of the target, thus letting you automate websites that are responsive.
Element visibility - Checks whether the UI element is visible or not. There are three options available from the drop-down menu:
- None - Does not check for visibility.
- Interactive (for Fuzzy selector) - Default option. Checks if the element is potentially visible, ignoring page scroll and obstructions by other apps, or the fact that the application is minimized. This check is useful when trying to ensure that you are not targeting invisible elements that exist in the DOM but are hidden.
- Fully visible - Ensures that the UI element is visible on the screen.

Window selector

The Window Selector section can be expanded to view and edit the selector of the indicated target application. You can also use the Open in UI Explorer button to edit the window selector in the UI Explorer.

Target

A target is a UI element you want to interact with, for example a button to click, a text box to type into, or a part of a web page from which to copy text.

The under-the-hood process of finding an UI element uses several targeting methods:

These targeting methods are redundant to one another and there is a ranking in place according to which the targeting methods are used. The hierarchy takes into account the targeting power and the resilience to UI changes of each targeting method. Thus, the order to find the UI element is the following:

The primary targeting method is represented by selectors: Strict selector or Fuzzy selector.
The secondary targeting method is Computer Vision.
The tertiary targeting method is Image (disabled by default).

The icon is displayed next to the leading targeting method at design time, simulating the scenario at runtime.

Default project settings for targeting methods across technologies

The below table displays the default project settings for targeting methods across different technologies.

Targeting methods	Desktop applications	Web browsers	Java	SAP
Strict selector
Fuzzy selector
Computer Vision
Image

Project setting: Wait for primary targeting method until timeout

The Wait for primary targeting method until timeout project setting (Project Settings > UI Automation Modern), which is enabled by default in new projects starting with activity package version 23.10.3 and which can be enabled manually for projects created prior to this activity package version, offers the primary targeting method some extra time to consume its full timeout until any fallback targeting method can be used.

If one of the primary targeting methods manages to find the UI element until the timeout expires, then it's going to be used as the method to interact with the UI element. Strict selector is the targeting method that first searches for the UI element, followed shortly by the Fuzzy selector. Also, Strict selector has priority over Fuzzy selector.

When the UI element is not found by the primary targeting methods until the timeout expires, so if after all the retries within the allocated timeout neither Strict selector nor Fuzzy selector find the UI element, only then the secondary and the tertiary targeting methods start searching for the UI element within a timeout of their own.

This project setting also brings more predictability and visibility into selector related issues so that you can have better control over your UI elements targeting, while also having the solid fallback that Computer Vision provides.

If you disable this project setting, then all the targeting methods compete in a single timeout to find the UI element according to the hierarchy: a target found by lower ranking targeting method is used only if a higher ranking targeting method fails to do so.

Project setting: Show closest matches in the 'Element not found' error message

The Show closest matches in the 'Element not found' error message project setting (Project Settings > UI Automation Modern) is enabled by default in new projects starting with activity package version 23.10.3 and it can be enabled manually for projects created prior to this activity package version. When this setting is enabled alongside with the Wait for primary targeting method until timeout project setting, if a selector fails, a warning is displayed in the logs showcasing the method used to find the UI element together with ten suggestions of the closest selector matches that can be used to replace the faulty selector.

It is not recommended to manually edit the descriptor, as this entire process is thought out to be seamless for the user and should provide good results without the need for advanced configuration. Still, if you want to configure more advanced features for your descriptor, you can enable, disable, and edit the targeting methods.

Strict selector

A selectors is an XML fragment that contains the attributes of an UI element and its parents, including the target application. Selectors are used for targeting UI elements on the screen, by inspecting them and their attributes, and matching them with the attributes in the selector. You can find out more on what a selector is in the UiPath ecosystem by viewing this page.

The Strict selector of the target element can be edited from its dedicated section in the Selection Options window. You can also use the Open in UI Explorer button to edit the selector in the UI Explorer.

The Strict selector field supports the use and creation of variables and arguments, from the context menu (accessible by right-clicking).

You can also select a part of the selector and use the context menu to directly convert it into a variable that is pre-populated with your selection.

You can also select an already created variable from the Variable Name drop-down menu and use it in your selector.

The Override default value check box enables you to replace the value in your predefined variable with your current selection.

The Confirm button saves the changes you have made in your selector.

The following hotkeys are available:

Ctrl+Space: choose variable
Ctrl+Shift+Space: choose argument
Ctrl+K: create variable
Ctrl+M: create argument

Important:

If there are both a variable and an argument with the same name, the variable is always defaulted to and used at runtime.

Arguments with names identical to variables are hidden from the variable selection menu in the Selection Screen.

Changes done with variables and arguments can be propagated throughout all the other instances where they are found. For example, if a variable is used in both the Strict selector and Fuzzy selector fields, changing it in one of the fields generates a message that informs you the same value was found in another field, and offering you the option to replace all instances with the new value.

Strict selectors also support wildcards, which can be used to replace zero or multiple characters in a string. You can find out more about this functionality here.

Fuzzy selector

A fuzzy selector is a selector that takes into account possible changes in the attributes of an element, identifying them based on a pattern, rather than on an exact match. This method works best when used with anchors.

The fuzzy selector is very similar to Strict selector, with the difference that its attributes are fuzzy matched instead of perfectly matched. This means that a fuzzy selector can output multiple elements, as opposed to Strict selector which outputs a single element at a time.

The fuzzy selector can be edited from its dedicated section in the Selection Options window.

The matching accuracy of the fuzzy selector is based on the Levenshtein distance algorithm to count the similarity of selectors. The default value is 0.5 and it can be configured either from the Accuracy slider or directly from the input field by typing a valid value. The matching accuracy can be set within a range of 0.4 to 1 where a lower value indicates a higher degree of fuzziness, meaning the fuzzy selector outputs more elements, and a higher value indicates a lower degree of fuzziness, meaning the fuzzy selector outputs less elements.

Just like the Strict selector, the fuzzy selector field supports the use and creation of variables and arguments and it also supports wildcards. For more details, refer to the information provided above.

If the indicated element contains text, the text is displayed in the input box below the fuzzy selector edit window. If text is displayed in this input box, at runtime it must be found in the element itself to be considered a match.

The drop-down menu next to the input box has the following options:

Text is/InnerText is/FullText is (depending on the text scraping method used) - Displays the text that is scraped; the best scraping method is automatically identified and it is recommended to use it.
Ignore text - Ignores the text specified in the input box.

When the target or anchor search using a fuzzy selector fails, a custom exception is thrown displaying the closest matches to repair the failing selector.

The UI element could not be located using the configured Fuzzy selector for Target or Anchor.

'Target' search failed for selector:
	<ctrl name='Use*' role='radio button' /><ctrl name='Use mimi' role='XYZ text' automationid='' />
These are the closest selector matches found:
	[84%] <ctrl name='Use Amount' role='radio button' /><ctrl role='text' automationid='' />
	[83%] <ctrl name='Use Both' role='radio button' /><ctrl role='text' automationid='' />
	[82%] <ctrl name='Use Piece Count' role='radio button' /><ctrl role='text' automationid='' />
	
'Anchor 1' search failed for selector:
	<ctrl name='Configuration' role='text' /><ctrl name='Configuration' role='text' automationid='' check:text='XYZ Config' />
These are the closest selector matches found:
	[78%] <ctrl name='Configuration' role='text' /><ctrl name='Configuration' role='text' automationid='' check:text='Configuration' />
	[64%] <ctrl name='Transaction #:' role='text' /><ctrl name='Transaction #:' role='text' automationid='' check:text='Transaction #:' />
	[63%] <ctrl name='Deposit transaction' role='text' /><ctrl name='Deposit transaction' role='text' automationid='' check:text='Deposit transaction' />
	
'Anchor 2' search failed for selector:
	<ctrl name='Use Both' role='radio button' automationid='' check:fulltext='Use XYZZZZZZ' />
These are the closest selector matches found:
	[83%] <ctrl name='Use Both' role='radio button' automationid='' check:fulltext='Use Both&#xD;&#xA;' />
	[74%] <ctrl name='Use Amount' role='radio button' automationid='' check:fulltext='Use Amount&#xD;&#xA;' />
	[68%] <ctrl name='Use Piece Count' role='radio button' automationid='' check:fulltext='Use Piece Count&#xD;&#xA;' />The UI element could not be located using the configured Fuzzy selector for Target or Anchor.

'Target' search failed for selector:
	<ctrl name='Use*' role='radio button' /><ctrl name='Use mimi' role='XYZ text' automationid='' />
These are the closest selector matches found:
	[84%] <ctrl name='Use Amount' role='radio button' /><ctrl role='text' automationid='' />
	[83%] <ctrl name='Use Both' role='radio button' /><ctrl role='text' automationid='' />
	[82%] <ctrl name='Use Piece Count' role='radio button' /><ctrl role='text' automationid='' />
	
'Anchor 1' search failed for selector:
	<ctrl name='Configuration' role='text' /><ctrl name='Configuration' role='text' automationid='' check:text='XYZ Config' />
These are the closest selector matches found:
	[78%] <ctrl name='Configuration' role='text' /><ctrl name='Configuration' role='text' automationid='' check:text='Configuration' />
	[64%] <ctrl name='Transaction #:' role='text' /><ctrl name='Transaction #:' role='text' automationid='' check:text='Transaction #:' />
	[63%] <ctrl name='Deposit transaction' role='text' /><ctrl name='Deposit transaction' role='text' automationid='' check:text='Deposit transaction' />
	
'Anchor 2' search failed for selector:
	<ctrl name='Use Both' role='radio button' automationid='' check:fulltext='Use XYZZZZZZ' />
These are the closest selector matches found:
	[83%] <ctrl name='Use Both' role='radio button' automationid='' check:fulltext='Use Both&#xD;&#xA;' />
	[74%] <ctrl name='Use Amount' role='radio button' automationid='' check:fulltext='Use Amount&#xD;&#xA;' />
	[68%] <ctrl name='Use Piece Count' role='radio button' automationid='' check:fulltext='Use Piece Count&#xD;&#xA;' />

The closest matches are not displayed if you alter the default selector by adding or removing selector nodes or if the target or anchor search failed because of multiple matches.

You can find more information about selectors in the Studio guide.

Computer Vision

Computer Vision is an AI powered engine enabling UiPath Robots to see and understand every UI element on a computer screen. Computer Vision enhances targeting: when the driver fails to 'see' some UI elements, Computer Vision can enable the UI automation. So, it increases reliability and resilience to change in UI-based automations and it is a powerful fallback for the other targeting methods.

Computer Vision is widely available: whether it is on Studio Desktop for Windows and cross-platform projects or on Studio Web.

To use Computer Vision as a targeting method, you need to go to Project Settings > Computer Vision and make sure to set the correct server endpoint. You can leave this field as it is and use UiPath Cloud, which is filled-in by default, or you can enter another value either by selecting any of the other public endpoints specific to a server region or your own on-premises server.

Another way of using Computer Vision as a targeting method is to set Use Local Server to True. This implies that the UiPath.ComputerVision.LocalServer package is installed in your project. When Local Server is enabled, the server value is over-written. However, the Local Server is not available in cross-platform projects.

Image

This targeting method uses an image in the application window that identifies the element, for example an icon or a button.

If Image automation is used for your target, you can also view the screenshot of the element, as well as adjust the image accuracy. The default value is 0.8 and it can be configured either from the Accuracy slider or directly from the input field by typing a valid value. The accuracy can be set within a range of 0.4 to 1. For further information, check the Image and Text Automation section.

If you want to define Image-only targets, you can use the Image region selection (F3) or draw a box selection of the area you want to use as your target/anchor while indicating.

You can also use the refresh button to manually refresh the image that you have selected, in case the image of the element changed since it was indicated.

The Image targeting method is disabled by default in Project Settings.

To enable it at project level, go to Project Settings > UI Automation Modern > Targeting methods - Desktop applications / Targeting methods - Web browsers / Targeting methods - Java / Targeting methods - SAP (depending on what type of application you want to automate) > Image > select True.

To enable the method on a specified element, the element must be valid.

Native text

If you selected the Enable Native text check box when indicating the element, the Native Text targeting method is used. This method enables you to use all the text labels that are identified with the Native method for identifying the element you want to interact with.

The text label identified through this method can be edited from its dedicated section, and you can specify whether you want the text to be case sensitive or not by using the Toggle case sensitive button.

Offline editing

When an application is no longer open but you want to edit its descriptor, selecting Edit target (E) from the contextual menu in the body of the activity prompts you to Retry, Edit offline, or Cancel.

Selecting the Edit offline option opens the selection screen in offline editing mode. In offline editing mode, you cannot add anchors, validate the descriptor, or other functionality that requires a live application.

If you select the Do not show again checkbox, the next time you want to edit the target, it opens the selection screen directly.

Validating the descriptor

The validation process enables you to check how effective the current selection is in identifying the target by checking it using all selected methods and anchors. You can validate your selection by using the Validate button in the Selection Screen. The result of the validation is displayed at the top of the Selection Options window. If adjustments are needed, a message is displayed with information on how to improve the selection.

If you have a target and anchor pair, each of them uses all the available methods for targeting. To find the final target, all possible combinations for each of these elements are matched to elements on the screen and a score is given after the match.

After performing the validation, if there are any duplicates found, they are highlighted in the target application. If no duplicates are found, icons appear alongside the targeting methods, informing you of the state of each targeting method in the given situation.

The performance of each method is displayed in the Configuration section of the Selection Options window using one of the following icons:

- The targeting method that was fastest to successfully identify the element.

- The method successfully identified the element, but it was not the fastest.