- Overview
- UI Automation
- Applications and technologies automated with UI Automation
- Project compatibility
- UI-ANA-016 - Pull Open Browser URL
- UI-ANA-017 - ContinueOnError True
- UI-ANA-018 - List OCR/Image Activities
- UI-DBP-006 - Container Usage
- UI-DBP-013 - Excel Automation Misuse
- UI-DBP-030 - Forbidden Variables Usage In Selectors
- UI-PRR-001 - Simulate Click
- UI-PRR-002 - Simulate Type
- UI-PRR-003 - Open Application Misuse
- UI-PRR-004 - Hardcoded Delays
- UI-REL-001 - Large Idx in Selectors
- UI-SEC-004 - Selector Email Data
- UI-SEC-010 - App/Url Restrictions
- UI-USG-011 - Non Allowed Attributes
- UX-SEC-010 - App/Url Restrictions
- UX-DBP-029 - Insecure Password Use
- UI-PST-001 - Audit Log Level in Project Settings
- UiPath Browser Migration Tool
- Clipping region
- App/Web Recorder
- Advanced descriptor configuration
- UI elements extraction
- Table Extraction
- Linux Robots
- UI Automation Browser Connection
- Computer Vision Recorder
- Activate
- Anchor Base
- Attach Browser
- Attach Window
- Block User Input
- Callout
- Check
- Click
- Click Image
- Click Image Trigger
- Click OCR Text
- Click Text
- Click Trigger
- Close Application
- Close Tab
- Close Window
- Context Aware Anchor
- Copy Selected Text
- Element Attribute Change Trigger
- Element Exists
- Element Scope
- Element State Change Trigger
- Export UI Tree
- Extract Structured Data
- Find Children
- Find Element
- Find Image
- Find Image Matches
- Find OCR Text Position
- Find Relative Element
- Find Text Position
- Get Active Window
- Get Ancestor
- Get Attribute
- Get Event Info
- Get From Clipboard
- Get Full Text
- Get OCR Text
- Get Password
- Get Position
- Get Source Element
- Get Text
- Get Visible Text
- Go Back
- Go Forward
- Go Home
- Google Cloud Vision OCR
- Hide Window
- Highlight
- Hotkey Trigger
- Hover
- Hover Image
- Hover OCR Text
- Hover Text
- Image Exists
- Indicate On Screen
- Inject .NET Code
- Inject Js Script
- Invoke ActiveX Method
- Key Press Trigger
- Load Image
- Maximize Window
- Microsoft Azure Computer Vision OCR
- Microsoft OCR
- Microsoft Project Oxford Online OCR
- Minimize Window
- Monitor Events
- Mouse Trigger
- Move Window
- Navigate To
- OCR Text Exists
- On Element Appear
- On Element Vanish
- On Image Appear
- On Image Vanish
- Open Application
- Open Browser
- Refresh Browser
- Replay User Event
- Restore Window
- Save Image
- Select Item
- Select Multiple Items
- Send Hotkey
- Set Clipping Region
- Set Focus
- Set Text
- Set To Clipboard
- Set Web Attribute
- Show Window
- Start Process
- System Trigger
- Take Screenshot
- Tesseract OCR
- Text Exists
- Tooltip
- Type Into
- Type Secure Text
- Use Foreground
- Wait Attribute
- Wait Element Vanish
- Wait Image Vanish
- Application Event Trigger
- Check/Uncheck
- Check App State
- Check Element
- Click
- Click Event Trigger
- Close Popup
- Drag and Drop
- Extract Table Data
- For Each UI Element
- Get Attribute
- Get Browser Data
- Get Text
- Get URL
- Go to URL
- Highlight
- Hover
- Inject Js Script
- Keyboard Shortcuts
- Keypress Event Trigger
- Mouse Scroll
- Navigate Browser
- Select Item
- Set Browser Data
- Set Runtime Browser
- Set Text
- Take Screenshot
- Type Into
- Use Application/Browser
- Perform browser search and retrieve results using UI Automation APIs
- Web Browsing
- Find Images
- Click Images
- Trigger and Monitor Events
- Create and Override Files
- HTML Pages: Extract and Manipulate Information
- Window Manipulation
- Automated List Selection
- Find and Manipulate Window Elements
- Manage Text Automation
- Load and Process Images
- Manage Mouse Activated Actions
- Automate Application Runtime
- Automated Run of a Local Application
- Browser Navigation
- Web Automation
- Trigger Scope Example
- Computer Vision Local Server
- Mobile Automation
- Release notes
- Project compatibility
- Get Log Types
- Get Logs
- Get Page Source
- Get Device Orientation
- Get Session Identifier
- Install App
- Manage Current App
- Manage Other App
- Open DeepLink
- Open URL
- Mobile Device Connection
- Directional Swipe
- Draw Pattern
- Positional Swipe
- Press Hardware Button
- Set Device Orientation
- Take Screenshot
- Take Screenshot Part
- Element Exists
- Execute Command
- Get Attribute
- Get Selected Item
- Get Text
- Set Selected Item
- Set Text
- Swipe
- Tap
- Type Text
- Terminal
Advanced descriptor configuration
For more information on using the Selection Screen's more advanced features, check out the video below.
Indicating a target may produce duplicates, which are other UI elements that are perceived as extremely similar to the target UI element and might cause the automation to fail at runtime. When duplicates are found, they are highlighted in yellow and have a label above them containing a letter which corresponds to the targeting method that found each duplicate:
- S - Strict selector
- F - Fuzzy selector
- CV - Computer Vision
- I - Image
One of the best ways to avoid duplicates is indicating an anchor.
To create a reliable descriptor, it is recommended to indicate an anchor after the target is indicated.
An anchor is another nearby element that helps to uniquely identify the target. When you indicate a target, UiPath tries to identify and automatically select an anchor. You can also manually add up to three anchors.
If only Strict selector is enabled as a targeting method, anchors are not used.
Anchors are useful when a target element is not unique. For example, a form in a desktop application typically contains multiple similar text boxes into which to enter various data. The way you identify what data needs to go into each one is by looking at the labels next to them. Similarly, when the label of a text box is added as its anchor, the robot uses the anchor to identify the text box to type into.
Sometimes multiple anchors are required to distinguish a target from other similar elements called duplicates. For example, if a form contains billing address and shipping address sections, the text boxes in them appear twice with the same label, so you need to add both the labels (e.g. "Zip code") and the title of the section (e.g. "Billing address") as anchors to identify the correct one.
Anchors can be found in several scenarios, as described below:
After indicating a target, an attempt is made to auto-identify an anchor for it.
In this section, you can find out more technical details about the method in which this search is performed.
Once a target is indicated, the type of the indicated UI element is identified. The following elements and directions are searched for:
- Button - inside of the element.
- Check box - to the right of the element.
- Drop-down - to the left and above the element.
- Input box - to the left and above the
element. In the case of browser automation, the label is also searched inside the
input box, looking for the
AreaLabeledBy
attribute. If this label is not identified, an element with thetag=label
andfor='id of the input box'
is searched for. - Radio button - to the right of the element.
- Image - to the right and below the element.
Depending on the type of element that is identified, a corresponding anchor is searched for the element automatically.
An automatically identified anchor is in the form of a text element that is always present near the target element and thus can be used to identify the element at runtime with more precision.
If none of the above directions provide results, a generic case is used, which searches in the element hierarchy for a suitable anchor.
In this situation, you can manually select an anchor for your target element. When hovering another element, it is highlighted blue and connected with a line to the target you have selected. To select the anchor you want, you can simply click the text you want to use as an anchor.
You can also edit the a specific element (be it target or anchor) by accessing the menu that is displayed when hovering over the target element. This menu has multiple options:
The button closes the indicate process and saves the changes you have made.
The button enables you to target elements regardless of their specific text content. This can be helpful when trying to select the first element in a search list when you don't know exactly what the list is going to be populated with, or when interacting with a dynamic element that changes its label due to various reasons. Using this feature when indicating a target uses only the Fuzzy selector method for identification, and strips down the Fuzzy selector to make sure the element is still targeted regardless of label changes.
By clicking the button, you can select an adjacent UI element that acts as an additional anchor. You can indicate up to three anchors for any target.
The button expands the target section in the Selection Options, enabling you to configure the targeting methods used for the current descriptor.
The button deletes the target you have indicated and lets you choose another one.
The Options section can be expanded to reveal the following options:
- Auto-apply changes - When selected, auto-applies changes to target and anchor elements.
- Responsive websites - When selected, enables the anchor to automatically move from left to the top of the target, or from top to the left of the target, thus letting you automate websites that are responsive.
- Element visibility - Checks
whether the UI element is visible or not. There are three options available from the
drop-down menu:
- None - Does not check for visibility.
- Interactive (for Fuzzy selector) - Default option. Checks if the element is potentially visible, ignoring page scroll and obstructions by other apps, or the fact that the application is minimized. This check is useful when trying to ensure that you are not targeting invisible elements that exist in the DOM but are hidden.
- Fully visible - Ensures that the UI element is visible on the screen.
The Window Selector section can be expanded to view and edit the selector of the indicated target application. You can also use the Open in UI Explorer button to edit the window selector in the UI Explorer.
A target is a UI element you want to interact with, for example a button to click, a text box to type into, or a part of a web page from which to copy text.
The under-the-hood process of finding an UI element uses several targeting methods:
- The primary targeting method is represented by selectors: Strict selector or Fuzzy selector.
- The secondary targeting method is Computer Vision.
- The tertiary targeting method is Image (disabled by default).
The icon is displayed next to the leading targeting method at design time, simulating the scenario at runtime.
Default project settings for targeting methods across technologies
The below table displays the default project settings for targeting methods across different technologies.
Targeting methods | Desktop applications | Web browsers | Java | SAP |
---|---|---|---|---|
Strict selector | ||||
Fuzzy selector | ||||
Computer Vision | ||||
Image |
Project setting: Wait for primary targeting method until timeout
The Wait for primary targeting method until timeout project setting (Project Settings > UI Automation Modern), which is enabled by default in new projects starting with activity package version 23.10.3 and which can be enabled manually for projects created prior to this activity package version, offers the primary targeting method some extra time to consume its full timeout until any fallback targeting method can be used.
If one of the primary targeting methods manages to find the UI element until the timeout expires, then it's going to be used as the method to interact with the UI element. Strict selector is the targeting method that first searches for the UI element, followed shortly by the Fuzzy selector. Also, Strict selector has priority over Fuzzy selector.
When the UI element is not found by the primary targeting methods until the timeout expires, so if after all the retries within the allocated timeout neither Strict selector nor Fuzzy selector find the UI element, only then the secondary and the tertiary targeting methods start searching for the UI element within a timeout of their own.
This project setting also brings more predictability and visibility into selector related issues so that you can have better control over your UI elements targeting, while also having the solid fallback that Computer Vision provides.
If you disable this project setting, then all the targeting methods compete in a single timeout to find the UI element according to the hierarchy: a target found by lower ranking targeting method is used only if a higher ranking targeting method fails to do so.
Project setting: Show closest matches in the 'Element not found' error message
The Show closest matches in the 'Element not found' error message project setting (Project Settings > UI Automation Modern) is enabled by default in new projects starting with activity package version 23.10.3 and it can be enabled manually for projects created prior to this activity package version. When this setting is enabled alongside with the Wait for primary targeting method until timeout project setting, if a selector fails, a warning is displayed in the logs showcasing the method used to find the UI element together with ten suggestions of the closest selector matches that can be used to replace the faulty selector.
It is not recommended to manually edit the descriptor, as this entire process is thought out to be seamless for the user and should provide good results without the need for advanced configuration. Still, if you want to configure more advanced features for your descriptor, you can enable, disable, and edit the targeting methods.
A selectors is an XML fragment that contains the attributes of an UI element and its parents, including the target application. Selectors are used for targeting UI elements on the screen, by inspecting them and their attributes, and matching them with the attributes in the selector. You can find out more on what a selector is in the UiPath ecosystem by viewing this page.
The Strict selector of the target element can be edited from its dedicated section in the Selection Options window. You can also use the Open in UI Explorer button to edit the selector in the UI Explorer.
The Strict selector field supports the use and creation of variables and arguments, from the context menu (accessible by right-clicking).
You can also select a part of the selector and use the context menu to directly convert it into a variable that is pre-populated with your selection.
You can also select an already created variable from the Variable Name drop-down menu and use it in your selector.
The Override default value check box enables you to replace the value in your predefined variable with your current selection.
The Confirm button saves the changes you have made in your selector.
The following hotkeys are available:
Ctrl
+Space
: choose variableCtrl
+Shift
+Space
: choose argumentCtrl
+K
: create variableCtrl
+M
: create argument
If there are both a variable and an argument with the same name, the variable is always defaulted to and used at runtime.
Arguments with names identical to variables are hidden from the variable selection menu in the Selection Screen.
Changes done with variables and arguments can be propagated throughout all the other instances where they are found. For example, if a variable is used in both the Strict selector and Fuzzy selector fields, changing it in one of the fields generates a message that informs you the same value was found in another field, and offering you the option to replace all instances with the new value.
Strict selectors also support wildcards, which can be used to replace zero or multiple characters in a string. You can find out more about this functionality here.
A fuzzy selector is a selector that takes into account possible changes in the attributes of an element, identifying them based on a pattern, rather than on an exact match. This method works best when used with anchors.
The fuzzy selector is very similar to Strict selector, with the difference that its attributes are fuzzy matched instead of perfectly matched. This means that a fuzzy selector can output multiple elements, as opposed to Strict selector which outputs a single element at a time.
The fuzzy selector can be edited from its dedicated section in the Selection Options window.
The matching accuracy of the fuzzy selector is based on the Levenshtein distance algorithm to count the similarity of selectors. The default value is 0.5 and it can be configured either from the Accuracy slider or directly from the input field by typing a valid value. The matching accuracy can be set within a range of 0.4 to 1 where a lower value indicates a higher degree of fuzziness, meaning the fuzzy selector outputs more elements, and a higher value indicates a lower degree of fuzziness, meaning the fuzzy selector outputs less elements.
Just like the Strict selector, the fuzzy selector field supports the use and creation of variables and arguments and it also supports wildcards. For more details, refer to the information provided above.
If the indicated element contains text, the text is displayed in the input box below the fuzzy selector edit window. If text is displayed in this input box, at runtime it must be found in the element itself to be considered a match.
The drop-down menu next to the input box has the following options:
- Text is/InnerText is/FullText is (depending on the text scraping method used) - Displays the text that is scraped; the best scraping method is automatically identified and it is recommended to use it.
- Ignore text - Ignores the text specified in the input box.
When the target or anchor search using a fuzzy selector fails, a custom exception is thrown displaying the closest matches to repair the failing selector.
The UI element could not be located using the configured Fuzzy selector for Target or Anchor.
'Target' search failed for selector:
<ctrl name='Use*' role='radio button' /><ctrl name='Use mimi' role='XYZ text' automationid='' />
These are the closest selector matches found:
[84%] <ctrl name='Use Amount' role='radio button' /><ctrl role='text' automationid='' />
[83%] <ctrl name='Use Both' role='radio button' /><ctrl role='text' automationid='' />
[82%] <ctrl name='Use Piece Count' role='radio button' /><ctrl role='text' automationid='' />
'Anchor 1' search failed for selector:
<ctrl name='Configuration' role='text' /><ctrl name='Configuration' role='text' automationid='' check:text='XYZ Config' />
These are the closest selector matches found:
[78%] <ctrl name='Configuration' role='text' /><ctrl name='Configuration' role='text' automationid='' check:text='Configuration' />
[64%] <ctrl name='Transaction #:' role='text' /><ctrl name='Transaction #:' role='text' automationid='' check:text='Transaction #:' />
[63%] <ctrl name='Deposit transaction' role='text' /><ctrl name='Deposit transaction' role='text' automationid='' check:text='Deposit transaction' />
'Anchor 2' search failed for selector:
<ctrl name='Use Both' role='radio button' automationid='' check:fulltext='Use XYZZZZZZ' />
These are the closest selector matches found:
[83%] <ctrl name='Use Both' role='radio button' automationid='' check:fulltext='Use Both
' />
[74%] <ctrl name='Use Amount' role='radio button' automationid='' check:fulltext='Use Amount
' />
[68%] <ctrl name='Use Piece Count' role='radio button' automationid='' check:fulltext='Use Piece Count
' />
The UI element could not be located using the configured Fuzzy selector for Target or Anchor.
'Target' search failed for selector:
<ctrl name='Use*' role='radio button' /><ctrl name='Use mimi' role='XYZ text' automationid='' />
These are the closest selector matches found:
[84%] <ctrl name='Use Amount' role='radio button' /><ctrl role='text' automationid='' />
[83%] <ctrl name='Use Both' role='radio button' /><ctrl role='text' automationid='' />
[82%] <ctrl name='Use Piece Count' role='radio button' /><ctrl role='text' automationid='' />
'Anchor 1' search failed for selector:
<ctrl name='Configuration' role='text' /><ctrl name='Configuration' role='text' automationid='' check:text='XYZ Config' />
These are the closest selector matches found:
[78%] <ctrl name='Configuration' role='text' /><ctrl name='Configuration' role='text' automationid='' check:text='Configuration' />
[64%] <ctrl name='Transaction #:' role='text' /><ctrl name='Transaction #:' role='text' automationid='' check:text='Transaction #:' />
[63%] <ctrl name='Deposit transaction' role='text' /><ctrl name='Deposit transaction' role='text' automationid='' check:text='Deposit transaction' />
'Anchor 2' search failed for selector:
<ctrl name='Use Both' role='radio button' automationid='' check:fulltext='Use XYZZZZZZ' />
These are the closest selector matches found:
[83%] <ctrl name='Use Both' role='radio button' automationid='' check:fulltext='Use Both
' />
[74%] <ctrl name='Use Amount' role='radio button' automationid='' check:fulltext='Use Amount
' />
[68%] <ctrl name='Use Piece Count' role='radio button' automationid='' check:fulltext='Use Piece Count
' />
The closest matches are not displayed if you alter the default selector by adding or removing selector nodes or if the target or anchor search failed because of multiple matches.
You can find more information about selectors in the Studio guide.
Computer Vision is widely available: whether it is on Studio Desktop for Windows and cross-platform projects or on Studio Web.
To use Computer Vision as a targeting method, you need to go to Project Settings > Computer Vision and make sure to set the correct server endpoint. You can leave this field as it is and use UiPath Cloud, which is filled-in by default, or you can enter another value either by selecting any of the other public endpoints specific to a server region or your own on-premises server.
Another way of using Computer Vision as a targeting method is to set Use Local Server to True. This implies that the UiPath.ComputerVision.LocalServer package is installed in your project. When Local Server is enabled, the server value is over-written. However, the Local Server is not available in cross-platform projects.
This targeting method uses an image in the application window that identifies the element, for example an icon or a button.
If Image automation is used for your target, you can also view the screenshot of the element, as well as adjust the image accuracy. The default value is 0.8 and it can be configured either from the Accuracy slider or directly from the input field by typing a valid value. The accuracy can be set within a range of 0.4 to 1. For further information, check the Image and Text Automation section.
If you want to define Image-only targets, you can use the Image region selection (F3) or draw a box selection of the area you want to use as your target/anchor while indicating.
You can also use the refresh button to manually refresh the image that you have selected, in case the image of the element changed since it was indicated.
The Image targeting method is disabled by default in Project Settings.
To enable it at project level, go to Project Settings > UI Automation Modern > Targeting methods - Desktop applications / Targeting methods - Web browsers / Targeting methods - Java / Targeting methods - SAP (depending on what type of application you want to automate) > Image > select True.
To enable the method on a specified element, the element must be valid.
If you selected the Enable Native text check box when indicating the element, the Native Text targeting method is used. This method enables you to use all the text labels that are identified with the Native method for identifying the element you want to interact with.
The text label identified through this method can be edited from its dedicated section, and you can specify whether you want the text to be case sensitive or not by using the Toggle case sensitive button.
Selecting the Edit offline option opens the selection screen in offline editing mode. In offline editing mode, you cannot add anchors, validate the descriptor, or other functionality that requires a live application.
If you select the Do not show again checkbox, the next time you want to edit the target, it opens the selection screen directly.
The validation process enables you to check how effective the current selection is in identifying the target by checking it using all selected methods and anchors. You can validate your selection by using the Validate button in the Selection Screen. The result of the validation is displayed at the top of the Selection Options window. If adjustments are needed, a message is displayed with information on how to improve the selection.
If you have a target and anchor pair, each of them uses all the available methods for targeting. To find the final target, all possible combinations for each of these elements are matched to elements on the screen and a score is given after the match.
After performing the validation, if there are any duplicates found, they are highlighted in the target application. If no duplicates are found, icons appear alongside the targeting methods, informing you of the state of each targeting method in the given situation.
The performance of each method is displayed in the Configuration section of the Selection Options window using one of the following icons:
- The targeting method that was fastest to successfully identify the element.
- The method successfully identified the element, but it was not the fastest.
- The method found duplicates and the correct element was identified by using this method alongside other targeting methods.
- The method found duplicates and the correct element could not be identified by using this method.
- The method failed to identify any elements.