Introduction

AI Computer Vision is a machine-learning based method used to visually identify all the UI elements on a computer screen and interact with them via UiPath Robots, simulating human interaction. It doesn't require or use the underlying properties of applications, but only the aspect and relationship of various screen elements.

Rather than relying on selectors, AI Computer Vision uses AI (Object Detection, OCR, fuzzy text-matching, image-matching for icons) and an anchoring system to tie it all together. More exactly, to visually locate elements on the screen, AI Computer Vision performs an element detection (on the machine-learning server) and a text (OCR) detection, and combines these two into a full understanding of the UI. The relationship between elements detected with these two methods is then encoded into a multi-anchor descriptor, which uniquely identifies the targeted element.

AI Computer Vision is composed of a set of activities, that are part of the UI Automation activity package, as well as a server (which can be cloud, on-premises, or local) hosting an AI model, which is needed to perform the actual analysis of the UI you're automating. By default, our UiPath cloud server is used and also recommended for all AI Computer Vision and UI Automation activities. You can use AI Computer Vision cloud regardless of your deployment type. For instance, it does not matter if you have Orchestrator on-premises or Orchestrator cloud, you can run Computer Vision cloud with no special configuration required.

Alternatively, you can host and manage your own on-premises AI Computer Vision server and use it to run the AI Computer Vision activities. When using this type of server, you need to have your own hardware infrastructure (GPUs) or cloud environment. Also, you need to deploy, update, and maintain your own environment locally. Compared to the UiPath cloud server, you might also run into issues with backwards compatibility when upgrading the AI model. For further details on how to avoid this kind of issues, go to Model update resilience.

Local server is another flavour you can opt for. It runs on local CPU and it is the most portable version. However, it is slower and has a slightly lower detection accuracy.