Training High Performing Models

To choose an OCR engine, you should create different Data Manager sessions, configure different OCR engines, and try to import the same files into each of them to examine differences. You should focus on those areas which you intend to extract. For example, if you need to extract company names that appear as part of logos on invoices you may want to see which OCR engine performs better on the text in logos.

Your default option should be UiPath Document OCR since it is included with Document Understanding licenses at no charge. However, in cases where some unsupported languages are required, or some very hard-to-read documents are involved, you might want to try Google Cloud (Cloud only) or Microsoft Read (Cloud or On Premises), which have better language coverage. These engines come at a cost, indeed it is low, but if the accuracy is higher on some critical data fields for your business process, it is strongly recommended to use the best OCR available – it will be well worth your time later on since everything downstream depends on it.

Please be aware that the Digitize Document activity has the ForceApplyOCR setting set to False by default, which means when operating on .pdf documents by default the OCR engine is not used at all, the text is extracted directly from the .pdf document itself in the case of native .pdf documents. However, many native .pdf documents may contain logos or even headers or footers which are not picked up. These may contain the identifying information of the company, such as its name, address, VAT code or payment information, which is highly relevant in business processes. If your process has this situation, then set ForceApplyOCR to True to ensure that all text is detected, though it might slow down your process.

Machine Learning technology has the main benefit of being able to handle complex problems with high diversity. When estimating the size of a training dataset, one looks first at the number of fields and their types and the number of languages. A single model can handle multiple languages as long as they are not Chinese/Japanese/Korean. CJK scenarios will generally require separate Training datasets and separate models.

There are 3 types of fields:

The above guidelines assume that you are solving a high diversity scenario like invoices or purchase orders with dozens to hundreds or thousands of layouts. However, if you are solving a low-diversity scenario like a tax form or invoices with very few layouts (under 5-10), then the dataset size is determined more by the number of layouts. In this case, you should start with 20-30 pages per layout and add more if needed - especially if the pages are very dense with large number of fields to be extracted. For instance, creating a model for extracting 10 fields from 2 layouts might require 60 pages, but if you need to extract 50 or 100 fields from 2 layouts, then you might start with 100 or 200 pages and add more as needed to get the desired accuracy. In this case, the distinction regular fields/column fields is less important.

In addition, these estimates assume that most pages contain all or most of the fields. In cases where you have documents with multiple pages but most fields are on a single page, then the relevant number of pages is the number of examples of that one page where most fields appear.

The numbers above are general guidelines, not strict requirements. In general, you can start with a smaller dataset, and then keep adding data until you get good accuracy. This is especially useful in order to parallelize the RPA work with the model building. Also, a first version of the model can be used to prelabel additional data (see Settings view and Predict button in Data Manager) which can accelerate labeling additional Training data.

Whereas when referring to Training sets the pages and the number of pages are the most important, in the case of Evaluation sets, we refer only to documents and the number of documents. The scores for releases v2021.10 and after are calculated per document.

Evaluation datasets can be smaller. They can be 50-100 documents (or even just 30-50 in low diversity scenarios), and they can grow over time to be a few hundred documents. It is important that they are representative of the production data flow. So, a good approach is to randomly select from the documents processed in the RPA workflow. Even if some vendors will be overrepresented, that is ok. For example, if a single vendor represents 20% of your invoice traffic, it is ok to have that vendor be 20% of your Evaluation set too, so the Evaluation metrics approximate your business metric, i.e. reduction in person-months spent on manual document processing.

When importing Evaluation data into Data Manager you need to make sure you check the “Make this a evaluation set” box on the Import dialog window. In this way, you guarantee that the data is held out when training, and also you can easily export it for running Evaluations using the evaluation-set option in the Filter dropdown in Data Manager.

Defining fields is a conversation that needs to happen with the Subject Matter Expert or Domain Expert who owns the business process itself. For invoices, it would be the Accounts Payable process owner. This conversation is critical, it needs to happen before any documents are labeled to avoid wasted time, and it requires looking together at a minimum of 20 randomly chosen document samples. A one-hour slot needs to be reserved for this, and it often needs to be repeated after a couple of days, as the person preparing the data runs into ambiguous situations or edge cases.

It is not uncommon that the conversation starts with the assumption that you need to extract, let's say, 10 fields, and later you end up with 15. Some examples are described in the subsections below.

Some key configurations you need to be aware of:

The most important configuration is the Content Type, with the exception of String:

These impact post-processing, especially the cleanup, the parsing, and the formatting. The most complex is the Date formatting, but also Number formatting requires determining the decimal point separator and thousands decorator. In some cases, if parsing fails your option is to report the issue to UiPath support and to fall back on the String content type, which does no parsing. In that case, you would need to parse the value in your RPA workflow logic.

Another relevant configuration is the Multi-line checkbox which is relevant mainly for String type fields. Whenever some other field produces unexpected results or no results, the first thing to try is to change it to String Multi-line field, to see the unaltered output of the model prediction.

When labeling Training data you need to focus on the bounding boxes of the words in the document pane of Data Manager. The parsed values in the right or top sidebars are not important as they are not used for training.

Whenever a field appears multiple times on a page, as long as they represent the same concept (see the Define fields section above), all of them should be labeled.

When the OCR misses a word or gets a few characters wrong, just label the bounding box if there is one, and if not, then just skip it and keep going. There is no possibility to add a word in Data Manager because even if you did, the word would still be missing at run time so adding it would not help the model at all.

As you label remain vigilant about fields that may have multiple or overlapping meanings/concepts, in case you might need to split a field into two separate fields, or fields that you potentially do not explicitly need, but which, if labeled, might help you to do certain validation or self-consistency check logic in the RPA workflow. Typical examples are quantity,unit-price, and line-amount on invoice line items. Line-amount is the product of quantity and unit-price, but this is very useful to check for consistency without the need for confidence levels.

When labeling Evaluation datasets (also called Test datasets) you need to focus on something slightly different from labeling Training datasets. Whereas for Training datasets only the bounding boxes of the fields on the document matter, for Evaluation datasets only the values of the fields matter. You may edit them by clicking on the value in the right or top sidebar and editing it. To return to the automatically parsed value click on the lock icon.

Exporting the entire dataset, including both Training and Test batches is allowed because the Training pipelines in AI Center ignore Test data. However, Evaluation pipelines will run the evaluation on the whole evaluation dataset regardless if it is comprised of train or test data. The type of a given document is displayed right under the file name, at the top center of the Data Manager window.

When evaluating a ML model the most powerful tool is the evaluation.xlsx file generated in the artifacts/eval_metrics folder. In this Excel file you can see what predictions are failing and on which files, and you can see immediately if it is an OCR error or a ML Extraction or parsing error, and if it may be fixed by simple logic in the RPA workflow, or it requires a different OCR engine, more training data, or improving the labelling or the field configurations in Data Manager.

This Excel file is also very useful to identify the most relevant business rules you need to apply in the RPA workflow in order to catch common mistakes for routing to Validation Station in Action Center for manual review. Business rules are by far the most reliable way to detect errors.

For those errors which cannot be caught by business rules, you may also use confidence levels. The Excel file also contains confidence levels for each predictions so you can use Excel features like sorting and filtering to determine what a good confidence threshold is for your business scenario.

Overall, the evaluation.xlsx Excel file is a key resource you need to focus on in order to get the best results from your AI automation.

In this step, you should be concerned with model errors and how to detect them. There are 2 main ways to detect errors:

The most effective and reliable way to detect errors is by defining business rules. Confidence levels can never be 100% perfect, there will always be a small but non-zero percentage of correct predictions with low confidence or wrong predictions with high confidence. In addition, and perhaps most importantly, a missing field has no confidence, so a confidence threshold can never catch errors whereby a field is not extracted at all. Consequently, confidence level thresholds should only be used as a fallback, a safety net, but never as the main way to detect business-critical errors.

Examples of business rules:

In particular, the confidence levels of column fields should almost never be used as an error detection mechanism, since column fields (e.g., line items on invoices or POs) can have dozens of values, so setting a minimum threshold over so many values can be especially unreliable, as one value is more than likely to have small confidence, so this would lead to most/all documents being routed to human validation, many times unnecessarily.

Business rules must be enforced as part of the RPA workflow, and it would be ideal if the business rule failure is passed to the human validator so as to direct their attention and expedite the process.

After the business rules have been defined, sometimes there might remain a small number of fields for which there are no business rules in place, or for which the business rules are unlikely to catch all errors. For this, you may need to use a confidence threshold as a last resort.

The main tool to set this threshold is the Evaluation pipeline feature in AI Center, and specifically, the Excel spreadsheet which is output by the Evaluation pipeline in the Outputs > artifacts > eval_metrics folder.

This spreadsheet contains a column for each field and a column for the confidence level of each prediction. You can add a column called min_confidence which takes the minimum of all the confidences over all fields which are important for your business process and are not already covered by business rules. For instance, you may not want to put a threshold on the line items confidences, but rather on vendor name, total amount, date, due date, invoice number, and other essential fields. By sorting the table based on the min_confidence column you may see where the errors start appearing and set a threshold above that level to ensure that only correctly extracted documents are sent straight through.

Validation Station data can help improve the model predictions, yet, in many cases, it turns out that most errors are not due to the model itself but to the OCR, labelling errors or inconsistencies, or to postprocessing issues (e.g., date or number formatting). So, the first key aspect is that Validation Station data should be used only after the other Data Extraction Components have been verified and optimized to ensure good accuracy, and the only remaining area of improvement is the model prediction itself.

The second key aspect is that Validation Station data has lower information density than the data labeled in Data Manager. Fundamentally, the Validation Station user only cares about getting the right value once. If an invoice has 5 pages, and the invoice number appears on every page, the Validation Station user validates it only on the first page. So, 80% of the values remain unlabeled. In Data Manager, all the values are labeled.

Finally, keep in mind Validation Station data needs to be added to the original manually labelled dataset, so that you always have a single training dataset which increases in size over time. You will always need to train on the ML Package with the 0 (zero) minor version, which is the version released by UiPath Out of the Box.

Validation station data can potentially be of much higher volume since it is used in the production workflow. As a consequence, you need a guideline for how much data is likely to be useful, given that model Training requires time and infrastructure. Also, you do not want the dataset to become overwhelmed with Validation Station data because this can degrade the quality of the model due to the information density issue mentioned above.

The recommendation is to add a maximum of 2-3X the number of pages of Data Manager data and, beyond that, only cherry pick those vendors or samples where you see major failures. If there are known major changes to the production data, such as a new language, or a new geographic region being onboarded to the business process (expanding from US to Europe or South Asia), then representative data for those languages and regions should be added to Data Manager for manual labelling. Validation Station data is not appropriate for such major scope expansion.

Here is a sample scenario. You have chosen a good OCR engine, labeled 500 pages in Data Manager, resulting in good performance and you have deployed the model in a production RPA workflow. Validation Station is starting to generate data. You should randomly select up to a maximum of 1000-1500 pages from Validation Station and import them into the Data Manager together with the first 500 pages and retrain your ML model. After that, you should run an Evaluation Pipeline to make sure the model actually improved, and then you should deploy the new model to production.

The Auto-Fine-tuning loop is a Preview capability that becomes useful for maintaining a high-performing model which you have already created using the steps described above. To ensure that auto-fine-tuning produces better versions of the model it is critical that you have a good Evaluation dataset and that you use an automatically rescheduled Full Pipeline which runs both Training and Evaluation at the same time. In this way, you can see if the most recent Training produced a more accurate model than the previous one, and if so, you are ready to deploy the new model to the ML Skill invoked by the Robots in your business process.

AI Center offers the possibility to automatically update the ML Skill when a new version of an ML Package is retrained. However, this automatic update does not take into account the score of the Full Pipeline, and therefore it is not recommended to use this feature with Document Understanding auto-retraining pipelines.

As mentioned above in the Create an Evaluation Dataset section, Evaluation Pipeline implementations for ML Packages release 21.10 or later calculate scores on a per-document basis - which reflects accurately the results you see in an RPA workflow. This assumes your dataset was labelled on a per-document basis in Data Manager. You can tell if a multi-page document is labeled on a per-document basis if you can scroll naturally through the pages like in a regular PDF reader. If you need to click next to pass from one page to the next, then each page is considered a separate document.

Make sure to use the Document Understanding Process from the Templates section in the Studio start screen in order to apply best practices in Enterprise RPA architecture.