Data Volume

We make a clear distinction between the performance of the data used for an Application and the Connector. Although they make use of the same platform, there are some differences, i.e. what is acceptable for the users (developers versus end users) and the type of actions performed.

Large amounts of data can have both an impact on the Connector and Application, but all can be solved in the Connector.

The performance end-users will experience is directly related to the data volume. The data volume is determined by the number of rows in the biggest tables. In general, only the number of rows determine the performance end users experience. The number of columns is only a factor when the data is loaded from the database.

Processes with about 5.000.000 (5M) cases and up to about 50.000.000 (50M) events per process would be ideal. With more cases and events parsing the data and showing the visualization will take longer.

The UiPath Process Mining platform will continue to work, however, when large amounts of data are inserted, the reaction speed may drop. It is recommended to check the data amount beforehand. If it exceeds the above numbers, it is advised to consider optimizing or limiting the dataset.

A higher level of detail will take a higher response time which impacts the performance.

The exact tradeoff between the amount of data, the level of detail, and the waiting time needs to be discussed with the end users. Sometimes historical data can be very important, but often only the last few years are needed.

Limiting the number of records that will show up in the tour dataset will not only improve the performance of the application, it will also improve the comprehensibility of the process and in turn, improve acceptance by the business.

The scoping of the data can be done in the Connector.

One of the options for scoping is to limit the time frame to look by filtering out dates or periods. For example, you could limit the timeframe from 10 years to one year. Or from 1 year to one month. See the illustration below.

A limited amount of activities is advised, especially in the start of any process mining effort. From there you can build up as the expertise starts to ramp up.

Below is a guideline for the range of activities:

Below are some suggestions for filtering data:

If there is one case with a lot of events (outlier), it will impact some expressions which calculate aggregates on the event level. The from/to dashboard item filter is impacted by this and can be time-consuming to calculate if you have these outliers. It is recommended to filter out these cases in the Connector to take them out of the dataset.

In other instances, the outliers may be the key area to focus on. If your process is going well or you adopt Six Sigma methodologies, you want to focus on the things going wrong. Instead of showing all the cases going right, you only show the cases going wrong.

See the illustration below.

In the Connector, you can remove attributes that have a lot of detail. For example, long strings in the Event Detail attribute.

When finished developing a lot of unused attributes may end up in your dataset. It is recommended to only set the availability of the attributes that are used in the output dataset of the Connector to the public. Set the availability of other attributes to private.

Pre-aggregation is a technique that is employed by many BI tools to gain insights into large data volumes. It involves aggregating data over specific attributes to reduce the number of records in a dataset. In BI this would typically be summing the value of each supplier, so only have one record for each supplier.

See the illustration below.

Process mining requires more configuration, but a starting point is to only aggregate on process variants. For each variant you would have one case record and a related number of events. This can significantly reduce the data volumes.

To show correct results you would also have to show how many records each variant represents, for the event ends you could use a median duration of each event. Aggregating only using variants might be too high so it would be wise to check most common filters used, e.g. a combination of variants, case type and month of the case end (to show trends over time).

However, adding attributes has a quadratic effect on the number of records so this requires a careful balance between performance and use case.

Pre-aggregation is most applicable for an overview of your process and spotting general trends.

Sampling is a technique where you take a percentage of the cases and their events happening in a specific period. You can for instance set that only 10% of all cases and their events are shown. In this way you still have exceptions or outliers since each case has a similar chance of showing up in the dataset.

See illustration below.

Cascaded sampling is a technique where the sampling percentage drops over time with a certain percentage. An example of this shows 100% of last week’s data, 90% of two weeks ago, 80% of three weeks ago, and so on.

Data sharding is a technique of the data scoping solution, which allows organizations to split up the data into multiple datasets, rather than just slicing off one part. This setup does require additional configuration since the application needs to be split up by using modules and multiple smaller dataset need to be exported from the connector.

With data sharding, the original dataset is divided into multiple shards. The smaller each shard is, the faster it will be. When a user logs in to the application, only the applicable data shard will be loaded.

A typical unit for sharding would be “Company code” or “Department”. For example, in the case of 50 company codes, each shard will contain one company code, and essentially be about 50 times faster than the original dataset.

See the illustration below for an overview of sharding.