Overview

This article offers guidelines for the communications data volumes required to optimize the training experience and maximize the value provided by analytics and automation.

To get the most out of your Communications Mining™. implementation, we recommend to start with high-volume use cases. These cases benefit from Communications Mining's ability to process large amounts of message data efficiently, both for historical analytics and live monitoring, as well as automations.

The effort required to deploy a use case does not increase significantly with higher message volumes. Therefore, high-volume use cases tend to offer a better return on investment in terms of implementation effort compared to lower-volume use cases. This is important for organizations with limited resources or those that require external support for implementation.

However, if you have lower-volume scenarios with high business value, you should also consider these use cases. Many low-volume use cases are technically feasible and should not be dismissed.

Many use cases have a level of complexity—in terms of the number and complexity of labels and fields to be extracted—that is not well-suited for very low volumes of messages. This is because there may be insufficient examples in the dataset of varied and complex concepts or fields to effectively fine-tune and validate Communications Mining specialized models. This applies to both the automated training provided by generative annotation, and further examples annotated by model trainers.

While some use cases may be technically feasible and have sufficient examples, lower volumes can sometimes result in a poorer annotation experience for model trainers. A larger data pool makes it easier for Communications Mining's active learning modes to identify and surface useful examples to annotate. A small pool of data can create fewer quality examples across the taxonomy. Fewer quality examples cause users to rely on annotating elusive or more complex examples.

Before you proceed with qualifying and implementing a use case based on the considerations based on complexity and ROI, it's important to consider the technical limits for Communications Mining.

For generating clusters, Communications Mining requires a minimum of 2048 messages in a dataset (which can be made up of multiple similar sources). Datasets smaller than 2048 messages allow you to use all Comms Mining features, besides clusters and generated label suggestions for clusters.

Use cases with less than 2048 messages should be very simple in terms of the number and complexity of labels/fields. It should also be expected that a much higher proportion of total messages will need to be annotated for fine-tuning and validation purposes compared to higher volume use cases. It is likely that there may be insufficient examples to annotate for some labels and/or fields if they are not frequently occurring.

To ensure meaningful validation data, Communications Mining also expects a minimum of 25 annotated examples per label and field. Therefore, it’s important that you are able to source at least this number of examples from the data available.