communications-mining
latest
false
Communications Mining Developer Guide
Last updated Nov 7, 2024

Comments

Each message in Communications Mining is represented by a single comment object in the API. As a result, they can be considered equivalent. The developer documentation and API will refer primarily to comments, whilst the user guide and the Communications Mining UI refers primarily to messages.
When uploading data to Communications Mining or fetching data from Communications Mining, it is important to understand how different types of data (such as emails or support tickets) should be represented as comments. This page explains how to model your data as Communications Mining comments to prepare it for upload, and how to understand data fetched from Communications Mining.
Example of a comment created from an email

Communications Mining comment created from a review

The Overview section describes the overall structure of a comment object. If you want to upload data to Communications Mining via the API, or to understand how to process data uploaded to Communications Mining via the API, check the Comments created via the API section. You can find detailed descriptions for each of the commonly used types of comments (emails or support tickets). If you want to better understand how to process data uploaded to Communications Mining via an integration, check the Comments created by integrations section. Finally, for a full list of available comment object fields, check the Reference section.

Overview

Communications Mining works with various types of text data such as emails, survey responses, support tickets, or customer reviews. What these types of data have in common is that they all consist of units of communication (an email, a survey response, a support ticket, a customer review). In Communications Mining, a single message is represented as a comment, for example.

No matter what kind of communication unit a comment symbolizes, it consistently maintains this fundamental structure:

{
  "id": <UNIQUE ID>,
  "timestamp": <TIMESTAMP>,
  "messages": [
    {
      "body": { "text": <TEXT> },
      ...
    }
  ],
  "user_properties": { ... },
}{
  "id": <UNIQUE ID>,
  "timestamp": <TIMESTAMP>,
  "messages": [
    {
      "body": { "text": <TEXT> },
      ...
    }
  ],
  "user_properties": { ... },
}

As shown in the code snippet above, in addition to the actual piece of text, a comment always has an ID and a timestamp. The ID needs to be unique within the source containing the message. The timestamp is used in the platform UI to filter and sort by date, and to generate date-based analytics.

In addition to these required fields, other fields should be set depending on the type of the comment. If your data has been uploaded to Communications Mining via an integration, Communications Mining automatically populates all necessary fields. Check the following sections for a more detailed description.

Comments created via the API

Emails

While the easiest way to sync emails into Communications Mining is via the Exchange integration, in cases where you do your own email extraction, you can sync emails via the API. Use the sync-raw-emails endpoint for raw emails, and the sync endpoint for processed emails.

When syncing raw emails, provide the extracted MIME email headers and email body as-is (check the Reference for a description of the raw email format). Communications Mining parses the headers and cleans the email body.

Note: The raw email example below shows a very small number of headers for brevity. Send all the extracted headers to Communications Mining, which are likely to be much longer than in the example.
Important:

How does Communications Mining process raw emails?

  • Sets the email-specific fields in the message object messages[0]
  • Sets the thread_id field and thread_properties object
  • Cleans up the email body by stripping quoted emails and putting the signature into a separate signature field
  • Populates the user_properties object with metadata extracted from email headers.
If a field is not present in the email, it won't be set in the comment at all (rather than being set to a null or empty value). For example, the comment in the following example does not contain a BCC: field.

If you enrich emails with other data prior to uploading to Communications Mining, you can provide this additional data in the user properties of the comment.

The processed raw email looks like the processed email example below - see the number of additional fields created by Communications Mining. If you want to upload processed emails, structure them as in the processed email example.
Example email

Raw Email
{
  "raw_email": {
    "body": {
      "plain": "Hi Bob,\n\nCould you send me the figures for today?\n\nThanks,\nAlice"
    },
    "headers": {
      "raw": "From: Alice Smith <alice@example.com>\nDate: Tue, 3 Aug 2021 10:57:42 +0100\nMessage-ID: <e7784b5b@mail.example.com>\nSubject: Figures for today\nTo: Bob <bob@company.com>\nCc: Joe <joe@company.com>"
    }
  },
  "user_properties": {
    "string:Team": "Team XYZ"
  }
}{
  "raw_email": {
    "body": {
      "plain": "Hi Bob,\n\nCould you send me the figures for today?\n\nThanks,\nAlice"
    },
    "headers": {
      "raw": "From: Alice Smith <alice@example.com>\nDate: Tue, 3 Aug 2021 10:57:42 +0100\nMessage-ID: <e7784b5b@mail.example.com>\nSubject: Figures for today\nTo: Bob <bob@company.com>\nCc: Joe <joe@company.com>"
    }
  },
  "user_properties": {
    "string:Team": "Team XYZ"
  }
}
Processed Email
{
  "comment": {
    "id": "3c6537373834623562406d61696c2e6578616d706c652e636f6d3e",
    "timestamp": "2021-08-03T09:57:42Z",
    "user_properties": {
      "string:Has Signature": "Yes",
      "string:Sender": "alice@example.com",
      "string:Thread": "<e7784b5b@mail.example.com>",
      "string:Message ID": "<e7784b5b@mail.example.com>",
      "number:Recipient Count": 2,
      "number:Participant Count": 3,
      "number:Position in Thread": 1,
      "string:Sender Domain": "example.com",
      "string:Team": "Team XYZ"
    },
    "messages": [
      {
        "body": {
          "text": "Hi Bob,\n\nCould you send me the figures for today?"
        },
        "signature": {
          "text": "Thanks,\nAlice"
        },
        "subject": {
          "text": "Figures for today"
        },
        "to": ["\"Bob\" <bob@company.com>"],
        "cc": ["\"Joe\" <joe@company.com>"],
        "sent_at": "2021-08-03T09:57:42Z",
        "from": "\"Alice Smith\" <alice@example.com>"
      }
    ],
    "thread_id": "3c6537373834623562406d61696c2e6578616d706c652e636f6d3e"
  },
  "thread_properties": {
    "duration": null,
    "response_time": null,
    "num_messages": 1,
    "num_participants": 3,
    "first_sender": "alice@example.com",
    "thread_position": 0
  }
}{
  "comment": {
    "id": "3c6537373834623562406d61696c2e6578616d706c652e636f6d3e",
    "timestamp": "2021-08-03T09:57:42Z",
    "user_properties": {
      "string:Has Signature": "Yes",
      "string:Sender": "alice@example.com",
      "string:Thread": "<e7784b5b@mail.example.com>",
      "string:Message ID": "<e7784b5b@mail.example.com>",
      "number:Recipient Count": 2,
      "number:Participant Count": 3,
      "number:Position in Thread": 1,
      "string:Sender Domain": "example.com",
      "string:Team": "Team XYZ"
    },
    "messages": [
      {
        "body": {
          "text": "Hi Bob,\n\nCould you send me the figures for today?"
        },
        "signature": {
          "text": "Thanks,\nAlice"
        },
        "subject": {
          "text": "Figures for today"
        },
        "to": ["\"Bob\" <bob@company.com>"],
        "cc": ["\"Joe\" <joe@company.com>"],
        "sent_at": "2021-08-03T09:57:42Z",
        "from": "\"Alice Smith\" <alice@example.com>"
      }
    ],
    "thread_id": "3c6537373834623562406d61696c2e6578616d706c652e636f6d3e"
  },
  "thread_properties": {
    "duration": null,
    "response_time": null,
    "num_messages": 1,
    "num_participants": 3,
    "first_sender": "alice@example.com",
    "thread_position": 0
  }
}

Thread Properties

The following thread properties are available.

NAMEDESCRIPTION
thread_positionPosition of comment in thread, calculated by ordering the comment by timestamp. Starts at 0.
num_messagesNumber of comments in thread.
num_participantsTotal number of unique participants (From, To, CC, BCC) in thread.
first_senderSender of the first comment in thread.
durationDifference (in seconds) between the timestamps of first and last comment in thread. Will be set to null if num_messages

is 1 (i.e. thread contains only 1 comment).

Note: The timestamp of a comment corresponds to the sent_at field of the corresponding raw email.
response_timeDifference (in seconds) between the first comment in thread and the first response in thread. The first response in thread is the oldest comment where sender is not first_sender. Will be set to null if there are no responses in thread (i.e. if all emails in thread are from the same sender).

Each time a new comment is added to the platform, the thread properties of the corresponding thread are updated.

Note: Apart from thread_position, all properties are the same for each comment in thread.

Support tickets

In addition to the main text, a typical support ticket submitted via a form may have a subject, information about the sender (such as name or email address), and additional structured data (such as the topic of the ticket) which can be uploaded as part of the user properties of the comment.

The example below shows how to format a support ticket as a Communications Mining comment and how that comment is displayed in the platform's UI. Your user properties may be different depending on the data you collect.
Example support ticket

{
  "id": "dbcb03ad",
  "timestamp": "2020-02-26T16:09:00Z",
  "messages": [
    {
      "body": {
        "text": "Hi Support Team\n\nPlease could you look into my broadband service network status. I don't have any signal."
      },
      "subject": {
        "text": "Network Outage for over 24 hours - Customer account number 1234567"
      },
      "from": "alice.smith@example.com"
    }
  ],
  "user_properties": {
    "string:Customer Name": "Alice Smith",
    "string:Source": "Support Form",
    "string:Topic": "Broadband"
  }
}{
  "id": "dbcb03ad",
  "timestamp": "2020-02-26T16:09:00Z",
  "messages": [
    {
      "body": {
        "text": "Hi Support Team\n\nPlease could you look into my broadband service network status. I don't have any signal."
      },
      "subject": {
        "text": "Network Outage for over 24 hours - Customer account number 1234567"
      },
      "from": "alice.smith@example.com"
    }
  ],
  "user_properties": {
    "string:Customer Name": "Alice Smith",
    "string:Source": "Support Form",
    "string:Topic": "Broadband"
  }
}

Comments created by integrations

Emails (Microsoft Exchange)

Microsoft Exchange emails ingested into Communications Mining via the Exchange integration are automatically converted into comment objects in the same way as raw emails.

Attachments and attachment contents

Comments may have files attached to them. If a comment has attachments, the attachments field contains metadata about them:
```json
{
    "id": "3c484531505230324d423",
    "attachments": [
        {
            "name": "account-statement.pdf",
            "size": 49078,
            "content_type": "application/pdf",
        }
    ],
    // other comment fields omitted
    ...
},
``````json
{
    "id": "3c484531505230324d423",
    "attachments": [
        {
            "name": "account-statement.pdf",
            "size": 49078,
            "content_type": "application/pdf",
        }
    ],
    // other comment fields omitted
    ...
},
```
In addition, you can also download the attachment's content. Downloading the attachment's content returns the attachment_reference field:
```json
{
    "id": "3c484531505230324d423",
    "attachments": [
        {
            "name": "account-statement.pdf",
            "size": 49078,
            "content_type": "application/pdf",
            "attachment_reference": "CjQSEIExTHEqtdntoxz2WtbZDNEiIIVqcP1Sfx2L4epyRQDasa1RSODvheQ3bvLhj3L-_81G"
        }
    ],
    // other comment fields omitted
    ...
},
``````json
{
    "id": "3c484531505230324d423",
    "attachments": [
        {
            "name": "account-statement.pdf",
            "size": 49078,
            "content_type": "application/pdf",
            "attachment_reference": "CjQSEIExTHEqtdntoxz2WtbZDNEiIIVqcP1Sfx2L4epyRQDasa1RSODvheQ3bvLhj3L-_81G"
        }
    ],
    // other comment fields omitted
    ...
},
```
Use attachment_reference to retrieve the binary file content from [the attachments API](#FIXME). For the example above, you fetch the following URL: https://cloud.uipath.com/<organisation>/<tenant>/reinfer_/api/v1/attachments/CjQSEIExTHEqtdntoxz2WtbZDNEiIIVqcP1Sfx2L4epyRQDasa1RSODvheQ3bvLhj3L-_81G.

Check the [API Reference](#FIXME) for further details about this type of request.

If the attachment object doesn't have an attachment_reference property, you can't download the attachment's content. This may be because:
  • Communications Mining didn't receive the attachment's content.
  • The attachment content exceeded the size limit for uploading to Communications Mining.
  • Communications Mining processed the attachment before it supported file contents.

Learn more about the Attachment contents on the Attachment page.

Reference

Comments

See the table below for a list of available comment fields. If you are unfamiliar with Communications Mining comment objects, check the Overview.

NAMETYPEREQUIREDDESCRIPTION
idstringyesIdentifies a comment uniquely within a source. Any hexadecimal string of up to 1024 characters is valid (conforms to /[0-9a-f]{1,1024}/).
timestampstringyesA ISO-8601 timestamp indicating when the comment was created. If the timestamp does not specify a timezone, UTC will be assumed. The timestamp must be in the range 1950-01-01T00:00:00Z to 2049-12-31T23:59:59Z inclusive.
messagesarray<Message>yesAn array of zero or one message.
user_propertiesmap<string, string | number>noAny user-defined metadata that applies to the comment. There are two possible types: string and number. The key of a user property has the format "type:name", eg. "string:Domain Name" or "number:Star Rating". The user property name may consist of letters, numbers, spaces, and underscores, and may contain up to 32 characters (conforms to /\w([\w ]{0,30}\w)?/). The value must be a string or a number depending on the type of the user property.
thread_idstringnoAn ID uniquely identifying an email thread. Any hexadecimal string of up to 1024 characters is valid (conforms to /[0-9a-f]{1,1024}/).
uidstringset by Communications MiningA combined source and comment ID in the form of source_id.comment_id. You should not be setting this field directly as it's automatically generated by Communications Mining for uploaded comments.
created_atstringset by Communications MiningA ISO-8601 timestamp with the same constraints as the timestamp field. You should not be setting this field directly as it's automatically generated by Communications Mining when the comment is created.
updated_atstringset by Communications MiningA ISO-8601 timestamp with the same constraints as the timestamp field. You should not be setting this field directly as it's automatically generated by Communications Mining when the comment is updated.
attachmentsarray<Attachment>noAn array of zero or more attachments. An attachment represents a file attached to a comment.
NAMETYPEREQUIREDDESCRIPTION
namestringyesThe attachment's file name.
sizenumberyesThe size of the attachment's file content in bytes.
content_typestringyesThe [Media type](https://en.wikipedia.org/wiki/Media_type) of the attachment. For a list of possible values, see the [IANA Media Types](https://www.iana.org/assignments/media-types/media-types.xhtml) list.
attachment_referencestringnoUsed to retrieve the binary file content from [the attachments API](#FIXME)

Where Message has the following format:

NAMETYPEREQUIREDDESCRIPTION
bodyContentyesAn object containing the main body text of the message.
subjectContentnoAn object containing the message's subject.
signatureContentnoAn object containing the message's signature.
fromstringnoThe message sender.
toarray<string>noAn array of primary recipients.
ccarray<string>noAn array of carbon-copy recipients.
bccarray<string>noAn array of blind carbon-copy recipients.
sent_atstringnoA ISO-8601 timestamp indicating when the message was created. If the timestamp does not specify a timezone, UTC will be assumed.
languagestringnoThe original language of the message. If this is supplied, both text and translated_from should be supplied for the Content fields.

Where Content has the following format:

NAMETYPEREQUIREDDESCRIPTION
textstringyesIf language (other than the source's language) has been supplied, this should be the translated text of the content. Otherwise, it should be in the original language it was collected; it will be translated if not in the source's language and the source has should_translate set to true. Maximim 65536 characters.
translated_fromstringnoIf language (other than the source's language) has been supplied, this should by the original text of the content. Supplying this field without having supplied a language will result in an error. At most 65536 characters.

Raw Emails

See the table below for a list of available raw email fields.

NAMETYPEREQUIREDDESCRIPTION
headersHeadersyesAn object containing the headers of the email.
bodyBodyyesAn object containing the main body of the email.

Where Headers has the following format:

NAMETYPEREQUIREDDESCRIPTION
rawstringnoOne of raw and parsed is required. The raw email headers, given as a single string, with each header on its own line.
parsedmap<string, string | array<string>>no
One of raw and parsed is required. The parsed email headers, given as an object with string keys and string or array<string> values.

Each key must be ASCII, and represents one email header. Value strings may be any valid UTF-8.

Lists of values will be concatenated with , before being set as a single header value. If you require duplicate header keys, please use raw instead.

Where Body has the following format:

NAMETYPEREQUIREDDESCRIPTION
plainstringnoAt least one of plain and html is required. The plaintext content of the email. At most 65536 characters.
htmlstringnoAt least one of plain and html is required. The HTML content of the email.

Was this page helpful?

Get The Help You Need
Learning RPA - Automation Courses
UiPath Community Forum
Uipath Logo White
Trust and Security
© 2005-2024 UiPath. All rights reserved.