Sending data to Elasticsearch

Overview

In this topic you can learn how to leverage the Insights real-time data export feature to send data to ElasticSearch and use it to populate real-time monitoring data into Kibana dashboard.

Prerequisites

Before you can start viewing real-time monitoring data in Kibana, the following hosts must be set up:

Setting up an ElasticSearch Host

You can choose to host ElasticSearch yourself or to use the cloud option.

For more information, check the official Elastic website.

Setting up a FileBeat Host

Overview

FileBeat is a tool used to gather data from a source (EventHub in this case) and send it into ElasticSearch.

Installation

You need to host your own FileBeat server.

To install FileBeat, first provision your hosts and follow the instructions from the official Elastic website for your operating system.

Configuration

To configure FileBeat, use the following instructions:

Open the filebeat.yml file.

Set up a connection to ElasticSearch:

 cloud.id: "cloud:YOUR_CLOUD_ID"
cloud.auth: "USERNAME:YOUR_PASSWORD"cloud.id: "cloud:YOUR_CLOUD_ID"
cloud.auth: "USERNAME:YOUR_PASSWORD"

You can get your cloud ID from the Deployments tab in ElasticSearch.

Go to the filebeat.inputs section in the filebeat.yml file.

Add your EventHub connection settings in the filebeat.inputs section.

 <ul>
<li>type: azure-eventhub
  enabled: true
  eventhub: <YOUR EVENTHUB>
  consumer_group: "filebeat"
  connection_string: <YOUR CONNECTION STRING>
  storage_account_key: <YOUR STORAGE ACCOUNT KEY>
  storage_account_container: ""
  resource_manager_endpoint: """</li>
</ul><ul>
<li>type: azure-eventhub
  enabled: true
  eventhub: <YOUR EVENTHUB>
  consumer_group: "filebeat"
  connection_string: <YOUR CONNECTION STRING>
  storage_account_key: <YOUR STORAGE ACCOUNT KEY>
  storage_account_container: ""
  resource_manager_endpoint: """</li>
</ul>

Go to the processors section in the filebeat.yml file.

Add the following lines in the processors section:

 <ul>
<li>decode_json_fields:
      fields: ["message"]
      process_array: false
      max_depth: 2
      target: ""
      overwrite_keys: false
      add_error_key: true</li>
</ul><ul>
<li>decode_json_fields:
      fields: ["message"]
      process_array: false
      max_depth: 2
      target: ""
      overwrite_keys: false
      add_error_key: true</li>
</ul>

Starting up the FileBeat service

After you successfully installed and configured FileBeat, follow the procedure from the official Elastic website to start the service.

View data in Kibana

To view your data in Kibana, use the following steps.

Open ElasticSearch.
Expand the Analytics category from the dashboard on the left.
Select Discover.
Set the data view to filebeat-*.

You can now view your data in Kibana.

Insights

Was this page helpful?