02 Elasticsearch FAQ

NTOPNG - Elasticsearch - Kibana FAQ

---

Q: Do I need Logstash to insert ntopng data in Elasticsearch?

A: No, ntopng is able to directly and properly feed data to Elasticsearch Bulk API

Example:
ntopng -F 'es;ntopng;ntopng-%Y.%m.%d;http://elasticsearch:9200/_bulk;'

Format:
         es;<idx type>;<idx name>;<es URL>;<http:auth>
Usage:
         es;ntopng;ntopng-%Y.%m.%d;http://localhost:9200/_bulk;
         Note: the <idx name> accepts the strftime() format.

---

Q: How often will ntopng push data to Elasticsearch?

A: Once started, ntopng will push ES flows that are expired or periodically send (every 5 mins) partial flows for long lasting flows. The @timestamp field will be derived by the host time settings.

---

Q: I need to use Authentication and/or HTTPS to reach my ES Cluster

A: ntopng natively supports both - use the optional parameters, ie:

ntopng -F 'es;ntopng;ntopng-%Y.%m.%d;https://elasticsearch:80/_bulk;http_user:password;'

---

Q: How will ntopng data be indexed?

A: ntopng will populate index and type fields based on its configuration:

Example:
         es;ntopng;ntopng-%Y.%m.%d;http://localhost:9200/_bulk;

{
  "_index": "ntopng-2015.09.26",
  "_type": "ntopng",
  "_id": "ykXCN6sqQCueiyEH-mSv-w",
  "_score": 1,
  "_source": {
    "IPV4_SRC_ADDR": "127.0.0.1",
    "L4_SRC_PORT": 60091,
    "IPV4_DST_ADDR": "127.0.0.1",
    "L4_DST_PORT": 3000,
    "PROTOCOL": 6,
    "L7_PROTO": 7,
    "L7_PROTO_NAME": "HTTP",
    "TCP_FLAGS": 27,
    "IN_PKTS": 5,
    "IN_BYTES": 908,
    "OUT_PKTS": 5,
    "OUT_BYTES": 415,
    "FIRST_SWITCHED": 1443299288,
    "LAST_SWITCHED": 1443299288,
    "CLIENT_NW_LATENCY_MS": 0.003,
    "SERVER_NW_LATENCY_MS": 0.002,
    "HTTP_HOST": "localhost",
    "HTTP_URL": "\/js\/jquery.js",
    "HTTP_METHOD": "GET",
    "HTTP_RET_CODE": 304,
    "@timestamp": "2015-09-26T20:28:08.0Z",
    "@version": 1,
    "type": "ntopng"
  }
}

Note: The content of each flow will vary depending on the protocol, sources, etc.

---

Q: How can I have ES/Kibana recognize correct data types from ntopng?

A: Create a Template with the correct mapping for ntopng fields, ie:

curl -XPUT localhost:9200/_template/ntopng_template -d '
{
  "template" : "ntopng-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
       "_all" : {"enabled" : true, "omit_norms" : true},
       "dynamic_templates" : [ {
             "string_fields" : {
               "match" : "*",
               "match_mapping_type" : "string",
               "mapping" : {
                 "type" : "string", "index" : "analyzed", "omit_norms" : true,
                   "fields" : {
                     "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
                   }
               }
             }
       }, {
             "geo_fields" : {
               "match" : "*_IP_LOCATION",
               "mapping": {
                      "type": "geo_point"
                }
             }
       }, {
             "ip_fields" : {
               "match" : "IPV4_*",
               "match_mapping_type" : "string",
               "mapping": {
                      "type": "ip"
                }
             }
       } ],
       "properties" : {
         "@version": { "type": "string", "index": "not_analyzed" }
       }
    }
  }
}
'

The above mapping should produce the following fields:

---

Q: IP Location coordinates are not recognized as "geo_point"

A: Make sure your template provides the correct mapping for ntopng location fields, ie:

curl -XPUT localhost:9200/_template/ntopng_template -d '
{
    "template" : "ntopng-*",
    "mappings" : {
        "ntopng" : {
                "dynamic_templates": [
                { "ntopng_geopoint": { 
                    "match": "*_IP_LOCATION",
                    "mapping": {
                      "type": "geo_point"
                    }
                }}
             ]
        }
    }
}
'

ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

---