Skip to content

02 Elasticsearch FAQ

Jump to bottom
Lorenzo Mangani edited this page Sep 27, 2015 · 24 revisions

NTOP

NTOPNG - Elasticsearch - Kibana FAQ

Q: Do I need Logstash to insert ntopng data in Elasticsearch?
A: No, ntopng is able to directly and properly feed data to Elasticsearch Bulk API
Example:
ntopng -F 'es;ntopng;ntopng-%Y.%m.%d;http://elasticsearch:9200/_bulk;'

Format:
         es;<idx type>;<idx name>;<es URL>;<http:auth>
Usage:
         es;ntopng;ntopng-%Y.%m.%d;http://localhost:9200/_bulk;
         Note: the <idx name> accepts the strftime() format.
Q: How often will ntopng push data to Elasticsearch?
A: Once started, ntopng will push ES flows that are expired or periodically send (every 5 mins) partial flows for long lasting flows. The @timestamp field will be derived by the host time settings.
Q: I need to use Authentication and/or HTTPS to reach my ES Cluster
A: ntopng natively supports both - use the optional parameters, ie:
ntopng -F 'es;ntopng;ntopng-%Y.%m.%d;https://elasticsearch:80/_bulk;http_user:password;'
Q: How will ntopng data be indexed?
A: ntopng will populate index and type fields based on its configuration:
Example:
         es;ntopng;ntopng-%Y.%m.%d;http://localhost:9200/_bulk;

{
  "_index": "ntopng-2015.09.26",
  "_type": "ntopng",
  "_id": "ykXCN6sqQCueiyEH-mSv-w",
  "_score": 1,
  "_source": {
    "IPV4_SRC_ADDR": "127.0.0.1",
    "L4_SRC_PORT": 60091,
    "IPV4_DST_ADDR": "127.0.0.1",
    "L4_DST_PORT": 3000,
    "PROTOCOL": 6,
    "L7_PROTO": 7,
    "L7_PROTO_NAME": "HTTP",
    "TCP_FLAGS": 27,
    "IN_PKTS": 5,
    "IN_BYTES": 908,
    "OUT_PKTS": 5,
    "OUT_BYTES": 415,
    "FIRST_SWITCHED": 1443299288,
    "LAST_SWITCHED": 1443299288,
    "CLIENT_NW_LATENCY_MS": 0.003,
    "SERVER_NW_LATENCY_MS": 0.002,
    "HTTP_HOST": "localhost",
    "HTTP_URL": "\/js\/jquery.js",
    "HTTP_METHOD": "GET",
    "HTTP_RET_CODE": 304,
    "@timestamp": "2015-09-26T20:28:08.0Z",
    "@version": 1,
    "type": "ntopng"
  }
}

Note: The content of each flow will vary depending on the protocol, sources, etc.
Q: How can I have ES/Kibana recognize correct data types from ntopng?
A: Create a Template with the correct mapping for ntopng fields, ie:
curl -XPUT localhost:9200/_template/ntopng_template -d '
{
    "template" : "ntopng-*",
    "mappings" : {
        "ntopng" : {
                "dynamic_templates": [
		        { "ntopng_geopoint": { 
		            "match": "*_IP_LOCATION",
		            "mapping": {
		              "type": "geo_point"
		            }
		        }},
		        { "ntopng_ip": { 
		            "match": "IPV4_*",
		            "mapping": {
		              "type": "ip"
		            }
		        }},
		        { "ntopng_bytes": { 
		            "match": "*_BYTES",
		            "mapping": {
		              "type": "byte"
		            }
		        }}
     		 ]
        }
    }
}
'
Q: IP Location coordinates are not recognized as "geo_point"
A: Create a Template with the correct mapping for ntopng location fields, ie:
curl -XPUT localhost:9200/_template/ntopng_template -d '
{
    "template" : "ntopng-*",
    "mappings" : {
        "ntopng" : {
                "dynamic_templates": [
                { "ntopng_geopoint": { 
                    "match": "*_IP_LOCATION",
                    "mapping": {
                      "type": "geo_point"
                    }
                }}
             ]
        }
    }
}
'

ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

Clone this wiki locally