MongoDB custom filter

The river settings support a new filter attribute which can be used as a "trigger" for the document that should be indexed.

Let"s say there is a document { "name": "richard", "needs_idexing": 0 } stored in MongoDB. This document should be only indexed by ES if needs_indexing is 1. The filter attribute could be used for this scenario. The filter value will be: {"o.needs_indexing": 1}. So the river setting will be:

{
 "type": "mongodb",
 "mongodb": {
  "servers": [
   {
    "host": "localhost",
    "port": "27017"
   }
  ],
  "db": "testriver",
  "collection": "documents",
  "gridfs": "false",
  "filter": "{\"o.score\" : 1}"
 },
 "index": {
  "name": "documentsindex"
 }
}

Filter has been only tested with MongoDB dot notation.

Filter value should be prefix by o. because of the structure of the document in oplog.rs collection. More details available here Replication Internals.

Current limitation: The indexed document will not be remove from ES index if the filter condition does not match anymore. In our example needs_indexing has been set to 1 so the document is indexed by ES. A new update in MongoDB set needs_indexing to 0 the document will not be remove from ES.