Reading mongodb docs into a DocumentStore

[predoctech]

Tracking previous discussions on this forum it is surprising to see very few discussions about accessing documents on Mongodb as a datasource. On surface this seems doable utilizing pymongo as the connector, but I hope someone can provide some sample codes to see how this can be done. Say I have FAQ pairs stored in Mongodb and wish to read them into a Haystack Documentstore, and start my FAQPipeline from there. Some pointers to what a typical script in that regard would look like will be much appreciated.

[sjrl]

Hi @predoctech at this moment we do not have any plans to support Mongodb as a data source. However, what you proposed is something that could be accomplished by writing a custom node (custom node explanation here: https://haystack.deepset.ai/pipeline_nodes/custom-nodes) in a style similar to the Crawler node (https://haystack.deepset.ai/docs/latest/apicrawlermd) which crawls text from websites and converts that into Haystack documents which can then be used within Haystack document stores.

So the idea would be to write a custom node that reads stuff from Mongodb, does any kind of postprocessing, and creates Haystack documents that could be sent to a document store node.

Please let me know if this helps you get started :)

[predoctech]

Thanks @sjrl for coming back.
What I am wondering is if it can be done outside of the Pipeline infrastructure.
In order to access Mongodb we can utilize pymongo and make use of a MongoClient.
Mongo document can be queried as a dict, and impute into Haystack document object using the existing from_dict() function. I just realized that there is a built-in mapping function to match column names so that should come in handy.
Is that a plausible approach you think?

[sjrl]

Yes, I believe that would be a viable approach.

If you wanted you could also use that same functionality when creating a custom node giving you the option of putting it into the Pipeline infrastructure. A strength of having it in the Pipeline infrastructure would be to allow you to convert a large amount of Mongo queries into Haystack documents using an Indexing pipeline. Or even combine multiple input sources such as a Mongodb, and files by adding multiple-input nodes to your indexing pipeline. You can find more info about indexing pipelines here: https://haystack.deepset.ai/docs/latest/pipelinesmd