[FEATURE] Understandable logical documentation

[adityamohan93]

Is your feature request related to a problem?
I found this repo opensearch-spark looking for a way to write to our OpenSearch service domain using Spark on EMR directly instead of making calls from a REST endpoint from a service or OpenSearch Ingestion pipeline. However the README is confusing to say the least. It introduces something called flint but never explains what it is and by saying OpenSearch Flint is ... It consists of four modules:.

What solution would you like?

1. Provide documentation in README on whether and how this repo can be used to write to OpenSearch using Spark and add more details around doing so with EMR and Glue.
2. Explain what Flint is.
3. What is the entire purpose of this repo? The repo name is opensearch-spark but the description is Spark Accelerator framework ; It enables secondary indices to remote data stores. If it only meant for that or can it help other use cases?

What alternatives have you considered?
Discard the Spark path for my write use case.

Do you have any additional context?
None.

[YANG-DB]

Hi @adityamohan93
Thanks for your comment - we are currently working on such an example use case - adding a specific docker-based framework that will show such API/Queries usage examples - please review and let me know what you think and if you have any suggestions:

• WIP: Docker integ test with async API #1003
• update spark-docker example with iceberg tables #1001
• Created docker files for an integ test cluster (#601) #986