[Enhancement] Fetch system index mappings from json file instead of string constants

[pyek-bot]

Description

Moves the index mappings for system indices into json files and fetches the version from the file itself.
Version should be provided under "_meta"."schema_version"

All mappings were fetched by using the below method in MlIndex.java

public static void main(String[] args) {
        for (MLIndex index : values()) {
            System.out.println(index.getIndexName() + "\n"
                    + index.getMapping() + "\n");
        }
    }

Related Issues

Resolves #2951

Enhancements

• Add a required_fields object under _meta object to validate if these fields are present in the mapping
• Support placeholders to enable reusability

These enhancements will be raised in a separate PR: https://github.com/pyek-bot/ml-commons/pull/1/files

Testing

• Mappings were compared by fetching the existing mappings from main and comparing against new mappings using the above mentioned code block
• Added validation on mappings to ensure they follow a particular schema
• Added a test case that initializes the enum so that any failures in mapping creation can be caught before runtime
  https://github.com/pyek-bot/ml-commons/pull/1/files
• Added UTs for helper methods
• Ran integration tests to ensure all indices are created
• Manually created all system indices, fetched them via the GET api and validated the mappings

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dhrubo-os]

How can we check that all the json files are property formatted? Can we have corresponding validator test for all those files?

[dhrubo-os]

IndexUtilsTest > testGetMappingFromFilesMalformedJson FAILED
    java.lang.AssertionError at IndexUtilsTest.java:88
        Caused by: java.lang.IllegalArgumentException at IndexUtilsTest.java:88
        ```

[pyek-bot]

IndexUtilsTest > testGetMappingFromFilesMalformedJson FAILED
    java.lang.AssertionError at IndexUtilsTest.java:88
        Caused by: java.lang.IllegalArgumentException at IndexUtilsTest.java:88
        ```

This was due to the latest change with the error handling, i will fix it

[pyek-bot]

How can we check that all the json files are property formatted? Can we have corresponding validator test for all those files?

In this enhancements PR: https://github.com/pyek-bot/ml-commons/pull/1/files, I have added a test so that we can catch any malformed mappings or failures before runtime: common/src/test/java/MLIndexTest.java

It simply references the Enum object so that the mapping path for all indices is triggered. The mapping path has inbuilt verification for these json files.

I can add explicit checks for each file but as we add system indices that will require that each file have its own test. This will also help with keeping the test suite small and not have redundant checks.

Moreover, I have added schema validation to the files themselves, the enforced schema can be found here: common/src/main/resources/index-mappings/schema.json in the above PR.

Let me know what you think!

[dhrubo-os]

How can we check that all the json files are property formatted? Can we have corresponding validator test for all those files?

In this enhancements PR: https://github.com/pyek-bot/ml-commons/pull/1/files, I have added a test so that we can catch any malformed mappings or failures before runtime: common/src/test/java/MLIndexTest.java

It simply references the Enum object so that the mapping path for all indices is triggered. The mapping path has inbuilt verification for these json files.

I can add explicit checks for each file but as we add system indices that will require that each file have its own test. This will also help with keeping the test suite small and not have redundant checks.

Moreover, I have added schema validation to the files themselves, the enforced schema can be found here: common/src/main/resources/index-mappings/schema.json in the above PR.

Let me know what you think!

It's a trade off between generic check vs tailored check. We can follow hybrid approach like for most important indices (like model, connector, config), we can add more deliberate tests. And Overall for all the indices we can have a generic malformed json checks.

[pyek-bot]

How can we check that all the json files are property formatted? Can we have corresponding validator test for all those files?

In this enhancements PR: https://github.com/pyek-bot/ml-commons/pull/1/files, I have added a test so that we can catch any malformed mappings or failures before runtime: common/src/test/java/MLIndexTest.java
It simply references the Enum object so that the mapping path for all indices is triggered. The mapping path has inbuilt verification for these json files.
I can add explicit checks for each file but as we add system indices that will require that each file have its own test. This will also help with keeping the test suite small and not have redundant checks.
Moreover, I have added schema validation to the files themselves, the enforced schema can be found here: common/src/main/resources/index-mappings/schema.json in the above PR.
Let me know what you think!

It's a trade off between generic check vs tailored check. We can follow hybrid approach like for most important indices (like model, connector, config), we can add more deliberate tests. And Overall for all the indices we can have a generic malformed json checks.

Sure, I will add a couple tests in that PR (https://github.com/pyek-bot/ml-commons/pull/1/files) since it has most of the validation and verification logic!

[pyek-bot]

I have addressed your comments @dhrubo-os @Zhangxunmt , please check and let me know

@dhrubo-os for explicit tests, u can find it here: pyek-bot@d8390ca

once this PR is merged, I will raise another PR for the enhancements to this feature to make it complete

[dhrubo-os]

Have you done some sanity testing (end to end testing including agents setup) with this PR changes?

[pyek-bot]

Have you done some sanity testing (end to end testing including agents setup) with this PR changes?

Yup, I have done some functional testing for these components. However, the most critical test imo is that I have compared the new mappings and old mappings via a script after having extracted them as mentioned in the description.

I plan to run the same script for the enhancements PR as well.