[Backport 2.18] Adds preset contentRegistry for IngestProcessors (#3281) #3318
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…3281) * add preset xContentRegistry to ingestProcessors for custom parametized local models Curently local models that use the parameters map within the payload to create a request can not create objects to be used for local model prediction. This requires a opensearch core change because it needs the contentRegistry,however given there is not much dependency on the registry (currently) we can give it the preset registry given in the MachineLearningPlugin class vai the getNamedXContent() class Signed-off-by: Brian Flores <[email protected]> * Adds UT for proving models depend on xContentRegistry for prediction Signed-off-by: Brian Flores <[email protected]> * apply spotless Signed-off-by: Brian Flores <[email protected]> * Adds IT for Asymmetric Embedding scenario with MLInferenceIngestProcessor We needed to make sure that a IT existed so that the preset content registry on the processor could work with parametized local models. By providing an IT that uses the asymetric embedding model its proven that the content registry is needed to create the embeddings. In this specific test case I used a ingest pipeline to convert passage embeddings, by simulating the pipeline to save test time. Signed-off-by: Brian Flores <[email protected]> --------- Signed-off-by: Brian Flores <[email protected]> (cherry picked from commit df1b1ef)
brianf-aws
requested review from
b4sjoo,
dhrubo-os,
jngz-es,
model-collapse,
rbhavna,
ylwu-amzn,
zane-neo,
Zhangxunmt,
austintlee and
HenryL27
as code owners
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Error
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Error
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Failure
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Failure
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Failure
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Error
6 tasks
The test passes on JAVA 21 but for some reason Java 11 has trouble passing in a message so it returns null when getMessage occurs so when Mockito tries to invoke getMessage it will get get a second NPE. and thus is why the exception is caught. This code change is to get around the gap that Java 11 has. Signed-off-by: Brian Flores <[email protected]>
brianf-aws
temporarily deployed
to
ml-commons-cicd-env-require-approval
GitHub Actions
Inactive
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Error
brianf-aws
temporarily deployed
to
ml-commons-cicd-env-require-approval
GitHub Actions
Inactive
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Failure
brianf-aws
temporarily deployed
to
ml-commons-cicd-env-require-approval
GitHub Actions
Inactive
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Failure
Signed-off-by: Brian Flores <[email protected]>
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Error
brianf-aws
temporarily deployed
to
ml-commons-cicd-env-require-approval
GitHub Actions
Inactive
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Failure
brianf-aws
temporarily deployed
to
ml-commons-cicd-env-require-approval
GitHub Actions
Inactive
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Error
brianf-aws
temporarily deployed
to
ml-commons-cicd-env-require-approval
GitHub Actions
Inactive
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Error
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Error
brianf-aws
had a problem deploying
to
ml-commons-cicd-env-require-approval
GitHub Actions
Failure
brianf-aws
requested a deployment
to
ml-commons-cicd-env-require-approval
GitHub Actions
Waiting
brianf-aws
requested a deployment
to
ml-commons-cicd-env-require-approval
GitHub Actions
Waiting
brianf-aws
requested a deployment
to
ml-commons-cicd-env-require-approval
GitHub Actions
Waiting
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
2.18 : Backport PR for add preset xContentRegistry to ingestProcessors for custom parametized local models
See: #3316
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.