Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 2.x] Remove ignore decorator for testCohereClassifyModel #3330

Merged
merged 1 commit into from
Jan 9, 2025
Merged

[Backport 2.x] Remove ignore decorator for testCohereClassifyModel #3330

merged 1 commit into from
Jan 9, 2025

Conversation

opensearch-trigger-bot[bot]
Copy link
Contributor

Backport b5dd5a1 from #3324

@mingshl
Copy link
Collaborator

mingshl commented Jan 4, 2025

flaky bwc test

REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:mlCommonsBwcCluster#mixedClusterTask' --tests "org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT.testBackwardsCompatibility" -Dtests.seed=723B0C2625B468B -Dtests.security.manager=false -Dtests.locale=el-CY -Dtests.timezone=Pacific/Niue -Druntime.java=11

Suite: Test class org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT
  2> REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:mlCommonsBwcCluster#mixedClusterTask' --tests "org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT.testBackwardsCompatibility" -Dtests.seed=723B0C2625B468B -Dtests.security.manager=false -Dtests.locale=el-CY -Dtests.timezone=Pacific/Niue -Druntime.java=11
  2> org.junit.ComparisonFailure: expected:<[m_l_limit_exceeded]_exception> but was:<[null_pointer]_exception>
        at __randomizedtesting.SeedInfo.seed([723B0C2625B468B:EC1C8D0C18809301]:0)
        at org.junit.Assert.assertEquals(Assert.java:117)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT.testBackwardsCompatibility(MLCommonsBackwardsCompatibilityIT.java:219)
  2> NOTE: leaving temporary files on disk at: /home/runner/work/ml-commons/ml-commons/plugin/build/testrun/mlCommonsBwcCluster#mixedClusterTask/temp/org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT_723B0C2625B468B-001
  2> NOTE: test params are: codec=Asserting(Lucene912): {}, docValues:{}, maxPointsInLeafNode=469, maxMBSortInHeap=5.839382668696142, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=el-CY, timezone=Pacific/Niue
  2> NOTE: Linux 6.8.0-1017-azure amd64/Azul Systems, Inc. 11.0.25 (64-bit)/cpus=4,threads=1,free=415328768,total=536870912
  2> NOTE: All tests run in this JVM: [MLCommonsBackwardsCompatibilityIT]


1 test completed, 1 failed
Tests with failures:

 - org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT.testBackwardsCompatibility
=== Standard output of node `node{:opensearch-ml-plugin:mlCommonsBwcCluster0-0}` ===

»    ↓ errors and warnings from /home/runner/work/ml-commons/ml-commons/plugin/build/testclusters/mlCommonsBwcCluster0-0/logs/opensearch.stdout.log ↓
» WARN ][o.o.g.DanglingIndicesState] [mlCommonsBwcCluster0-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
»   ↑ repeated 2 times ↑
» WARN ][o.o.d.FileBasedSeedHostsProvider] [mlCommonsBwcCluster0-0] expected, but did not find, a dynamic hosts list at [/home/runner/work/ml-commons/ml-commons/plugin/build/testclusters/mlCommonsBwcCluster0-0/config/unicast_hosts.txt]
»   ↑ repeated 12 times ↑
» WARN ][o.o.c.c.ClusterFormationFailureHelper] [mlCommonsBwcCluster0-0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [mlCommonsBwcCluster0-0, mlCommonsBwcCluster0-1, mlCommonsBwcCluster0-2] to bootstrap a cluster: have discovered [{mlCommonsBwcCluster0-0}{S52StZoARW2szh3KhDdC0w}{sZ3baUsGQXaU3ruQXEN1Xw}{127.0.0.1}{127.0.0.1:45437}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]; discovery will continue using [] from hosts providers and [{mlCommonsBwcCluster0-0}{S52StZoARW2szh3KhDdC0w}{sZ3baUsGQXaU3ruQXEN1Xw}{127.0.0.1}{127.0.0.1:45437}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
» ERROR][o.o.m.c.MLCommonsClassLoader] [mlCommonsBwcCluster0-0] Failed to init instance for type PREDICTION
»  java.lang.reflect.InvocationTargetException: null
»  	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
»  	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
»  	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
»  	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
»  	at org.opensearch.ml.common.MLCommonsClassLoader.init(MLCommonsClassLoader.java:242) [opensearch-ml-common-2.19.0.0-SNAPSHOT.jar:?]
»  	at org.opensearch.ml.common.MLCommonsClassLoader.initMLInstance(MLCommonsClassLoader.java:206) [opensearch-ml-common-2.19.0.0-SNAPSHOT.jar:?]
»  	at org.opensearch.ml.common.output.MLOutput.fromStream(MLOutput.java:36) [opensearch-ml-common-2.19.0.0-SNAPSHOT.jar:?]
»  	at org.opensearch.ml.common.transport.MLTaskResponse.<init>(MLTaskResponse.java:39) [opensearch-ml-common-2.19.0.0-SNAPSHOT.jar:?]
»  	at org.opensearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:85) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:52) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1508) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1495) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.handleResponse(NativeMessageHandler.java:405) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:174) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:796) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:[78](https://github.com/opensearch-project/ml-commons/actions/runs/12604095837/job/35130368000?pr=3330#step:5:79)8) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.115.Final.jar:4.1.115.Final]
»  	at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
»  Caused by: java.io.EOFException
»  	at org.opensearch.core.common.io.stream.InputStreamStreamInput.readByte(InputStreamStreamInput.java:74) ~[opensearch-core-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.core.common.io.stream.FilterStreamInput.readByte(FilterStreamInput.java:55) ~[opensearch-core-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:586) ~[opensearch-core-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.ml.common.output.MLPredictionOutput.<init>(MLPredictionOutput.java:72) ~[?:?]
»  	... 42 more
» WARN ][r.suppressed             ] [mlCommonsBwcCluster0-0] path: /_plugins/_ml/_train_predict/kmeans, params: {algorithm=kmeans}
»  java.lang.NullPointerException: null
»  	at org.opensearch.ml.common.transport.MLTaskResponse.toXContent(MLTaskResponse.java:64) ~[?:?]
»  	at org.opensearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:62) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:57) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:49) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:52) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.rest.action.RestActionListener.onResponse(RestActionListener.java:64) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:70) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1517) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.doHandleResponse(NativeMessageHandler.java:427) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.handleResponse(NativeMessageHandler.java:419) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:174) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:[79](https://github.com/opensearch-project/ml-commons/actions/runs/12604095837/job/35130368000?pr=3330#step:5:80)6) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:2[80](https://github.com/opensearch-project/ml-commons/actions/runs/12604095837/job/35130368000?pr=3330#step:5:81)) [netty-handler-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.115.Final.jar:4.1.115.Final]
»  	at java.base/java.lang.Thread.run(Thread.java:[82](https://github.com/opensearch-project/ml-commons/actions/runs/12604095837/job/35130368000?pr=3330#step:5:83)9) [?:?]
» ERROR][o.o.m.a.MLModelAutoReDeployer] [mlCommonsBwcCluster0-0] Failed to query need undeploy models, no action will be performed
»  org.opensearch.index.IndexNotFoundException: no such index [.plugins-ml-model]

@dhrubo-os dhrubo-os merged commit 6726a3a into 2.x Jan 9, 2025
9 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mingshl mingshl mingshl approved these changes

@ylwu-amzn ylwu-amzn ylwu-amzn approved these changes

@b4sjoo b4sjoo Awaiting requested review from b4sjoo b4sjoo is a code owner

@dhrubo-os dhrubo-os Awaiting requested review from dhrubo-os dhrubo-os is a code owner

@jngz-es jngz-es Awaiting requested review from jngz-es jngz-es is a code owner

@model-collapse model-collapse Awaiting requested review from model-collapse model-collapse is a code owner

@rbhavna rbhavna Awaiting requested review from rbhavna rbhavna is a code owner

@zane-neo zane-neo Awaiting requested review from zane-neo zane-neo is a code owner

@Zhangxunmt Zhangxunmt Awaiting requested review from Zhangxunmt Zhangxunmt is a code owner

@austintlee austintlee Awaiting requested review from austintlee austintlee is a code owner

@HenryL27 HenryL27 Awaiting requested review from HenryL27 HenryL27 is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@mingshl @ylwu-amzn @dhrubo-os @b4sjoo