mixcoord cannot find querycoord

[jubingc]

The following error logs are from a mixcoord pod. This milvus cluster has been running for a while without any issues until a recent deployment for configs/user.yaml. What could be the problem and how I should fix this issue?

etcd pods are working fine. I tried to scale in the mixcoord deployment to 0, wait for a few minutes, scale out to 1. However, the issue still exists. Please help.

milvus-mixcoor.log

{"level":"WARN","time":"2024/12/12 22:39:48.676 +00:00","caller":"grpcclient/client.go:474","message":"grpc client is nil, maybe fail to get client in the retry state","cl
ient_role":"querycoord","error":"empty grpc client: find no available querycoord, check querycoord state","errorVerbose":"empty grpc client: find no available querycoord,
check querycoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspa
ce/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.co
m/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/intern
al/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*Clie
ntBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[..
[]\n  | \t/workspace/source/internal/distributed/querycoord/client/client.go:100\n  | github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetri
cs\n  | \t/workspace/source/internal/distributed/querycoord/client/client.go:272\n  | github.com/milvus-io/milvus/internal/rootcoord.getQueryCoordMetrics\n  | \t/workspace
/source/internal/rootcoord/util.go:239\n  | github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func1\n  | \t/workspace/source/internal/rootcoord/
quota_center.go:393\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n  | runtime.goexit\n  | \t/usr/l
ocal/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available querycoord, check querycoord state\nError types: (1) *withstack.withStack
(2) *errutil.withPrefix (3) *errors.errorString"}
{"level":"WARN","time":"2024/12/12 22:39:48.678 +00:00","caller":"grpcclient/client.go:248","message":"failed to get client address","error":"find no available querycoord,
 check querycoord state"}
{"level":"WARN","time":"2024/12/12 22:39:48.678 +00:00","caller":"grpcclient/client.go:460","message":"fail to get grpc client in the retry state","client_role":"querycoor
d","error":"find no available querycoord, check querycoord state"}

[jubingc]

Is it because "Not authorized to perform sts:AssumeRoleWithWebIdentity", so that the query coordinator cannot start properly?

[jubingc]

The trust relationships that the IAM role uses somehow got changed. So the mix coord pod cannot assume the role correctly. Once the IAM trust relationship policy is fixed, mixcoord pods become normal.