You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the catalog connection has no timeout, if the connection blocked for a long time, the whole Gravino sever will hang as the lock is unable to be released.
Let's take an example:
Then create a scheme and create a fileset under the schema:
Log in Gravitino server side:
2024-10-30 14:09:42.031 INFO [Gravitino-webserver-60] [org.apache.gravitino.server.web.rest.SchemaOperations.createSchema(SchemaOperations.java:109)] - Received create schema request: s3_metalake.oss_catalog.schema
2024-10-30 14:09:42.208 INFO [Gravitino-webserver-60] [org.apache.gravitino.connector.BaseCatalog.initAuthorizationPluginInstance(BaseCatalog.java:198)] - Authorization provider is not set!
2024-10-30 14:09:42.385 WARN [Gravitino-webserver-60] [org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:60)] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-10-30 14:09:42.472 INFO [Gravitino-webserver-60] [org.apache.gravitino.server.web.rest.SchemaOperations.lambda$createSchema$3(SchemaOperations.java:125)] - Schema created: s3_metalake.oss_catalog.schema
2024-10-30 14:09:42.518 INFO [Gravitino-webserver-46] [org.apache.gravitino.server.web.rest.SchemaOperations.listSchemas(SchemaOperations.java:81)] - Received list schema request for catalog: s3_metalake.oss_catalog
2024-10-30 14:09:42.542 INFO [Gravitino-webserver-46] [org.apache.gravitino.server.web.rest.SchemaOperations.lambda$listSchemas$1(SchemaOperations.java:93)] - List 2 schemas in catalog s3_metalake.oss_catalog
2024-10-30 14:09:53.307 INFO [Gravitino-webserver-48] [org.apache.gravitino.server.web.rest.SchemaOperations.loadSchema(SchemaOperations.java:144)] - Received load schema request for schema: s3_metalake.oss_catalog.oss_schema1
2024-10-30 14:09:53.315 INFO [Gravitino-webserver-47] [org.apache.gravitino.server.web.rest.FilesetOperations.listFilesets(FilesetOperations.java:85)] - Received list filesets request for schema: s3_metalake.oss_catalog.oss_schema1
2024-10-30 14:09:53.332 INFO [Gravitino-webserver-48] [org.apache.gravitino.server.web.rest.SchemaOperations.lambda$loadSchema$4(SchemaOperations.java:152)] - Schema loaded: s3_metalake.oss_catalog.oss_schema1
2024-10-30 14:09:53.343 INFO [Gravitino-webserver-47] [org.apache.gravitino.server.web.rest.FilesetOperations.lambda$listFilesets$1(FilesetOperations.java:96)] - List 1 filesets under schema: s3_metalake.oss_catalog.oss_schema1
2024-10-30 14:09:53.633 INFO [Gravitino-webserver-65] [org.apache.gravitino.server.web.rest.FilesetOperations.listFilesets(FilesetOperations.java:85)] - Received list filesets request for schema: s3_metalake.oss_catalog.oss_schema1
2024-10-30 14:09:53.646 INFO [Gravitino-webserver-65] [org.apache.gravitino.server.web.rest.FilesetOperations.lambda$listFilesets$1(FilesetOperations.java:96)] - List 1 filesets under schema: s3_metalake.oss_catalog.oss_schema1
2024-10-30 14:09:56.391 INFO [Gravitino-webserver-49] [org.apache.gravitino.server.web.rest.FilesetOperations.listFilesets(FilesetOperations.java:85)] - Received list filesets request for schema: s3_metalake.oss_catalog.schema
2024-10-30 14:09:56.398 INFO [Gravitino-webserver-45] [org.apache.gravitino.server.web.rest.SchemaOperations.loadSchema(SchemaOperations.java:144)] - Received load schema request for schema: s3_metalake.oss_catalog.schema
2024-10-30 14:09:56.406 INFO [Gravitino-webserver-49] [org.apache.gravitino.server.web.rest.FilesetOperations.lambda$listFilesets$1(FilesetOperations.java:96)] - List 0 filesets under schema: s3_metalake.oss_catalog.schema
2024-10-30 14:09:56.409 INFO [Gravitino-webserver-45] [org.apache.gravitino.server.web.rest.SchemaOperations.lambda$loadSchema$4(SchemaOperations.java:152)] - Schema loaded: s3_metalake.oss_catalog.schema
2024-10-30 14:09:56.447 INFO [Gravitino-webserver-53] [org.apache.gravitino.server.web.rest.FilesetOperations.listFilesets(FilesetOperations.java:85)] - Received list filesets request for schema: s3_metalake.oss_catalog.schema
2024-10-30 14:09:56.451 INFO [Gravitino-webserver-53] [org.apache.gravitino.server.web.rest.FilesetOperations.lambda$listFilesets$1(FilesetOperations.java:96)] - List 0 filesets under schema: s3_metalake.oss_catalog.schema
2024-10-30 14:10:24.897 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:114)] - Start to check the dead lock...
2024-10-30 14:10:24.897 INFO [tree-lock-cleaner-0] [org.apache.gravitino.lock.LockManager.lambda$startNodeCleaner$3(LockManager.java:159)] - Total tree lock node count: 5
2024-10-30 14:10:24.898 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:116)] - Finish to check the dead lock...
2024-10-30 14:10:37.216 INFO [Gravitino-webserver-51] [org.apache.gravitino.server.web.rest.FilesetOperations.createFileset(FilesetOperations.java:119)] - Received create fileset request: s3_metalake.oss_catalog.schema.wrong_fileset
2024-10-30 14:10:37.392 INFO [Gravitino-webserver-51] [com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.<clinit>(GoogleHadoopFileSystemBase.java:611)] - GHFS version: 1.9.4-hadoop3
2024-10-30 14:10:37.797 WARN [Gravitino-webserver-51] [com.aliyun.oss.common.utils.LogUtils.logException(LogUtils.java:62)] - [Client]Unable to execute HTTP request: UnknownHost
2024-10-30 14:10:38.110 INFO [Gravitino-webserver-65] [org.apache.gravitino.server.web.rest.FilesetOperations.createFileset(FilesetOperations.java:119)] - Received create fileset request: s3_metalake.oss_catalog.schema.wrong_fileset
2024-10-30 14:10:38.402 WARN [Gravitino-webserver-51] [com.aliyun.oss.common.utils.LogUtils.logException(LogUtils.java:62)] - [Client]Unable to execute HTTP request: UnknownHost
2024-10-30 14:10:39.612 WARN [Gravitino-webserver-51] [com.aliyun.oss.common.utils.LogUtils.logException(LogUtils.java:62)] - [Client]Unable to execute HTTP request: UnknownHost
2024-10-30 14:10:42.015 WARN [Gravitino-webserver-51] [com.aliyun.oss.common.utils.LogUtils.logException(LogUtils.java:62)] - [Client]Unable to execute HTTP request: UnknownHost
2024-10-30 14:10:46.819 WARN [Gravitino-webserver-51] [com.aliyun.oss.common.utils.LogUtils.logException(LogUtils.java:62)] - [Client]Unable to execute HTTP request: UnknownHost
2024-10-30 14:10:53.505 INFO [Gravitino-webserver-61] [org.apache.gravitino.server.web.rest.FilesetOperations.createFileset(FilesetOperations.java:119)] - Received create fileset request: s3_metalake.oss_catalog.schema.wrong_fileset
2024-10-30 14:10:56.428 WARN [Gravitino-webserver-51] [com.aliyun.oss.common.utils.LogUtils.logException(LogUtils.java:62)] - [Client]Unable to execute HTTP request: UnknownHost
2024-10-30 14:11:11.689 INFO [Gravitino-webserver-62] [org.apache.gravitino.server.web.rest.FilesetOperations.createFileset(FilesetOperations.java:119)] - Received create fileset request: s3_metalake.oss_catalog.schema.wrong_fileset
2024-10-30 14:11:15.637 WARN [Gravitino-webserver-51] [com.aliyun.oss.common.utils.LogUtils.logException(LogUtils.java:62)] - [Client]Unable to execute HTTP request: UnknownHost
2024-10-30 14:11:24.897 INFO [tree-lock-cleaner-0] [org.apache.gravitino.lock.LockManager.lambda$startNodeCleaner$3(LockManager.java:159)] - Total tree lock node count: 5
2024-10-30 14:11:24.898 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:114)] - Start to check the dead lock...
2024-10-30 14:11:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=schema,hashCode=-907987520}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637231}
2024-10-30 14:11:24.900 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:11:24.901 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:11:24.901 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:11:24.901 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:11:24.901 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:11:24.902 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:11:24.902 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:11:24.902 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:11:24.903 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:11:24.903 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:116)] - Finish to check the dead lock...
2024-10-30 14:11:54.047 WARN [Gravitino-webserver-51] [com.aliyun.oss.common.utils.LogUtils.logException(LogUtils.java:62)] - [Client]Unable to execute HTTP request: UnknownHost
2024-10-30 14:12:24.894 INFO [tree-lock-cleaner-0] [org.apache.gravitino.lock.LockManager.lambda$startNodeCleaner$3(LockManager.java:159)] - Total tree lock node count: 5
2024-10-30 14:12:24.894 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:114)] - Start to check the dead lock...
2024-10-30 14:12:24.895 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=schema,hashCode=-907987520}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637231}
2024-10-30 14:12:24.895 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:12:24.896 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:12:24.896 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:12:24.896 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:12:24.896 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:12:24.896 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:12:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:12:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:12:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:12:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:12:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:12:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:12:24.898 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:116)] - Finish to check the dead lock...
2024-10-30 14:13:10.858 WARN [Gravitino-webserver-51] [com.aliyun.oss.common.utils.LogUtils.logException(LogUtils.java:62)] - [Client]Unable to execute HTTP request: UnknownHost
2024-10-30 14:13:24.702 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:73)] - Thread 20 start to collect garbage...
2024-10-30 14:13:24.702 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:76)] - Start to collect and delete legacy data by thread 20
2024-10-30 14:13:24.702 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:80)] - Try to physically delete METALAKE legacy data that has been marked deleted before 1729664004702
2024-10-30 14:13:24.723 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:80)] - Try to physically delete CATALOG legacy data that has been marked deleted before 1729664004702
2024-10-30 14:13:24.725 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:80)] - Try to physically delete SCHEMA legacy data that has been marked deleted before 1729664004702
2024-10-30 14:13:24.731 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:80)] - Try to physically delete TABLE legacy data that has been marked deleted before 1729664004702
2024-10-30 14:13:24.736 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:80)] - Try to physically delete COLUMN legacy data that has been marked deleted before 1729664004702
2024-10-30 14:13:24.737 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:80)] - Try to physically delete FILESET legacy data that has been marked deleted before 1729664004702
2024-10-30 14:13:24.741 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:80)] - Try to physically delete TOPIC legacy data that has been marked deleted before 1729664004702
2024-10-30 14:13:24.744 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:80)] - Try to physically delete USER legacy data that has been marked deleted before 1729664004702
2024-10-30 14:13:24.749 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:80)] - Try to physically delete GROUP legacy data that has been marked deleted before 1729664004702
2024-10-30 14:13:24.752 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:80)] - Try to physically delete ROLE legacy data that has been marked deleted before 1729664004702
2024-10-30 14:13:24.755 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:80)] - Try to physically delete TAG legacy data that has been marked deleted before 1729664004702
2024-10-30 14:13:24.758 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:80)] - Try to physically delete AUDIT legacy data that has been marked deleted before 1729664004702
2024-10-30 14:13:24.758 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:93)] - Start to collect and delete old version data by thread 20
2024-10-30 14:13:24.758 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:96)] - Try to softly delete METALAKE old version data that has been over retention count 1
2024-10-30 14:13:24.758 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:96)] - Try to softly delete CATALOG old version data that has been over retention count 1
2024-10-30 14:13:24.758 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:96)] - Try to softly delete SCHEMA old version data that has been over retention count 1
2024-10-30 14:13:24.758 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:96)] - Try to softly delete TABLE old version data that has been over retention count 1
2024-10-30 14:13:24.758 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:96)] - Try to softly delete COLUMN old version data that has been over retention count 1
2024-10-30 14:13:24.758 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:96)] - Try to softly delete FILESET old version data that has been over retention count 1
2024-10-30 14:13:24.761 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:96)] - Try to softly delete TOPIC old version data that has been over retention count 1
2024-10-30 14:13:24.761 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:96)] - Try to softly delete USER old version data that has been over retention count 1
2024-10-30 14:13:24.761 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:96)] - Try to softly delete GROUP old version data that has been over retention count 1
2024-10-30 14:13:24.761 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:96)] - Try to softly delete ROLE old version data that has been over retention count 1
2024-10-30 14:13:24.761 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:96)] - Try to softly delete TAG old version data that has been over retention count 1
2024-10-30 14:13:24.761 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:96)] - Try to softly delete AUDIT old version data that has been over retention count 1
2024-10-30 14:13:24.761 INFO [RelationalBackend-Garbage-Collector] [org.apache.gravitino.storage.relational.RelationalGarbageCollector.collectAndClean(RelationalGarbageCollector.java:111)] - Thread 20 finish to collect garbage.
2024-10-30 14:13:24.897 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:114)] - Start to check the dead lock...
2024-10-30 14:13:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=schema,hashCode=-907987520}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637231}
2024-10-30 14:13:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:13:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:13:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:13:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:13:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:13:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:13:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:13:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:13:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:13:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:13:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:13:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:13:24.898 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:116)] - Finish to check the dead lock...
2024-10-30 14:13:24.899 INFO [tree-lock-cleaner-0] [org.apache.gravitino.lock.LockManager.lambda$startNodeCleaner$3(LockManager.java:159)] - Total tree lock node count: 5
2024-10-30 14:14:24.895 INFO [tree-lock-cleaner-0] [org.apache.gravitino.lock.LockManager.lambda$startNodeCleaner$3(LockManager.java:159)] - Total tree lock node count: 5
2024-10-30 14:14:24.895 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:114)] - Start to check the dead lock...
2024-10-30 14:14:24.896 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=schema,hashCode=-907987520}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637231}
2024-10-30 14:14:24.896 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:14:24.896 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:14:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:14:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:14:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:14:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:14:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:14:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:14:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:14:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:14:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:14:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:14:24.898 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:116)] - Finish to check the dead lock...
2024-10-30 14:15:24.897 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:114)] - Start to check the dead lock...
2024-10-30 14:15:24.900 INFO [tree-lock-cleaner-0] [org.apache.gravitino.lock.LockManager.lambda$startNodeCleaner$3(LockManager.java:159)] - Total tree lock node count: 5
2024-10-30 14:15:24.900 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=schema,hashCode=-907987520}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637231}
2024-10-30 14:15:24.900 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:15:24.901 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:15:24.901 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:15:24.901 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:15:24.901 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:15:24.902 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:15:24.902 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:15:24.902 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:15:24.903 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:15:24.903 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:15:24.903 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:15:24.903 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:15:24.904 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:116)] - Finish to check the dead lock...
2024-10-30 14:15:44.475 WARN [Gravitino-webserver-51] [com.aliyun.oss.common.utils.LogUtils.logException(LogUtils.java:62)] - [Client]Unable to execute HTTP request: UnknownHost
2024-10-30 14:16:24.897 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:114)] - Start to check the dead lock...
2024-10-30 14:16:24.897 INFO [tree-lock-cleaner-0] [org.apache.gravitino.lock.LockManager.lambda$startNodeCleaner$3(LockManager.java:159)] - Total tree lock node count: 5
2024-10-30 14:16:24.897 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=schema,hashCode=-907987520}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637231}
2024-10-30 14:16:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:16:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:16:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:16:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:16:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:16:24.898 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:16:24.899 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:16:24.899 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:16:24.899 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:16:24.899 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:16:24.899 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:16:24.899 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:16:24.899 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:116)] - Finish to check the dead lock...
2024-10-30 14:17:24.901 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:114)] - Start to check the dead lock...
2024-10-30 14:17:24.900 INFO [tree-lock-cleaner-0] [org.apache.gravitino.lock.LockManager.lambda$startNodeCleaner$3(LockManager.java:159)] - Total tree lock node count: 5
2024-10-30 14:17:24.903 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=schema,hashCode=-907987520}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637231}
2024-10-30 14:17:24.903 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:17:24.903 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:17:24.903 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:17:24.904 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:17:24.904 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:17:24.905 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:17:24.905 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:17:24.905 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:17:24.905 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:17:24.905 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:17:24.905 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:17:24.905 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:17:24.905 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:116)] - Finish to check the dead lock...
2024-10-30 14:18:24.902 INFO [tree-lock-cleaner-0] [org.apache.gravitino.lock.LockManager.lambda$startNodeCleaner$3(LockManager.java:159)] - Total tree lock node count: 5
2024-10-30 14:18:24.902 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:114)] - Start to check the dead lock...
2024-10-30 14:18:24.903 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=schema,hashCode=-907987520}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637231}
2024-10-30 14:18:24.904 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:18:24.904 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:18:24.904 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:18:24.904 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=oss_catalog,hashCode=-1802225912}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:18:24.904 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:18:24.904 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:18:24.904 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:18:24.904 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=s3_metalake,hashCode=2012488818}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:18:24.904 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:18:24.905 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:18:24.905 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:18:24.905 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=s3_metalake.oss_catalog.schema}=1730268637230, ThreadIdentifier{thread=Thread[Gravitino-webserver-62,5,main], ident=s3_metalake.oss_catalog.schema}=1730268671689, ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=s3_metalake.oss_catalog.schema}=1730268638110, ThreadIdentifier{thread=Thread[Gravitino-webserver-61,5,main], ident=s3_metalake.oss_catalog.schema}=1730268653506}
2024-10-30 14:18:24.905 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:116)] - Finish to check the dead lock...
I clicked the create fileset button several times and it seems that there is a deadlock here.
How should we improve?
No response
The text was updated successfully, but these errors were encountered:
What would you like to be improved?
Currently, the catalog connection has no timeout, if the connection blocked for a long time, the whole Gravino sever will hang as the lock is unable to be released.
Let's take an example:
Then create a scheme and create a fileset under the schema:
Log in Gravitino server side:
I clicked the create fileset button several times and it seems that there is a deadlock here.
How should we improve?
No response
The text was updated successfully, but these errors were encountered: