Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata #14372

Open
Abdulsamirkhan opened this issue Nov 4, 2024 · 5 comments
Open

Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata #14372

Abdulsamirkhan opened this issue Nov 4, 2024 · 5 comments
Labels
ingestion real-time troubleshooting

Comments

@Abdulsamirkhan
Copy link

Abdulsamirkhan commented Nov 4, 2024 •
edited
Loading

can you please help me to fix this error, i tried all possible combinations

I am able to able to read topic metadata, produce message and consume message from c# code.

Pinot docker image: [apachepinot/pinot:latest]

  Command 
  bin/pinot-admin.sh AddTable -tableConfigFile  /tmp/SamirTestTable1.json -schemaFile /tmp/SamirTestSchema.json -controllerHost localhost -controllerPort 9000 -exec

Kafka :
server.properties :
message.max.bytes=2200000000
listeners=PLAINTEXT://localhost:9092,PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
zookeeper.connect=localhost:2181

Consumer.properties
    bootstrap.servers=localhost:29092
    group.id=test-consumer-group
    message.max.bytes=2200000000
    max.request.size=1200000000
    metadata.max.age.ms=300000
    heartbeat.interval.ms=1000
    request.timeout.ms=160000
    max.partition.fetch.bytes =2200000000
    fetch.metadata.timeout.ms =600000 
    session.timeout.ms = 145000 
    fetch.max.bytes = 2200000000

Compose Yaml
kafka:
image: confluentinc/cp-kafka:5.5.0
depends_on:
- zookeeper
ports:
- "29092:29092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

Schema Json file
{
"schemaName": "SamirTest",
"dimensionFieldSpecs": [
{"name": "uuid", "dataType": "STRING"}
],
"metricFieldSpecs": [
{"name": "count", "dataType": "INT"}
],
"dateTimeFieldSpecs": [
{"name": "SamirTestTime", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS"}
]
}

Table Json file
{
"tableName": "SamirTest_REALTIME",
"tableType": "REALTIME",
"segmentsConfig": {
"timeColumnName": "SamirTestTime",
"schemaName": "SamirTest",
"replication": "1"
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant"
},
"metadata": {},
"tableIndexConfig": {
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.consumer.type": "simple",
"stream.kafka.topic.name": "samirtestNew",
"stream.kafka.broker.list": "localhost:29092",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest",
"stream.kafka.consumer.prop.fetch.metadata.timeout.ms": "120000",
"stream.kafka.consumer.prop.session.timeout.ms": "60000",
"stream.kafka.consumer.prop.request.timeout.ms": "120000",
"realtime.segment.flush.threshold.rows": "50000",
"realtime.segment.flush.threshold.time": "24h"
}
}
}

Code used to get metadata and worked

              var config1 = new ConsumerConfig
              {
                  BootstrapServers = "localhost:29092",
                  GroupId = "metadata-fetcher-group",
                  AutoOffsetReset = AutoOffsetReset.Earliest,
                  EnableAutoCommit = false
              };
              
              using (var adminClient = new AdminClientBuilder(config).Build())
              {
                  try
                  {
                      // Fetch metadata for all topics
                      var metadata = adminClient.GetMetadata(TimeSpan.FromSeconds(30));
              
              
                      foreach (var topic in metadata.Topics)
                      {
                          Console.WriteLine($"Topic: {topic.Topic}");
                          Console.WriteLine($"Partition Count: {topic.Partitions.Count}");
                          foreach (var partition in topic.Partitions)
                          {
                              Console.WriteLine($"Partition: {partition.PartitionId}, Leader: {partition.Leader}, Replicas: {string.Join(",", partition.Replicas)}, ISR: {string.Join(",", partition.InSyncReplicas)}");
                          }
                      }
                  }
                  catch (KafkaException e)
                  {
                      Console.WriteLine($"An error occurred: {e.Error.Reason}");
                  }
              }

Error details :
2024-11-04 17:39:19 2024/11/04 12:09:19.104 ERROR [PinotTableIdealStateBuilder] [grizzly-http-server-5] Could not get PartitionGroupMetadata for topic: samirtestNew of table: SamirTest_REALTIME
2024-11-04 17:39:19 org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:19 at org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.fetchPartitionCount(KafkaStreamMetadataProvider.java:72) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.stream.StreamMetadataProvider.computePartitionGroupMetadata(StreamMetadataProvider.java:76) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:70) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:31) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGroupMetadataList(PinotTableIdealStateBuilder.java:93) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.getNewPartitionGroupMetadataList(PinotLLCRealtimeSegmentManager.java:799) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.setUpNewTable(PinotLLCRealtimeSegmentManager.java:325) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.addTable(PinotHelixResourceManager.java:1746) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.api.resources.TableConfigsRestletResource.addConfig(TableConfigsRestletResource.java:233) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2024-11-04 17:39:19 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
2024-11-04 17:39:19 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2024-11-04 17:39:19 at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
2024-11-04 17:39:19 Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:19 2024/11/04 12:09:19.302 ERROR [PinotHelixResourceManager] [grizzly-http-server-5] Caught exception while setting up table: SamirTest_REALTIME, cleaning it up
2024-11-04 17:39:19 java.lang.RuntimeException: org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGroupMetadataList(PinotTableIdealStateBuilder.java:99) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.getNewPartitionGroupMetadataList(PinotLLCRealtimeSegmentManager.java:799) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.setUpNewTable(PinotLLCRealtimeSegmentManager.java:325) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.addTable(PinotHelixResourceManager.java:1746) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.api.resources.TableConfigsRestletResource.addConfig(TableConfigsRestletResource.java:233) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2024-11-04 17:39:19 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
2024-11-04 17:39:19 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2024-11-04 17:39:19 at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
2024-11-04 17:39:19 Caused by: org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:19 at org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.fetchPartitionCount(KafkaStreamMetadataProvider.java:72) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.stream.StreamMetadataProvider.computePartitionGroupMetadata(StreamMetadataProvider.java:76) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:70) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:31) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 at org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGroupMetadataList(PinotTableIdealStateBuilder.java:93) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:19 ... 30 more
2024-11-04 17:39:19 Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:20 2024/11/04 12:09:20.044 ERROR [TableConfigsRestletResource] [grizzly-http-server-5] org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:20 java.lang.RuntimeException: org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:20 at org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGroupMetadataList(PinotTableIdealStateBuilder.java:99) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.getNewPartitionGroupMetadataList(PinotLLCRealtimeSegmentManager.java:799) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.setUpNewTable(PinotLLCRealtimeSegmentManager.java:325) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.addTable(PinotHelixResourceManager.java:1746) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.controller.api.resources.TableConfigsRestletResource.addConfig(TableConfigsRestletResource.java:233) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2024-11-04 17:39:20 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
2024-11-04 17:39:20 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2024-11-04 17:39:20 at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) [pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
2024-11-04 17:39:20 Caused by: org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
2024-11-04 17:39:20 at org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.fetchPartitionCount(KafkaStreamMetadataProvider.java:72) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.spi.stream.StreamMetadataProvider.computePartitionGroupMetadata(StreamMetadataProvider.java:76) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:70) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:31) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 at org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGroupMetadataList(PinotTableIdealStateBuilder.java:93) ~[pinot-all-1.3.0-SNAPSHOT-jar-with-dependencies.jar:1.3.0-SNAPSHOT-3057f1ad7b256c0fedbdf293e8758d29f25a22a1]
2024-11-04 17:39:20 ... 30 more
2024-11-04 17:39:20 Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
@Abdulsamirkhan
Copy link
Author

@Jackie-Jiang, any help much appreciated

@Jackie-Jiang
Copy link
Contributor

cc @swaminathanmanish @KKcorps to help take a look

@Abdulsamirkhan
Copy link
Author

@Jackie-Jiang @swaminathanmanish @KKcorps, Still i am not able to solve this any help much appreciated.

@Abdulsamirkhan
Copy link
Author

@Jackie-Jiang @swaminathanmanish @KKcorps, gentle reminder

@Jackie-Jiang
Copy link
Contributor

Can you double check the topic existence and kafka settings? This seems a Kafka issue to me.
You may also get more help in the slack troubleshooting channel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ingestion real-time troubleshooting
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Jackie-Jiang @Abdulsamirkhan