Thread limit exceeded replacing blocked worker

[gumeg1]

HI
I am fairly new to Cafeeine Cache. We are moving from Guava Cacheloader to Caffeine Cacheloader.
In large data environment, we can see exceptions :

"Exception thrown during refresh",
,"logger_name":"com.github.benmanes.caffeine.cache.BoundedLocalCache","stack_trace":
"java.util.concurrent.CompletionException: java.util.concurrent.RejectedExecutionException: Thread 
limit exceeded replacing blocked worker\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable
(CompletableFuture.java:314)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)\n\tat 
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)\n\tat
 java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)\n\tat
 java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)\n\tat
 java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)\n\tat 
 java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)\n\tat 
 java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)\n
 Caused by: java.util.concurrent.RejectedExecutionException: **Thread limit exceeded replacing
 blocked worker\n\tat java.base/java.util.concurre**nt.ForkJoinPool.tryCompensate(ForkJoinPool.java:1575)
 \n\tat java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3115)\n\tat 
 java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)\n\tat 
 java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)\n\tat 

It seems to happen when we call

CacheLoader.reload(CacheLoader.java:171)
 \n\tat com.github.benmanes.caffeine.cache.CacheLoader.reload(CacheLoader.java:171)\n\tat 
 com.github.benmanes.caffeine.cache.CacheLoader.lambda$asyncReload$2(CacheLoader.java:198)
 \n\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run

Can anyone help what's the issue? Any solutions? Thanks

[ben-manes]

This indicates that you might be abusing the ForkJoinPool shared in the JVM. That has a limit set by ForkJoinPool.DEFAULT_COMMON_MAX_SPARES, where it may create new threads if the workers are blocked. This often happens when futures block on other futures and cause pool starvation. Since the pool is shared, it may be that other parts of your application are abusing it and the extra usage here causes an overflow.

You can specify a different pool in Caffeine.executor or CacheLoader.asyncReload which delegates to reload. Guava defaulted to running on the calling thread, which you can sort of simulate using Caffeine.executor(Runnable::run).

[gumeg1]

thanks, will give that a try

[ben-manes]

@gumeg1 any luck on a fix?