Why ForkJoinPool.scan takes so many CPU

[ministat]

I used caffeine cache (2.3.1) in my app, and I found the performance is not as good as expected. I used JVM async profiler to check, and found the following, which indicates ForkJoinPool.scan take 55.48% CPU.

I used sized based eviction. Is there any way to reduce the overhead from ForkJoinPool?

          ns  percent  samples  top
  ----------  -------  -------  ---
231953308268   55.48%    19455  java.util.concurrent.ForkJoinPool.scan
 43136220969   10.32%     3597  java.util.concurrent.ForkJoinPool.signalWork
 13304004590    3.18%     1108  __pthread_cond_wait
 11903444750    2.85%      913  __pthread_cond_signal
 10348843079    2.48%      839  __pthread_mutex_unlock
  6196516146    1.48%      528  java.util.concurrent.ForkJoinPool.awaitWork
  5888444860    1.41%      484  com.github.benmanes.caffeine.cache.BoundedLocalCache$AddTask.run
  5333306060    1.28%      413  java.util.concurrent.ConcurrentHashMap.computeIfPresent

My cache stats is:

Caffeine cache stats: CacheStats{hitCount=6117343, missCount=143356, loadSuccessCount=71678, loadFailureCount=0, totalLoadTime=5130935, evictionCount=51678, evictionWeight=51678}

[ben-manes]

The cache does not need FJP for its own operations and you can instead supply a caller-runs / direct executor, e.g. Runnable::run. That works fine for many users and our non-concurrent tests use that for simplicity.

An executor is used for any async code that may call a foreign (user supplied) method, since we won't know how expensive that is. For example during eviction a listener can be supplied for notification within the atomic map computation that removes the entry. The cache's own operations are fast and O(1), but users might do something unexpectedly expensive. If done on the calling thread then it could impact user-facing latencies, so the executor helps hide that.

Caffeine also offers an async cache where the computation occurs through a CompletableFuture. That requires an executor, so we pass through the one configured on the cache. Java defaults to ForkJoinPool if not specified. Therefore we need an executor option regardless and using ForkJoinPool is the least surprising. When virtual threads become official then I think we'll switch our default since that has better characteristics for I/O workloads and FJP was an odd JVM choice since it is designed for cpu bound tasks.

Please also use the latest version of Caffeine, either 2.x (if Java 8) or 3.x (Java 11 or above). This will avoid you from suffering from known and fixed bugs, as that version is 6 years old. Similarly try to stay on a recent JDK for the same reason, e.g. there were bugs in FJP that impacted our users which we tried to mitigate and JDK fixes resolved. If an upgrade isn't possible then using a direct executor is probably an okay alternative.

[ministat]

Thanks very much for your prompt reply.
I also found neo4j used "ExecutorBasedCaffeineCacheFactory" which used Runnable::run. I'll try Runnable::run also. Meanwhile, I'll update to 2.9.3 per your suggestion.