Looking for suggestion for intensive write (and evict)

[renyingxin]

Hi,
[problem description]
we rely on caffeine eviction algorithm to keep hot keys in memory,
exist in cache? 1) yes return 2）no, do business logic and save to caffeine

but we have 10-20 millions or more keys, but in memory we can only hold a small portion of these keys(value is also very large, 1-10K)

[doubt]
this may cause many concurrent write and eviction(we alreay use strippedlock to controll concurrent write for each key), we found the temprary object is increasing rapidly and can not be revoked in short time (I guess the caffeine evict proccess may take some time before the cache object can be revoked)

really appreciate if can give any suggestion design or workaroud on this

this is related to #719

[ben-manes]

I'm not sure based on that description, so here are some random musings in case any of them strike home.

To support concurrent writes, a dedicated buffer is used so that writers don't serialize against an eviction lock. This buffer is capped to 128 * ceilingPowerOfTwo(NCPU), e.g. 1024 for an 8 core system. If populated then the cache tries to aggressively flush it. If full then writers fallback to acquiring the eviction lock, which deschedules to the thread and allows the holder to be prioritized. The thread evicting performs a batch of work by flushing the read and write buffers under the lock. Since this is a small, bounded buffer it doesn't cause runaway growth and can absorb small bursts of concurrent writes.

In benchmarks the write and eviction rate is in the many millions of operations per second, which should be more than adequate for real usages. That is also reduced simply by the policy's higher hit rate, as fewer misses means fewer loads and evictions.

The evicting thread is scheduled by the Caffeine.executor(e), defaulting to ForkJoinPool.commonPool(). Sometimes applications starve this executor with blocking calls, since it is the default for CompletableFuture. In that case it delays eviction until the cache's write buffer is full and it forces a fallback. In those cases you might try setting Caffeine.executor(Runnable::run) to see if it allows for more prompt behavior.

Other blocks can occur due to long running computations. That can be mitigated using AsyncCache to decouple the compute from ConcurrentHashMap's bin lock. A long compute risks that eviction pick a victim that resides in the same hashbin and collide on that lock, whereas a per-entry locking via the future makes these map operations immediate. See the FAQ for more details, which also shows how to use it without additional threads if that is a concern.

If the cache entries are eligible for collection then you might need to tune your GC. It has been a while, but I think increasing the young gen to reduce premature oldgen promotions helps, as it allows the GC to sweep those entries up. That was true for older collectors like CMS if you are still on Java 8, but not often a problem on newer (regional) collectors like the Java 9+ default, G1.

I hope some of that was helpful. Otherwise please provide some more details and we can try to figure it out.

[renyingxin]

1、The evicting thread is scheduled by the Caffeine.executor(e), defaulting to ForkJoinPool.commonPool(). Sometimes applications starve this executor with blocking calls, since it is the default for CompletableFuture. In that case it delays eviction until the cache's write buffer is full and it forces a fallback. In those cases you might try setting Caffeine.executor(Runnable::run) to see if it allows for more prompt behavior.

I tried setting executor, and print threadpool metrics in log every minute(activecount, queuesize, etc), but the metrics shows not very busy, and still not fix the problem

2、If the cache entries are eligible for collection then you might need to tune your GC. It has been a while, but I think increasing the young gen to reduce premature oldgen promotions helps, as it allows the GC to sweep those entries up. That was true for older collectors like CMS if you are still on Java 8, but not often a problem on newer (regional) collectors like the Java 9+ default, G1.

we use CMS in production, our young gen is already 12G, will try to increase this to 24G or try G1 to see whether it works

[ben-manes]

CMS was of known for not scaling up to large heaps, so that does sound like the culprit. It was removed in later jdks, once G1 became stable enough for adoption. G1 had a mixed reputation before that as it took many years to mature out of experimental mode. You’ll need to be mindful of what your jdk handles, as I don’t know how often GC changes were backported to jdk8. You can look into g1, shenandoah, zgc as options.

[ben-manes]

looking at some very old notes from a previous job, we set CMS with -XX:NewRatio=8 to fix a similar problem. HTH

[renyingxin]

In production our jvm setting is -Xms16384m -Xmx16384m -Xmn12288m, which means newratio is about 75%....

Before we introduce caffeine cache, we use redis as the cache, It's about 2-3 hours will do a FGC (we have a lot of temporary object, so we set the young gen to very large)

In order to reduce user latency and the dependecy on redis (redis may goes down...), we use caffeine+redis as mutlti-tier cache
but now our system do a FGC in 20-30 minutes, which means we get many more temporary objects... we are now trying different methods to fix the problem in dev enviroment

[ben-manes]

A full garbage collection usually means that the young generation became full and not enough space was reclaimed within a duty cycle, forcing a full collection to free up memory. That can happen at too high of an allocation rate, large objects forcing a promotion,
soft reference pollution, too small of a young gen causing early promotions, etc. This implies that your young generation cannot be evacuated effectively, seems concerning.

What is your cache configuration?

Caffeine normally won't allocate on a read, except if using weakKeys since it must use identity comparison. However, that wrapper key's lifetime is the map lookup, so it should be friendly to escape analysis (stack allocated) and easily GC'able.

On a write it does allocate an entry and a task to replay the operation against the policy. That task is short-lived and the buffer's maximum size enforces a backoff from runaway growth.

If you are using weak/soft references then that does impose additional work on the GC. That is usually fine for weak references, but soft references are mostly problematic.

A common problem with optimization is that improving one area can reduce performance by exacerbating a problem elsewhere. For example if resolving lock contention, many threads may be blocked higher up in the stack. When removing those contention points, all converge at a single lock lower in the stack, now being operated on at a much higher frequency than before. Of course removing that bottleneck then offers breakthrough speed, but at first it can degrade performance. You might have done similar by removing one bottleneck but creating a more severe one elsewhere. Java Flight Recorder is quite nice for investigating these cases.

CMS was deprecated and removed, so you should evaluate alternatives. A regional collector allows for more aggressive collection with lower pause times.

Hope that helps.