Does caffeine can dynamicly adjustment cache size by JVM memory watermark?

[guhanjie]

hi, @ben-manes , I have read your answer at this topic, i agree with you.

I met the likely problem, i use caffeine to cache plan entry(200K per object) in a SQL engine(actually Presto),
the cache policy is maximumSize=100000, my JVM memory is enough, will not OOM, but the cache entry will stay on Old Gen all the time, and it will lead to GC pressure.

I'm thinking about that, is there any GC-friendly policy for caffeine?
Does caffeine can dynamicly adjustment cache size by JVM memory watermark?

I used to realized it on hand, but the maximumSize seems to be static initialized with Cache, can not be update at runtime.

Anything tips?

[ben-manes]

The cache does not monitor available memory to adjust any setting. If you wish to do so, then it can be changed dynamically using the Cache.policy() api.

cache.policy().eviction().ifPresent(eviction -> {
  eviction.setMaximum(2 * eviction.getMaximum());
});

You can combine size-based eviction with soft references if you want a failsafe. This does add a small amount of per-entry memory overhead, though.

The cache's size eviction policy should be more GC friendly than a typical LRU, which violates the generational hypothesis (making it a GC benchmark for worst-case behavior). A SQL database is MRU / LFU biased, so the cache will auto-configure itself such that the admission policy will more readily reject a recent but infrequently used entry. As an MRU, this should increase the chance that the evicted entry is in the younger generation and reduce the GC pressure.

[krishna81m]

Ouch, totally forgot about weigher, was trying to break my head understanding how to leverage Policy instead. This example is what we needed.

Just to clarify, both maximumSize and maximumWeight are not possible together, but, eviction/expiry/refresh by time and maximumWeight based evictions should still work together? And entries with higher weights that are more frequently used should not be evicted following default TinyLFU algorithm.

[ben-manes]

Yep.

• maximumSize and maximumWeight are exclusive to each other as maximumSize is shorthand for using Weigher.singletonWeigher(). Internally the cache is always weight-aware and optimizes out unnecessary fields.
• size/weight, expiration, and refresh policies can all be combined and will work together.
• TinyLFU will only consider the entry's popularity, so an entry with a higher frequency is preferred regardless of its weight. There is a paper evaluating taking weights into consideration, but we have not yet adopted its recommendations.

[guhanjie]

@guhanjie , could you share your caffeine policy to evict based on Runtime.getRuntime().freeMemory() / Runtime.getRuntime().maxMemory() as it is impossible to do that. When you start evicting, actual GC to trigger freeing memory can happen much much later.

yes, GC would do free memory later, but if we make sure the some of cache has no reference(invalid by Caffeine, and not refer in any your own code), the GC will be easy to free them once triggered.

So, my policy is just simple to limit the cache size by monitoring jvm mem, not the strict limit, but the test is effective.

private final double USED_MEM_WATERMARK_RATIO = 0.75;
....        
scheduler.scheduleAtFixedRate(() -> {
            planCache.policy().eviction().ifPresent(eviction -> {
                long used = memoryUsage.getUsed();
                long total = memoryUsage.getMax();
                double ratio = used*1.0/total;
                if(ratio > USED_MEM_WATERMARK_RATIO) {
                    lastUpdateTime.set(System.currentTimeMillis());
                    long size = eviction.getMaximum()/2;
                    eviction.setMaximum(size); //mem watermark high, shrink cache size
                    log.info("Plan cache shrink, setMaximum %d, jvm memory water high: %d(used)/%d(total)", size, used, total);
                }
                else {
                    long durationInMills = System.currentTimeMillis()-lastUpdateTime.get();
                    if(eviction.getMaximum() < maximumSize && durationInMills > gracePeriodInMillis) {
                        lastUpdateTime.set(System.currentTimeMillis());
                        long size = Math.max(eviction.getMaximum()*2, maximumSize);
                        eviction.setMaximum(size); //mem watermark low, expand cache size
                        log.info("Plan cache expand, setMaximum %d, jvm memory water low: %d(used)/%d(total)", size, used, total);
                    }
                }
            });
        },5, 5, TimeUnit.SECONDS);

[adityamohan93]

@ben-manes Any chance the value level weight aware eviction is in the works? For our use case, we cache our database entries with String key and Long list value using Caffeine and there are usually queries that are not available in the database that results in added lookup time(ser-de) as well as database resource usage. We don't cache these unavailable queries as keys with something like an empty list in our Caffeine cache but considering it now. However it would lead to larger cache size as the String keys would be saved regardless and we might end up evicting non-empty lists prematurely. TinyLFU works on frequency, however if there is a weight factor also coupled with it, we can assign weight as zero to these empty cached lists so that they get evicted first if frequencies are similar between a zero and non-zero weighted key.

[ben-manes]

@adityamohan93,

A zero weighted key has an explicit meaning of don't size evict as it takes no capacity (used for pinning entries). So you could use negative caching without causing size evictions, but might also want to use an expiration policy to discard them to avoid runaway growth.

You shouldn't need to make business logic assumptions based on how the algorithm works. That's shared because it is fun CS stuff and I enjoy learning from the community's feedback. The size-aware TinyLFU only adds a modest benefit in the paper so you should still get excellent hit rates with our current approach. I also think your ideal might not work even then because we select by recency order (LRU lists) whereas you might be imagining an ordered queue based on a scoring function.