caffeine lock issue for using get with mappingFunction

[PatrickZhangOracle]

As caffeine use JAVA 8 concurrenthashmap as the data structure, when we use the method get(K key, Function<? super K, ? extends V> mappingFunction), it will invoke concurrenthashmap compute method, so it will lock the head node in the map bucket . So always mappingFunction is time consuming operation like fetch from database, then any other operations like put which hash to the same head node will be blocked by waiting lock. Is that cost too much? I see guava use a fake node to insert first ,after that judge if the node is loading then blocked in the same entry, it reduce the lock range to one specific node.

[ben-manes]

Most of the time this isn't a problem because the number of reads is much higher than writes. This library's focus on an excellent hit rate means fewer misses and therefore fewer loads. Since the map will evenly distribute entries there is also a good amount of inherent write concurrency. If profiling your application does indicate a bottleneck then there are two very simple approaches that can fix the problem.

1. The number of bins increases with the hash table's size, so a larger map will have a higher write concurrency level. A small map with a high write rate can set a larger initialCapacity to increase the number of bins. The implementation docs of the map state,

Lock contention probability for two threads accessing distinct elements is roughly 1 / (8 * # of elements) under random hashes.

2. To avoid holding a hashbin lock during the computation you might consider using AsyncCache. This is a bounded Map<K, CompletableFuture<V>> so the in-flight load is performed outside of the map. This is similar to your suggestion as Guava's underlying mechanism is a SettableFuture held by the entry.

[PatrickZhangOracle]

Thanks much for the reply. Will have a try for AsyncCache.

[ben-manes]

A difference worth noting is that the caller no longer performs the work. This can be a bit wasteful if your caller will merely block and requires another thread to be dispatched on (perhaps less so in the future with Loom's virtual threads). Its a little more work, but your code can insert the future and compute it on the caller to avoid additional threads. This is what Apache Solr does in their usage.

[PatrickZhangOracle]

Just test AsyncCache, its great! The node value is future type, and if one key expired, only one thread need invoke the mappingFunction. Also thanks for the great advice of "compute it on the caller to avoid additional threads".

[ben-manes]

I updated the FAQ to describe this solution. See the section on write contention regarding the initialCapacity trick, and on recursive computations for performing the load outside of the hash table. In that case it is to avoid a limitation of ConcurrentHashMap which disallows computations from writing back into the map, so we can use an identical technique for recursion.