Mutable Cache with Background Refreshes

[r39132]

Hi,
I have the following requirements for a local cache. I'm trying to understand if caffeine is a good fit. We have a high-fidelity data streams-as-a-service business with each microservice in the data plane running on Spring Boot. We currently use caffeine. One issue we have run into is that when a key is missing (either due to expiry or microservice cold-start), our data streams pause awaiting the cache entry to be load. This loading causes us to fetch data via our "cache wrapper microservice" which hits redis. The cache wrapper service does expose batch fetch APIs, so we can load and reload the entire local cache as needed, as long as it happens in the background and at a fixed cadence.

A wrinkle, on cache miss, we do want to block the data stream pipeline to fetch the latest data from the central cache via the cache wrapper.

Refer to details below:

I mentioned that I want key to stick around for an "infinite amount of time", by which I meant the lifetime of the app, but I don't mind replacement as long as expired keys can be refreshed in the background and on expiry. I don't want any pauses in data streaming... the caches are needed for stream processing.

I was originally planning on writing my own cache manager with each cache implemented as a loading cache, but then I noticed that it's immutable. A long time ago, I wrote a lot of the core caching functionality at Netflix -- I used soft and weak references and built a lot of logic around LInkedHashMap. I don't want to do that all again if there is a better way :-)

[ben-manes]

A wrinkle, on cache miss, we do want to block the data stream pipeline to fetch the latest data from the central cache via the cache wrapper.

You can avoid blocking by using getIfPresent to obtain the entry or null. The asMap() view might also be helpful.

Our refresh support is intended to be used with expiration, e.g. to allow active content to stay fresh and inactive content to fade away. A full reload is best suited by a scheduled, period task.

The cache.get is an alias for Map.computeIfAbsent, so you can use that only when helpful for blocking calls.

I was originally planning on writing my own cache manager with each cache implemented as a loading cache, but then I noticed that it's immutable.

I'm not sure what you mean by this, but I am also not very familiar with Spring Cache. We don't provide the cache manager and related framework, as it would duplicate efforts of the platform (spring, jcache, custom). It may be that Spring's needs to know all of the caches upfront.

A long time ago, I wrote a lot of the core caching functionality at Netflix -- I used soft and weak references and built a lot of logic around LInkedHashMap. I don't want to do that all again if there is a better way :-)

Caffeine is basically a fancy ConcurrentHashMap with various bounding options. If you think of it in those terms, like you did when writing your own, then you can choose the preferred features. It doesn't offer frameworky things beyond that, but rather provides the core data structure to base your own on top of.

[r39132]

Hi Ben,
I am trying to test the refreshAfterWrite and expireAfterWrite functionality in the LoadingCache, but it is not working.

Review the following classes:

1. This Utils class defines how the caches are created:

2. Here is the CacheLoader I have defined:

3. Here are 2 Tests that are failing:

4. Here are Exception stacks:

[ben-manes]

I recommend reading this wiki page for improving your tests.

A small tip is to use replace Thread.sleep with a Ticker, a logical clock. You can use Guava's testlib which has a FakeTicker utility. Then you can advance time.

A refresh occurs asynchronously, so you are subject to race conditions which can test failures. If you specify a direct, same-thread executor (e.g. Runnable::run) then the work will be performed by the caller.

To debug, you might add a log message in CacheLoader#reload. This by default calls load(key), but you can specify something else. This would help you see what the cache is doing.

[r39132]

Ben.
1. I have extended my code to support running in the calling thread and to use tickers. Note that I use com.google.common.base.Ticker in this class so that the Guava Testlib's Fake Ticker is compatible.

**2. I have changed my test to use the Fake Ticker and I have added log calls **

3. See the output. Some thing is strange. The loaded key is not visible in the next get() call. The stale value is still returned. I get the same result whether I run in the same thread executor or not.

I expect to get the Object with samplingRate == 12, but I still get 11.

[ben-manes]

I am free if you would like to talk on a Zoom, as to be honest it's a bit hard to debug code over screenshots. :)

[r39132]

Note : My refreshAfterExpiry() works after I fixed the sleep bug you caught. I will move it to ticker as well, but the test works.

[r39132]

RefreshAfterWrite
Why is this not working? I have set the refresh interval at 5 seconds and wait 10 seconds for the entry to refresh, but still get the same value for one of the fields in the object -- you will notice that I expect to get 11, 12, 13.... on subsequent calls.

RefreshAfterExpiry
Similar question here.. on key expiry, shouldn't I expect to get different objects.

Also, I do the load() function execute one time, but only on the first get. I don't see it get called when the entry should have expired or become stale.

17:35:43.197 [Test worker] INFO io.datazoom.connector.base.cache.ConnectorConfigCacheLoader - Load called for key = name returned CMD = ConnectorMetadata(mappingId=name, dzConnectorId=null, customerCode=null, apiUrl=null, active=true, dzInternal=false, samplingRate=11, metadata={})

[ben-manes]

Would you be able to generalize the code so that I can run it through a debugger?

FYI - Thread.sleep is in milliseconds, but your refreshIntervalSeconds is only 5, not 5000. You might be suffering from races.

[r39132]

Ah.. good catch on the thread.sleep :-)

[r39132]

Ben,
Our caffeine-based cache is now in production! I have observability (monitoring and alerting) via AWS Cloudwatch enabled. I am using caffeine's cache stats.

I have a question about the unit of the avg_cache_load_penalty. Is it in milliseconds or microseconds? If in milliseconds, the numbers I am seeing don't make sense. For example, 30,000,000 in microseconds would be 30s, but in milliseconds would be 8.3 hours.

Also, how is this stat computed within caffeine? I am using the Loading cache with a 5m refresh interval and 3 hour TTL for my cache entries.

[ben-manes]

That's wonderful!

That's a good JavaDoc fix. The time measurement is in nanoseconds, per totalLoadTime. The average doesn't state that, unfortunately. The calculation is totalLoadTime / (loadSuccessCount + loadFailureCount).

We call Ticker.read() prior to invoking the loading function, call it again when the function returns the value, and update the StatsCounter. This means it is the time that it took to acquire the entry lock + run your function. Every cache load, whether explicit or a refresh, is captured by these metrics.

[r39132]

Great! Super useful. Thanks for the clarification @ben-manes.

Nanoseconds makes sense.

Our load function calls a various APIs in a microservice called Cache-Wrapper, which adds some light business logic, transparent redis failover logic, and observability around all redis (central cache) operations. So, the numbers make sense. The p95 for the API calls is a few single millis, which is less than the 30 mills we see for avg_cache_load_penalty.