Caffeine cache instance size calculation

[myfrndjk]

Hi,

I am using caffeine cache for one of my usecase. The moment I added my first entry into caffeineCacheInstance the size is increasing upto to 10000%

Sample code

      String json = "{\"namasdccsxsascdsae\":\"Johdssdasdadwin\", \"ag89e\":\"0werweweweewe3weesax0\", " +
                "\"cweweeqewear\":\"12werewsdwrdwwrdwdwd3\"}";

        JSONParser parser = new JSONParser();
        JSONObject jsonObject = (JSONObject) parser.parse(json);
        SizeOf sizeOf = SizeOf.newInstance();
        HashMap<Integer, Cache<String, JSONObject>> allThemes = new HashMap<>();

        int totalThemes = 2;
        int entriesPertTheme = 50000;

        String lastThenename = null;
        for (int i = 0; i < totalThemes; i++) {

            Cache<String, JSONObject> caffineCacheInstance = buildLoadingCache(50000);
            System.out.println("Empty caffinetheme size" + sizeOf.deepSizeOf(caffineCacheInstance));
            for (int j = 0; j < 1; j++) {
                String randomKey = lastThenename = getRandomName();
                System.out.println("size of this jsonObject is " + sizeOf.deepSizeOf(jsonObject));
                System.out.println("size of this randomKey is " + sizeOf.deepSizeOf(randomKey));
                caffineCacheInstance.put(randomKey, jsonObject);
                System.out.println("size of this theme is " + sizeOf.deepSizeOf(caffineCacheInstance));
            }

            //System.out.println("size of this theme is " + sizeOf.deepSizeOf(caffineCacheInstance));

            caffineCacheInstance.cleanUp();
            Thread.sleep(1000);
            allThemes.put(i, caffineCacheInstance);
        }

        System.out.println("All entries inside entriesPertTheme map" + allThemes.size());
        System.out.println("size of 200 themes in bytes" + sizeOf.deepSizeOf(allThemes));

I am getting following response

Empty caffinetheme size 2128
size of this jsonObject is 576
size of this randomKey is 56
size of this theme after adding element 296968

Is this expected growth or am i doing something wrong.

[ben-manes]

I am unsure how your cache is configured, but the cache is allowed to exceed the maximum size by a modest amount to allow for improved concurrency.

Internally the cache uses a write buffer to enque pending operations that need to be replayed against the eviction policy. This avoids having all writing threads block and run sequentially, which would be the common case once the cache is full. Instead, after the map operation the policy work is handed off and scheduled to be processed immediately. A typical cache would synchronize all writes and cause lock contention, even though the work to maintain LRU is inexpensive. In this model a batch of work can be applied and writes to distinct keys can be fully concurrent. The write buffer is bounded so if the write rate greatly exceeds the eviction rate then back pressure is applied where writers block to assist in the eviction, thereby avoiding runaway growth.

You can run the eviction on the caller, e.g. Caffeine.executor(Runnable::run), which could mask this if there was no other concurrent writes. The cache's own maintenance work is cheap, but a user's eviction listener might not be so we default to being async to avoid penalizing the calling thread. This bit of slack is not too unlike GC as memory is not instantaneously reclaimed and instead performed by a fast batch operation, allowing for concurrent work elsewhere in the system.

We allow for 128 * NCPUs pending write ops, so this overflow is more visible on small caches in a tight insertion loop. It's more common to perform per-key loads though the cache, so with a good hit rate the writes are much less common than reads, meaning usually won't see it in practice. The cache favors minimizing the user-facing latencies before applying a strict threshold, so the maximum is a watermark but it will throttle once the tolerance limit was reached.