[OPIK-595]: Add Redis Caching part 1 #1015

thiagohora · 2025-01-09T17:35:53Z

Details

Add an annotation-based caching mechanism to allow us to define cacheable methods in the project.

Issues

OPIK-595

Testing

Adding Integration tests to the new mechanism

andrescrz

It's admirable that you went ahead with an implementation that tries to solve the problem in a general way, beyond just caching the automation rules.

This is also heavily based on the Spring Cache approach, which provides an interface with known annotations (@Cacheable, @CachePut and @CacheEvict).

However, this has many non-trivial edge cases and building a library like this is a problem by itself that requires a lot of focus. In our case it has to be just good enough, but already has some challenges: reactive vs non-reactive methods, distinguishing between types (collections and non-collections currently) etc. and this moves us away from focusing on the business cases strictly related to Opik.

I was wondering if you investigated some alternative that we can just use. Some examples:

Something in Dropwizard.
Some Java Library than can be wrapped around the Redis.
Integrating some parts of Spring Cache.

I'd be fine to move forward, but it's desirable to investigate this first, in order to keep focusing on building Opik use cases.

andrescrz · 2025-01-10T09:32:12Z

apps/opik-backend/pom.xml

@@ -198,6 +198,16 @@
            <artifactId>httpclient5</artifactId>
            <version>5.2.3</version>
        </dependency>
+        <dependency>
+            <groupId>org.mvel</groupId>
+            <artifactId>mvel2</artifactId>


Minor: we could evaluate using Drools as well. But this is fine.

If you believe Drools, it's better that we use it.

It should be ok to go with this as well.

andrescrz · 2025-01-10T09:53:59Z

apps/opik-backend/src/main/java/com/comet/opik/domain/AutomationRuleEvaluatorService.java

@@ -157,6 +157,7 @@ public <E, T extends AutomationRuleEvaluator<E>> T findById(@NonNull UUID id, @N
                        case LlmAsJudgeAutomationRuleEvaluatorModel llmAsJudge ->
                            AutomationModelEvaluatorMapper.INSTANCE.map(llmAsJudge);
                    })
+                    .map(evaluator -> (T) evaluator)


Minor: for a safer cast, better check if instance of in a filter. Anyhow, it might not ever happen or maybe it's ok to allow this error to bubble up if it ever happens an unexpected programming error.

Yeah, given the switch, I imagine this would not happen, but Sonar keeps complaining. I changed it so it only complains in one line instead of the whole method

apps/opik-backend/src/main/java/com/comet/opik/infrastructure/cache/CacheInterceptor.java

andrescrz · 2025-01-10T10:33:16Z

apps/opik-backend/src/main/java/com/comet/opik/infrastructure/cache/CacheInterceptor.java

+            key = getKeyName(name, keyAgs, invocation);
+        } catch (Exception e) {
+            // If there is an error evaluating the key, proceed without caching
+            log.warn("Cache will be skipped due to error evaluating key expression");


Please log the exception as well.

The error is logged on getKeyName

In that case, I think it's better you remove the catch entirely from getKeyName and just allow exceptions to bubble up as you're having a catch at Exception level here anyway. You don't need the conversion to IllegalArgumentException. Then, just move the exception log to this catch.

andrescrz · 2025-01-10T10:40:49Z

apps/opik-backend/src/main/java/com/comet/opik/infrastructure/cache/CacheInterceptor.java

+        if (isReactive) {
+            return action.apply(key, name);
+        }
+
+        return action.apply(key, name).block();


Can you avoid this distinction and return the result of applying the action without blocking? Would it work by delegating the block call to the handler of this result?

This code would fail if the method annotated returns a non-reactive type. Otherwise, we would restrict the annotation of reactive methods.

andrescrz · 2025-01-10T10:57:40Z

apps/opik-backend/src/main/java/com/comet/opik/infrastructure/cache/CacheInterceptor.java

+    }
+
+    private Mono<Object> processCacheEvictMethod(MethodInvocation invocation, boolean isReactive, String key) {
+        if (isReactive) {


The same applies for all the other methods. We should try to have a general implementation, that doesn't distinguish by reactive or not. Otherwise, this library won't work for other frameworks. Imagine that we use RxJava.

Yeah, I would love to do that, but we need to know the type to handle the cache properly without missing the business logic.

apps/opik-backend/src/main/java/com/comet/opik/infrastructure/cache/CacheInterceptor.java

andrescrz · 2025-01-10T11:03:34Z

apps/opik-backend/src/test/java/com/comet/opik/infrastructure/cache/CacheManagerTest.java

+                                        new CustomConfig("cacheManager.enabled", "true"),
+                                        new CustomConfig("cacheManager.defaultDuration", "PT0.500S"),
+                                        new CustomConfig("cacheManager.caches.%s".formatted(CACHE_NAME_2), "PT0.200S")))


Just configure these values in the test config yaml file instead of overwriting them.

I would keep the cache disabled by default and let the specific test enable it. Otherwise, the side effects may impact other tests.

thiagohora · 2025-01-10T11:27:52Z

It's admirable that you went ahead with an implementation that tries to solve the problem in a general way, beyond just caching the automation rules.

This is also heavily based on the Spring Cache approach, which provides an interface with known annotations (@Cacheable, @CachePut and @CacheEvict).

However, this has many non-trivial edge cases and building a library like this is a problem by itself that requires a lot of focus. In our case it has to be just good enough, but already has some challenges: reactive vs non-reactive methods, distinguishing between types (collections and non-collections currently) etc. and this moves us away from focusing on the business cases strictly related to Opik.

I was wondering if you investigated some alternative that we can just use. Some examples:

Something in Dropwizard.

Some Java Library than can be wrapped around the Redis.

Integrating some parts of Spring Cache.

I'd be fine to move forward, but it's desirable to investigate this first, in order to keep focusing on building Opik use cases.

Yeah, I looked for something similar to Spring Cache but couldn't find it. The closest one was https://github.com/bazaarvoice/dropwizard-caching-bundle, but it's archived. I also found some others using ehcache and caffeine, but all seem old and without maintenance.

https://github.com/gabber12/dropwizard-caffeine-configurator

Anyway, I agree with you it's definitely not a trivial problem, but I didn't try to implement a solution to all scenarios. Instead,d I tried to go with something small that would cover our basic scenario, and if needed, we could continue building it.

andrescrz · 2025-01-10T15:52:31Z

It's admirable that you went ahead with an implementation that tries to solve the problem in a general way, beyond just caching the automation rules.
This is also heavily based on the Spring Cache approach, which provides an interface with known annotations (@Cacheable, @CachePut and @CacheEvict).
However, this has many non-trivial edge cases and building a library like this is a problem by itself that requires a lot of focus. In our case it has to be just good enough, but already has some challenges: reactive vs non-reactive methods, distinguishing between types (collections and non-collections currently) etc. and this moves us away from focusing on the business cases strictly related to Opik.
I was wondering if you investigated some alternative that we can just use. Some examples:

Something in Dropwizard.

Some Java Library than can be wrapped around the Redis.

Integrating some parts of Spring Cache.

I'd be fine to move forward, but it's desirable to investigate this first, in order to keep focusing on building Opik use cases.

Yeah, I looked for something similar to Spring Cache but couldn't find it. The closest one was https://github.com/bazaarvoice/dropwizard-caching-bundle, but it's archived. I also found some others using ehcache and caffeine, but all seem old and without maintenance.

https://github.com/gabber12/dropwizard-caffeine-configurator

Anyway, I agree with you it's definitely not a trivial problem, but I didn't try to implement a solution to all scenarios. Instead,d I tried to go with something small that would cover our basic scenario, and if needed, we could continue building it.

Discussed offline, but leaving my proposal here as well. However, it requires some investigation from your side:

In Java there's a standard API for caching, it's name is JCache. Actually, Spring Cache is pretty similar to it, many classes and even the annotations have the same or equivalent names.

We use Redisson as Redis client, which I believe provides an implementation for JCache. I think there's a lot of overlap with your implementation, including the annotations and some classes or interfaces such as CacheManager.

Additionally, Redisson supports the reactive API and I think they have support for that in their JCache implementation.

I think there are chances that your implementation would be greatly simplified, an the problems that I'm pointing out in my review would go away (or mostly) with this approach.

I'm not completely sure, but it's likely to be a matter of glueing things together: Redisson and its JCache implementation, with our use cases.

thiagohora · 2025-01-14T08:27:24Z

It's admirable that you went ahead with an implementation that tries to solve the problem in a general way, beyond just caching the automation rules.
This is also heavily based on the Spring Cache approach, which provides an interface with known annotations (@Cacheable, @CachePut and @CacheEvict).
However, this has many non-trivial edge cases and building a library like this is a problem by itself that requires a lot of focus. In our case it has to be just good enough, but already has some challenges: reactive vs non-reactive methods, distinguishing between types (collections and non-collections currently) etc. and this moves us away from focusing on the business cases strictly related to Opik.
I was wondering if you investigated some alternative that we can just use. Some examples:

Something in Dropwizard.

Some Java Library than can be wrapped around the Redis.

Integrating some parts of Spring Cache.

I'd be fine to move forward, but it's desirable to investigate this first, in order to keep focusing on building Opik use cases.

Yeah, I looked for something similar to Spring Cache but couldn't find it. The closest one was https://github.com/bazaarvoice/dropwizard-caching-bundle, but it's archived. I also found some others using ehcache and caffeine, but all seem old and without maintenance.
https://github.com/gabber12/dropwizard-caffeine-configurator
Anyway, I agree with you it's definitely not a trivial problem, but I didn't try to implement a solution to all scenarios. Instead,d I tried to go with something small that would cover our basic scenario, and if needed, we could continue building it.

Discussed offline, but leaving my proposal here as well. However, it requires some investigation from your side:

In Java there's a standard API for caching, it's name is JCache. Actually, Spring Cache is pretty similar to it, many classes and even the annotations have the same or equivalent names.

We use Redisson as Redis client, which I believe provides an implementation for JCache. I think there's a lot of overlap with your implementation, including the annotations and some classes or interfaces such as CacheManager.

Additionally, Redisson supports the reactive API and I think they have support for that in their JCache implementation.

I think there are chances that your implementation would be greatly simplified, an the problems that I'm pointing out in my review would go away (or mostly) with this approach.

I'm not completely sure, but it's likely to be a matter of glueing things together: Redisson and its JCache implementation, with our use cases.

As we discussed offline, the Jcache implementation is focused on CacheManagers and its APIs. There are no benefits concerning cache resolution and abstraction. Having said that, we agree to keep thinking about it and improving this implementation in subsequent iterations.

andrescrz

Left a few more comments, but I'm happy for this to move forward and address them in a follow-up PR, as you need to send part 2 anyway.

Thanks for the whole investigation about JCache. Very well done!

andrescrz · 2025-01-14T07:59:49Z

apps/opik-backend/pom.xml

@@ -198,6 +198,16 @@
            <artifactId>httpclient5</artifactId>
            <version>5.2.3</version>
        </dependency>
+        <dependency>
+            <groupId>org.mvel</groupId>
+            <artifactId>mvel2</artifactId>


It should be ok to go with this as well.

andrescrz · 2025-01-14T08:01:27Z

apps/opik-backend/src/main/java/com/comet/opik/infrastructure/cache/CacheEvict.java

+@Retention(RetentionPolicy.RUNTIME)
+@Inherited
+@Documented
+public @interface CacheEvict {


We discussed the JCache approach, but what about just reusing their interface annotations. Any blocker?

The JCache annotations only have a cache name property and other configuration classes, such as a cache resolver and a key generator (which must be implemented using reflection based on the method signature). For those reasons, I kept these annotations, which have a cache name and key. We use MVEL to solve the key expression. I believe this is less verbose and a bit more flexible.

andrescrz · 2025-01-14T08:55:24Z

apps/opik-backend/src/main/java/com/comet/opik/infrastructure/cache/CacheInterceptor.java

+            key = getKeyName(name, keyAgs, invocation);
+        } catch (Exception e) {
+            // If there is an error evaluating the key, proceed without caching
+            log.warn("Cache will be skipped due to error evaluating key expression");


In that case, I think it's better you remove the catch entirely from getKeyName and just allow exceptions to bubble up as you're having a catch at Exception level here anyway. You don't need the conversion to IllegalArgumentException. Then, just move the exception log to this catch.

andrescrz · 2025-01-14T08:56:15Z

apps/opik-backend/src/main/java/com/comet/opik/infrastructure/cache/CacheInterceptor.java

+    private Mono<Object> processCacheEvictMethod(MethodInvocation invocation, boolean isReactive, String key) {
+        if (isReactive) {
+            try {
+                return ((Mono<?>) invocation.proceed())


Minor: maybe use map to convert these types and just remove the casting.

Sorry, I didn't understand. The invocation.proceed() returns Object, I need to cast to go from Object to Mono. Maybe it didn't understand you properly.

andrescrz · 2025-01-14T08:58:40Z

apps/opik-backend/src/main/java/com/comet/opik/infrastructure/cache/CacheInterceptor.java

+            if (evaluatedKey.isEmpty() || evaluatedKey.equals("null")) {
+                throw new IllegalArgumentException("Key expression cannot return an empty string");
+            }


Related to my other comment. This IllegalArgumentException is going to go to the catch statement below and wrapped into another IllegalArgumentException. You should be good to just remove this try-catch as it's just a catch and re-throw pattern that doesn't provide much value in this sceneario.

The issue here is to avoid the use of an empty string and a null string. Because otherwise it would be a valid string.

andrescrz · 2025-01-14T09:01:03Z

...-backend/src/test/java/com/comet/opik/api/resources/v1/priv/ChatCompletionsResourceTest.java

@@ -55,6 +55,8 @@
 /// - **Openai**: runs against a demo server and doesn't require an API key
 /// - **Anthropic**: set `ANTHROPIC_API_KEY` to your anthropic api key
 @TestInstance(TestInstance.Lifecycle.PER_CLASS)
+@Disabled


You can clean this up. It's no longer necessary. These tests have been disabled in another PR.

thiagohora requested a review from a team as a code owner January 9, 2025 17:35

thiagohora force-pushed the thiagohora/OPIK-595_redis_caching_part_1 branch from c9df8cc to 7b723b1 Compare January 9, 2025 17:36

[OPIK-595]: Add Redis Caching part 1

1247a9b

thiagohora force-pushed the thiagohora/OPIK-595_redis_caching_part_1 branch from 7b723b1 to 1247a9b Compare January 9, 2025 17:37

thiagohora added 2 commits January 9, 2025 20:31

Disable failing test

4c366db

Merge branch 'main' into thiagohora/OPIK-595_redis_caching_part_1

7bca1a5

thiagohora self-assigned this Jan 9, 2025

thiagohora added 3 commits January 10, 2025 09:38

Add more tests

ac810c3

Support to collections

f6050e6

Use compiler args

593da4a

thiagohora force-pushed the thiagohora/OPIK-595_redis_caching_part_1 branch from dbeeb59 to 593da4a Compare January 10, 2025 10:57

andrescrz reviewed Jan 10, 2025

View reviewed changes

Add validation

b281c47

thiagohora requested a review from andrescrz January 10, 2025 11:52

thiagohora added 2 commits January 10, 2025 15:06

Add support to Flux

f949ded

Merge branch 'main' into thiagohora/OPIK-595_redis_caching_part_1

30ebc9e

thiagohora added 4 commits January 13, 2025 10:29

Merge branch 'main' into thiagohora/OPIK-595_redis_caching_part_1

b208b65

Make wrapper property more generic

36ec5f5

Merge branch 'main' into thiagohora/OPIK-595_redis_caching_part_1

1455f48

Fix format

d6468a8

andrescrz approved these changes Jan 14, 2025

View reviewed changes

thiagohora merged commit 272dfb8 into main Jan 14, 2025
8 checks passed

thiagohora deleted the thiagohora/OPIK-595_redis_caching_part_1 branch January 14, 2025 09:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[OPIK-595]: Add Redis Caching part 1 #1015

[OPIK-595]: Add Redis Caching part 1 #1015

thiagohora commented Jan 9, 2025

andrescrz left a comment

andrescrz Jan 10, 2025

thiagohora Jan 10, 2025

andrescrz Jan 14, 2025

andrescrz Jan 10, 2025

thiagohora Jan 10, 2025

andrescrz Jan 10, 2025

thiagohora Jan 10, 2025

andrescrz Jan 14, 2025 •

edited

Loading

andrescrz Jan 10, 2025

thiagohora Jan 10, 2025 •

edited

Loading

andrescrz Jan 10, 2025

thiagohora Jan 10, 2025

andrescrz Jan 10, 2025

thiagohora Jan 10, 2025

thiagohora commented Jan 10, 2025

andrescrz commented Jan 10, 2025

thiagohora commented Jan 14, 2025

andrescrz left a comment

andrescrz Jan 14, 2025

andrescrz Jan 14, 2025

thiagohora Jan 14, 2025

andrescrz Jan 14, 2025 •

edited

Loading

andrescrz Jan 14, 2025

thiagohora Jan 14, 2025

andrescrz Jan 14, 2025

thiagohora Jan 14, 2025

andrescrz Jan 14, 2025

[OPIK-595]: Add Redis Caching part 1 #1015

[OPIK-595]: Add Redis Caching part 1 #1015

Conversation

thiagohora commented Jan 9, 2025

Details

Issues

Testing

andrescrz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrescrz Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thiagohora Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thiagohora commented Jan 10, 2025

andrescrz commented Jan 10, 2025

thiagohora commented Jan 14, 2025

andrescrz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrescrz Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrescrz Jan 14, 2025 •

edited

Loading

thiagohora Jan 10, 2025 •

edited

Loading

andrescrz Jan 14, 2025 •

edited

Loading