Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy Caching Support ("Diskcache") #58

Open
eyadgaran opened this issue May 23, 2021 · 2 comments
Open

Lazy Caching Support ("Diskcache") #58

eyadgaran opened this issue May 23, 2021 · 2 comments

Comments

@eyadgaran
Copy link
Owner

Definition:
reproducible outputs that are helpful to cache (eg probabilities, confusion matrix, etc)

Why not use artifacts?
artifacts must be created during training to save. cached objects may not be created until much later when they are actually necessary.

implementation:
only enable for structured inputs (need to be able to map input to cached output in order to be useful) like split objects or persistables

@eyadgaran
Copy link
Owner Author

could be coupled with #110 to guarantee inspectable types for function inputs

@eyadgaran
Copy link
Owner Author

pseudocode idea

def diskcache(save_pattern: str) -> Callable:
         def register(fn: Callable) -> Callable:
            register_cache_pattern(
                fn=fn.__name__, save_pattern=save_pattern
            )

            return partial(cache_wrapped_method, fn=fn)

         return register

def cache_wrapped_method(fn, *args, **kwargs):
       try: retrieve cache
       except: process, cache


@diskcache(default_save_pattern)
def expensive_operation(persistable, *args, **kwargs):
      ....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant