How the neetbox.integrations are designed

[visualDust]

The problem is, as we want a not-dependency-intensive neetbox, we are trying to remove codes with heavy dependencies from core functions. To be specific, in order to avoid installing package 'thop'(which depends on torch) for non-torch users, we have removed import torch from the headings of each python file into the line where the packages are required to overcome the import errors for most of the packages. For example, in neetbox.torch.profile:

from thop import profile as _profile
def profile():
    case 1: ... return
    case 2: ... return
    case that needs thop: ... return

was changed to:

def profile():
    case 1: ... return
    case 2: ... return
    case that needs thop: 
            assert pkg.is_installed('thop')
            from thop import profile as _profile
            ... 
            return

It was simple until we added neetbox.integrations. I'm not a pro for Python things, but I found it much more clear if we move things related to third parties into neetbox.integrations. So what does the neetbox.integrations stand for? In general, neetbox.integrations contains plugins for the original neetbox codes which empowers neetbox with third-party-compbilities.

I first mentioned the idea in #19 , in which I suggest that we should move 'nvidia-smi' and 'conda' command line getters out of neetbox.utils. However, since I'm a little confused about how to effectively add plugins into python class, I'm not sure what to do next to form the general framework for neetbox.integrations. Anyway, I think I need some help. Further discussions and methods are welcomed and appreciated.

[PommesPeter]

I think we can use modulefinder a builtin python module to integrate nvidia-smi or conda like command, and achieve the goal above. here is the doc. modulefinder

[visualDust]

Currently, I'm trying with patch as a workaround. But I think it's not gonna be so easy. I will put my new ideas here as soon I get them.

Not to mention, an example here:

# package neetbox.integrations.tensorboard
from neetbox.logging.logger import Logger
from neetbox.utils.utils import patch
from neetbox.integrations import pkg
from typing import Any

assert pkg.is_installed('tensorboard',try_install_if_not=True)
import tensorboard

@patch
def write_tensorboard(self:Logger, what:Any,):
    pass # to write something into tensorboard

The code adds a new function caller 'write_tensorboard' into class Logger. So later in my code, I can do such thing:

from neetbox.logging import logger
import neetbox.integrations

logger.write_tensorboard(whatever)

[visualDust]

It's a kinda solution, but idk how my IDEs perform with the code completion... I'll find it out later when I have time.

[visualDust]

Another example for patch in neetbox.utils.utils:

[visualDust]

@PaperCube @PommesPeter What do you think? Do you find it plausible?

[visualDust]

I'm running out of ideas. I'm rubbish indeed.

[PaperCube]

I think @patch is a nice design. Nevertheless, I think traditional OOP abstract classes might also be a feasible solution.

[visualDust]

another approach scans all submodules in neetbox.integrations and let them register a self loading function after workspace is loaded. #72
cbcc66d

it's kinda piece of scaffold, I will make it into a stronger implementation later, just keep you guys up to date.

[PommesPeter]

So, how should we design the neetbox.intergrations.

[visualDust]

So, how should we design the neetbox.intergrations.

neetbox.integrations is now moved to neetbox.extension. it automatically iterate submodules and try to initialize them. neetbox.extension provides a decorator which allows functions run on neetbox workspace initialization. for example, in neetbox/extension/environment/hardware.py:

from neetbox.extension import on_workspace_load

# run this function on workspace initialization
@on_workspace_load(name="hardware-monit")
def load_monit_hardware():
    cfg = get_module_level_config()
    if cfg["monit"]:  # if do monit hardware
        hardware.set_update_intervel(cfg["interval"])

        # watch updates in daemon
        @watch(name="hardware", _channel=SYSTEM_CHANNEL, interval=cfg["interval"], initiative=False)
        def update_env_stat():
            return hardware.json()

once after neetbox loads up its config, it runs this function. Therefore, you can add custom sub-modules as extensions in neetbox.extension, and specify what to do after workspace config is loaded. note that do not use get_module_level_config or any config relating things before neetbox finish reading workspace config, that is, make sure you are using get_module_level_config() only in function decorated by @on_workspace_load.

[visualDust]

this is a temporary workaround which will be redesigned in the next beta release.

[visualDust]

Important updates

in #76, there is a patch fix (ef5b792) which allows submodules in neetbox.extension to provide default config. Example:

in neetbox\extension\environment\hardware.py:

from neetbox.config import export_default_config, get_module_level_config
from neetbox.extension import on_workspace_loaded

...

@export_default_config()
def return_default_config() -> dict:
    return {"monit": True, "interval": 0.5}


# watch updates in daemon
@on_workspace_loaded(name="hardware-monit")
def load_monit_hardware():
    cfg = get_module_level_config()
    if cfg["monit"]:  # if do monit hardware
        hardware.set_update_intervel(cfg["interval"])

        # watch updates in daemon
        @watch(name="hardware", _channel=SYSTEM_CHANNEL, interval=cfg["interval"], initiative=False)
        def update_env_stat():
            return hardware.json()

in code above, @export_default_config() means that neetbox will take the returned value of the decorated function as default configuration for this module(neetbox.extension.environment.hardware). when user initialize a new neetbox workspace via neet init, the provided config will be written into user's config file. config generated for code above:

[extension.environment.hardware]
monit = true
interval = 0.5

plus, @on_workspace_loaded() works as we have mentioned before, but the name is changed from @on_workspace_load to @on_workspace_loaded.