On Artifact Repository Path and Implementation

[ttrently]

Hello all!

I wanted to continue the modularity overview discussion here, specifically on the artifact repository implementation. If you haven't checked out the discussion linked above, I suggest reading through that first as most of the following thoughts are based on the concepts proposed there.

Below I will show my take on the artifact repository implementation and its association with a package repository, but would like to know others thoughts on it as well.

Proposal

The interaction between a package repository and artifact repository lends itself to be that of a one-to-many connection in that a package repository can connect to and pull from multiple different artifact repositories, while an artifact repository can contain any number of package payloads associated with multiple package repositories.

When packages are then requested in a resolved context, package repositories that require artifact payloads will cache those payloads in a blocking way to ensure they are present for the environment.

Paths

With the introduction of an artifact repository association, package paths become a bit more complex to create that connection for the user. Default formats of repo_type@location would still work, but we need the ability to tell a repository to look for artifacts in a specific location. Because of this we may look at constructing a more robust string representation, possibly in the following format:

repo_type@location?art_repo@location&art_repo@location*

Where anything defined past the ? query character would be an associated artifact repository when the package repository is constructed and cached.

package_paths = [
    '~/packages',
    'mongo@my-mongo-uri-1?s3@s3://bucket-1&s3@s3://bucket-2',
    'mongo@my-mongo-uri-2?s3@s3://bucket-1'
]

In this example we have three package repositories and two artifact repositories defined.

• This first entry points to the users default filesystem packages repository on disc.
• The second entry uses a MongoDB instance to query packages from and can pull / cache from two different S3 artifact repositories.
• The final entry also uses a MongoDB instance at a different location but can pull / cache from the same S3 artifact repository that the second entry is aware of.

Since there is an increase in complexity here with the associations, we will probably want to create a simple interface for package repository paths, compared to the normal string manipulation that is done now. This new class would simply deconstruct the package repository path provided and split it.

class PackageRepositoryPath(object):

    _sep = '?'

    def __init__(self, path):
            self.package_repository_location = None
            self.artifact_repository_locations = []
            
            # Deconstruct the provided path here....

class PackageRepositoryManager(object):

    def get_repository(self, path):
        path = PackageRepositoryPath(path)
        
       # Do rest of normal get_repository method...

Package Repository

The package repository should behave as-is but be extended to have knowledge of it's associated artifact repositories. In this way the package repository would handle iterating package definitions but not own the logic for downloading or caching the payloads. This functionality should be the ownership of the artifact repository in conjunction with the individual resources. (This becomes apparent in the caching process where we have access to the resource which then has access to the repository.)

class PackageRepository(object):

    def __init__(self, location: PackageRepositoryPath, resource_pool):
        ...
        self.artifact_repositories = location.artifact_repositories


class Resource(object):

    def get_artifact(self):
        artifact = None
        for repo in self._repository.artifact_repositories:
            artifact = repo.find(name=self.name(), version=self.get('version'))
       return artifact

    def install(self, location=None):
        ...

The package cache would then be updated to use the new Resource.install method during add_variant which should then allow for more robust handling of variant caching from repositories in the future (package_cache.py line 292).

try:
    variant.resource.install(rootpath)
finally:
    still_copying = False

Alternative to this is that we wouldn't use the caching system to handle artifacts and instead download them into the users local filesystem repository and resolve again from there. This option could work but the caching systems seems like this is a good place for this to happen as it is cleanable and allows for re-caching of variants.

Artifact Repository

The artifact repository can be implemented as an additional plug-in type so that it can be extend by users and customized for independent needs. This repository should have the ability to pull, push, and find artifacts within it based on package or variant resources.

ArtifactRepository
    .copy_variant_to_path
    .copy_variant_from_path
    .variant_exists
    .find_variant
    .download_variant
    .install_variant

ArtifactRepositoryManager
    .get_repository

The difference between download and install variant here is to allow for packaged downloads to occur and the install process handling actually unpacking the payload. Ex. You may store an artifact package called foo in a zip format as foo.zip on your artifact repository. The download method would handle pulling foo.zip into a local temporary location, while install would handle unpacking the contents of the payload and moving them into the cache. (Install method here would always call download to ensure a payload is present to install).

Artifact repositories would also have the ability to copy packages between them to support move functionality (copy_variant_to_path, copy_variant_from_path). These methods would potentially be exposed in the package repository to allow for easy upload access points during a rez build or install.

PackageRepository.copy_variant_to_artifact_repository(variant_resource, target_artifact_repository)

It would then be possible for database based package repositories to also update based on the provided variant, ie. when a variant is copied, an additional entry is pushed to the package repository database based on a schema. This could be standardized in general for rez database package repositories to ensure specific behavior and format.

artifact_repository_schema = dict(
    name='foo',
    version='1.0.0',
    timestamp=12345,
    package='...', # Contents of foo/package.py
    location='s3://bucket-1', # Location in the artifact repository
)

Final Thoughts

If you made it this far, thanks for reading my ramblings! I'm sure I've missed something.

I'd be interested on any thoughts about this extension of the artifact repository proposal and it's implementation, especially if this seems close to how others envisioned an artifact style repository working and how it is associated with the package repository.

Some additional open questions:

• Should artifact download and installation have callback methods through the resource, package repository, or both?
• Maybe if the resource should already have that information from the package repository when it is created?
• Does the proposed path definition lend to future iterations and extension of the artifact repository?
• Is the package cache the best solution for handling artifact payloads?

[JeanChristopheMorinPerso]

Thanks @ttrently ! Great write up.

I still need to read it a couple more times and digests some stuff, but here are some quick thoughts, notes and questions:

• How are credentials handled? This is going to be a critical question to answer as I don't think admins would be happy to have credentials hardcoded in config files.
• Multithreading should be built-in
• Why does the artifact plugin has a find function? Shouldn't it just download/upload (or copy) and read? The package definition repository should he a rht responsibility of searching and finding. The artifact repo should only contain the package payloads (so everything except the package.py) (and note that we usually say package definition, not package.py. that's because the package definition can be in multiple formats (currently YAML and Python, but it can also be stored purely in SQL or a document store).
• I think it makes sense to use the cache mechanism for that because I think the main idea of using something else than the filesystem plugin to store artifact is to exactly get rid of shared filesystems to store artifacts.
• I haven't decided yet on if I like the package strings. I feel like the proposed solution would make things complicated, which might be a sign that we need something else. I'd like to see what other systems are doing (if there is any equivalent). So basically prior art.
• Have you explored other ideas so far? If so it could be interesting to document them in your proposal.
• We should come up with a list of use cases to see help determine what should be supported or not.

Feel free to ignore some of the stuff I mentioned. I'll come back later with more details on some points. I just wanted to drop my thoughts to not forget them.

[ttrently]

Thanks for reading through! I'll answer as many of these as I can below.

How are credentials handled? This is going to be a critical question to answer as I don't think admins would be happy to have credentials hardcoded in config files.

The way we are currently testing is using a settings.yaml or plugin based rezconfig file that defines some of the connection settings, and any sensitive information is pulled via windows keychain. This prevents us having to store user login information anywhere, but means that user accounts have to be consistent (maybe not as much of an issue as long as IT is controlling it).

    @classmethod
    def _format_location(cls, location):
        # construct location settings
        settings = _settings.get(MongoDB.name(), {})
        settings["location"] = location

        loc = "mongodb://{user}:{pass}@{location}/" \
            "?authMechanism=DEFAULT" \
            "&authSource={authSource}".format(**settings)

        return loc

The settings object here is just constructed in the package repository file where things like authSource and authMechanism are stored in the config, but user and pass are pulled from the keychain.

I'm not sure if this is the best method but it would make sense for users to potentially configure that location template connection string and set authentication protocols as well in a settings file.

Multithreading should be built-in

Correct, the default cache mechanism is multithreaded but can cause an issue where if the package payload hasn't been downloaded fast enough, that it is not found on resolve. I think we could add a config setting to allow for caching to be (optionally) set to a blocking process that ensures the package download before continuing.

Why does the artifact plugin has a find function? Shouldn't it just download/upload (or copy) and read? The package definition repository should he a rht responsibility of searching and finding. The artifact repo should only contain the package payloads (so everything except the package.py) (and note that we usually say package definition, not package.py. that's because the package definition can be in multiple formats (currently YAML and Python, but it can also be stored purely in SQL or a document store).

Oversight mostly on my part! You are correct though, the package repository should do searching and the artifact plugin would handle only pull / push. I think there's something to be said maybe for package verification or existence on the artifact repository, but maybe blind pulls are ok for duck typing.

I haven't decided yet on if I like the package strings. I feel like the proposed solution would make things complicated, which might be a sign that we need something else. I'd like to see what other systems are doing (if there is any equivalent). So basically prior art.

I'm with you on this one, I'm not sold entirely on more complex package strings, but also wasn't really sure if I wanted to make it (package paths implementation) more complex or not for this discussion. I'd be interested on other takes at how these should be defined or serialized.

I did toss around the idea of using an index based system where you would define package and artifact repository locations separately, then use the index to assign them to a package repository, but never went past that much thought.

Have you explored other ideas so far? If so it could be interesting to document them in your proposal.

I have and will try to pull them from past failed attempts to post here haha.

We should come up with a list of use cases to see help determine what should be supported or not.

Agreed, a full feature list attached to this enhancement would probably be the easiest point of entry to others reading through it.