Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improving performance of dependency git pulling #2641

Closed
Andrioden opened this issue May 3, 2023 · 7 comments · Fixed by #2709
Closed

Improving performance of dependency git pulling #2641

Andrioden opened this issue May 3, 2023 · 7 comments · Fixed by #2709
Labels
kind/feature New feature or feature request

Comments

@Andrioden
Copy link
Contributor

Andrioden commented May 3, 2023 •
edited
Loading

Description

We are seeing some performance cost due to our complex circular repo dependency graph. This can be speed up by using a cached version of the git repo that is being pulled. Logs below should make it clear.

Feature request 1 - Cache which git repos branches have been pulled, se solution in section below.

Feature request 2 - It would also be nice to skip downloading dependencies when the dependencies are filtered out by something like --dependency "-". Se below.

PS C:\Projects\NHN\HN-Configuration> devspace deploy --dependency "-" --debug
20:04:42 info Using namespace 'dev'
20:04:42 info Using kube context 'kind-hn-cluster'
20:04:43 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
20:04:45 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Felleslogg
20:04:46 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
20:04:47 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
20:04:49 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
20:04:52 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-STS
20:04:53 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
20:04:55 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Configuration
20:04:55 debug Cyclic dependency found:
hn-sts
hn-configuration
hn-sts
20:04:56 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Felleslogg
20:04:57 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
20:05:01 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Personvern
20:05:02 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
20:05:04 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Configuration
20:05:04 debug Cyclic dependency found:
hn-personvern
hn-configuration
hn-sts
hn-personvern
20:05:05 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
20:05:06 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-STS
20:05:06 debug Cyclic dependency found:
hn-personvern
hn-sts
hn-personvern
20:05:06 debug Run pipeline:

Is your feature request related to a problem?

No.

Which solution do you suggest?

Within a single devspace deploy command you should store in memory a dictionary for each unique pair of repo url + branch name, which points to a dependency cache folder. This can then be reused deeper down the dependency hierarchy.

Which alternative solutions exist?

None
@Andrioden Andrioden added the kind/feature New feature or feature request label May 3, 2023
@Andrioden Andrioden changed the title Devspace deploy should avoid downloading same git repo multiple times Devspace deploy should avoid downloading same git repo multiple times, and respect --dependency param May 3, 2023
@Andrioden Andrioden changed the title Devspace deploy should avoid downloading same git repo multiple times, and respect --dependency param Improving performance of dependency git pulling May 12, 2023
@Andrioden
Copy link
Contributor Author

Andrioden commented Jun 3, 2023 •
edited
Loading

@lizardruss I am sorry to tag you, but could this be considered. Look at the example below.

TLDR

  • Takes 1min, download 8 repos accumulated 46 times
  • Should not download any at all

Run

C:\Projects\NHN\HN-Tjenester> devspace deploy --dependency "-" --debug
16:38:43 info Using namespace 'dev'
...
dependencies:
    hn-configuration:
        name: hn-configuration
        git: https://[email protected]/nhnfelles/Helsenorge/_git/HN-Configuration
        branch: master
    hn-felleslogg:
        name: hn-felleslogg
        git: https://[email protected]/nhnfelles/Helsenorge/_git/HN-Felleslogg
        branch: master
    hn-meldingssenter:
        name: hn-meldingssenter
        git: https://[email protected]/nhnfelles/Helsenorge/_git/HN-Meldingssenter
        branch: master
    hn-oppsett:
        name: hn-oppsett
        git: https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
        branch: master
    hn-personverninnstillinger:
        name: hn-personverninnstillinger
        git: https://[email protected]/nhnfelles/Helsenorge/_git/HN-PersonvernInnst
        branch: master
    hn-representasjon:
        name: hn-representasjon
        git: https://[email protected]/nhnfelles/Helsenorge/_git/HN-Representasjon
        branch: master
    hn-resources:
        name: hn-resources
        git: https://[email protected]/nhnfelles/Helsenorge/_git/HN-Resources
        branch: master
    hn-sts:
        name: hn-sts
        git: https://[email protected]/nhnfelles/Helsenorge/_git/HN-STS
        branch: master


16:38:46 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Configuration
16:38:48 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:38:49 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Felleslogg
16:38:50 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:38:51 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:38:53 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:38:54 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-STS
16:38:55 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:38:56 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Configuration
16:38:56 debug Cyclic dependency found:
hn-sts
hn-configuration
hn-sts
16:38:57 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Felleslogg
16:38:58 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:00 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Personvern
16:39:01 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:02 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Configuration
16:39:02 debug Cyclic dependency found:
hn-personvern
hn-configuration
hn-sts
hn-personvern
16:39:03 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:04 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-STS
16:39:04 debug Cyclic dependency found:
hn-personvern
hn-sts
hn-personvern
16:39:06 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Felleslogg
16:39:07 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Meldingssenter
16:39:08 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:09 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Felleslogg
16:39:10 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:11 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Personvern
16:39:13 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-STS
16:39:14 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-TOD-Legacy
16:39:15 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:16 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Felleslogg
16:39:17 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:18 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:21 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-PersonvernInnst
16:39:22 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:23 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Configuration
16:39:24 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Felleslogg
16:39:26 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Meldingssenter
16:39:27 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:28 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Personvern
16:39:29 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-STS
16:39:30 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Representasjon
16:39:31 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:33 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Configuration
16:39:34 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Felleslogg
16:39:35 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:37 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-STS
16:39:38 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Resources
16:39:39 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:41 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-Oppsett
16:39:42 debug Pulled https://[email protected]/nhnfelles/Helsenorge/_git/HN-STS
16:39:42 debug Run pipeline:
name: deploy
...

@lizardruss
Copy link
Collaborator

@Andrioden We will take a look and see if we can get this prioritized.

@Andrioden
Copy link
Contributor Author

Hi, did you consider this? I would hope its a low-hanging fruit, that would be a great time-saver for many!

@Andrioden Andrioden mentioned this issue Sep 4, 2023
Merged
@Andrioden
Copy link
Contributor Author

@lizardruss - I have solved and tested the easiest part of this in this PR: #2708

@Andrioden Andrioden mentioned this issue Sep 4, 2023
Merged
@Andrioden
Copy link
Contributor Author

@lizardruss - Aaannd i solved the second part of this issue in this PR: #2709

@Andrioden
Copy link
Contributor Author

Any chance a new release of devspace could be made?

@Andrioden
Copy link
Contributor Author

@lizardruss - Any chance of a new release?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/feature New feature or feature request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: correctly use local git repo dependencies cache
2 participants
@Andrioden @lizardruss