tools: introduce git-change-exec tool #4202

christoph-zededa · 2024-09-03T17:12:01Z

This new tool detects if in your git tree changed files (compared to master branch and local-only changed files) and uses this information to run only the specified actions.

Here it is used to run pillar's go-tests only if something changed there, same for this tool itself and the get-deps tool.

christoph-zededa · 2024-09-03T17:53:20Z

No, this is not possible with what gh checks out here, so closing ...

christoph-zededa · 2024-09-03T18:19:43Z

No, this is not possible with what gh checks out here, so closing ...

hmm, seems it is possible, but I don't know what I am doing with those gh actions ...

OhmSpectator · 2024-09-04T09:31:46Z

Do you know how you're going to use it? Will developers use it for local runs, or will you make it part of CI?

christoph-zededa · 2024-09-04T09:37:18Z

Do you know how you're going to use it? Will developers use it for local runs, or will you make it part of CI?

It is used by make test which is also used by the CI.
So the answer is: both

deitch · 2024-09-04T09:43:49Z

I am a little bit confused. Is this tool's purpose to check which directories have changed files, and thus determine where we should run go test? For example, if pkg/a had changes and pkg/b did not, then run go test ./pkg/a as opposed to go test ./...?

Does it actually execute the go test, or just give the relevant paths, sort of like:

$ gce ./...
pkg/a
pkg/c
pkg/q

thus telling us which dirs changed? And we always can choose to ignore it and run all tests? And with it, I would run go test $(gce ./...)? Something like that?

A README would help, along with some sample of what it would look like when run.

OhmSpectator · 2024-09-04T09:46:29Z

Aha! I've got the idea. I like it. Good luck with the implementation =) I find it easier to write such tools in interpreted languages, but okay, let it be =)

christoph-zededa · 2024-09-04T09:47:35Z

I am a little bit confused. Is this tool's purpose to check which directories have changed files, and thus determine where we should run go test? For example, if pkg/a had changes and pkg/b did not, then run go test ./pkg/a as opposed to go test ./...?

Does it actually execute the go test, or just give the relevant paths, sort of like:
$ gce ./...
pkg/a
pkg/c
pkg/q
thus telling us which dirs changed? And we always can choose to ignore it and run all tests? And with it, I would run go test $(gce ./...)? Something like that?

A README would help, along with some sample of what it would look like when run.

It does not do magic, it just looks in actions.go what to do exactly, f.e.: https://github.com/lf-edge/eve/pull/4202/files#diff-5bf19742ee6060707f406c1625c23226124e161d2049d12a80a8a61ef076ace8R47

Let me just be lazy with the example: https://github.com/lf-edge/eve/actions/runs/10699251983/job/29660387351?pr=4202#step:3:120 :-)

tools/git-change-exec/Makefile

OhmSpectator · 2024-09-04T12:15:59Z

tools/git-change-exec/main.go

+			return dir
+		}
+
+		dir = filepath.Join(dir, "..")


I remember the case when Eden removed my home dir because it accessed something in .., and I'm afraid of such patterns now. Could you consider using git rev-parse --show-toplevel instead?

https://github.com/go-git/go-git ?

Doesn't git rev-parse --show-toplevel just do the same?

45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/agentlog", {st_mode=S_IFDIR|0755, st_size=226, ...}, 0) = 0 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/agentlog/.git", 0x7ffe67413ec0, 0) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/agentlog/.git/HEAD", 0x7ffe67413d70, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/agentlog/HEAD", 0x7ffe67413d70, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar", {st_mode=S_IFDIR|0755, st_size=1204, ...}, 0) = 0 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/.git", 0x7ffe67413ec0, 0) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/.git/HEAD", 0x7ffe67413d70, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/pillar/HEAD", 0x7ffe67413d70, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg", {st_mode=S_IFDIR|0755, st_size=804, ...}, 0) = 0 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/.git", 0x7ffe67413ec0, 0) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/.git/HEAD", 0x7ffe67413d70, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/pkg/HEAD", 0x7ffe67413d70, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve", {st_mode=S_IFDIR|0755, st_size=824, ...}, 0) = 0 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/.git", {st_mode=S_IFDIR|0755, st_size=240, ...}, 0) = 0 45077 newfstatat(AT_FDCWD, "/home/christoph/projects/eve/.git/HEAD", {st_mode=S_IFREG|0644, st_size=49, ...}, AT_SYMLINK_NOFOLLOW) = 0

https://github.com/go-git/go-git ?

I am not sure what you mean.

I'm afraid of manually adding logic to reverse directories. If there is a tool that does it for us, I would prefer it.

Now it is using https://github.com/lf-edge/eve/pull/4202/files#diff-711d61f4f130257efb2bca48eebbdc4d829cf9f3f687f00014abe0c9aafc2ccfR49

Yay, thanks! Will take a look at the rest of the script later...

OhmSpectator · 2024-09-09T07:24:34Z

tools/git-change-exec/actions-lint.go

+
+func (s *lintSpdx) gofix(path string) {
+	license := []string{
+		"// Copyright (c) 2024 Zededa, Inc.\n",


The year should not be hardcoded.
And we cannot put Zededa as it's the lf-edge-wide code... (we still can have such a tool in the zededa/eve-tools repo)

Agreed on both points. Check for current year and no check on the holder of the copyright.

The year is now the current year and the organization is read from the .gitconfig.

As the check is now using the spdx shell script, it does not check on the holder for the copyright.

christoph-zededa · 2024-09-09T09:51:07Z

I don't know what yetus complains about

results-full.txt from https://github.com/lf-edge/eve/actions/runs/10770711970/job/29864646035#step:5:20 is empty

make mini-yetus shows:

christoph@ponyhof ~/p/eve (git_change_detect)> cat /tmp/yetus.hBuc5lILQs/yetus-output/results-full.txt 
home/christoph/projects/eve/pkg/nvidia/Dockerfile:30:25:hadolint: invalid flag: --keep-git-dir
home/christoph/projects/eve/pkg/pillar/containerd/oci_test.go:19:2:revive: should not use dot imports https://revive.run/r#dot-imports
home/christoph/projects/eve/pkg/pillar/dpcreconciler/linux_test.go:16:2:revive: should not use dot imports https://revive.run/r#dot-imports
home/christoph/projects/eve/tools/check-commit-messages/check_commit_messages.py:15:0:pylint:[E0401(import-error), ] Unable to import 'git'
home/christoph/projects/eve/tools/parse-pkgs.sh:15:8:shellcheck: note: Double quote to prevent globbing and word splitting. [SC2086]
home/christoph/projects/eve/tools/parse-pkgs.sh:64:8:shellcheck: note: Useless echo? Instead of 'echo $(cmd)', just use 'cmd'. [SC2005]
home/christoph/projects/eve/tools/parse-pkgs.sh:64:8:shellcheck: warning: Quote this to prevent word splitting. [SC2046]
home/christoph/projects/eve/tools/parse-pkgs.sh:74:14:shellcheck: note: Use $(...) notation instead of legacy backticks `...`. [SC2006]
home/christoph/projects/eve/tools/parse-pkgs.sh:74:9:shellcheck: warning: Declare and assign separately to avoid masking return values. [SC2155]
home/christoph/projects/eve/tools/parse-pkgs.sh:84:10:shellcheck: note: Double quote to prevent globbing and word splitting. [SC2086]
home/christoph/projects/eve/tools/parse-pkgs.sh:92:16:shellcheck: note: Double quote to prevent globbing and word splitting. [SC2086]
home/christoph/projects/eve/tools/parse-pkgs.sh:92:8:shellcheck: note: Double quote to prevent globbing and word splitting. [SC2086]

yash-zededa · 2024-09-09T09:54:08Z

@christoph-zededa report.html shows

-1	patch	0m 2s		GH:4202 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

this should not be the case

christoph-zededa · 2024-09-09T10:31:26Z

@christoph-zededa report.html shows

-1	patch	0m 2s		GH:4202 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

this should not be the case

thx, I rebased; I guess now it works.

OhmSpectator · 2024-09-09T10:46:32Z

tools/git-change-exec/actions-test.go

+// Copyright (c) 2024 Zededa, Inc.
+// SPDX-License-Identifier: Apache-2.0
+
+// Copyright (c) 2024 Zededa, Inc.


No need for two SPDX headers =)

ups, that happened when I tested it on itself ...

OhmSpectator · 2024-09-09T12:40:27Z

tools/git-change-exec/main.go

+		actionCategories = append(actionCategories, c)
+	}
+
+	rootCmd := cobra.Command{


Am I right that we use this complicated pattern just to parse the "-d" argument? I mean: import cobra, create rootCmd with a lambda function as a value of one of the arguments, then - run two extra methods (BoolVarP(), Exec()).

For me it is not complicated, using go's flags is complicated ;-)

I switched to cobra because I wanted to have flags and arguments.

OhmSpectator · 2024-09-09T12:43:21Z

tools/git-change-exec/main.go

+		Run: func(cmd *cobra.Command, args []string) {
+			gce := newGitChangeExec()
+
+			for _, cat := range args {


What does cat mean?

I changed it to category

OhmSpectator · 2024-09-09T15:03:16Z

tools/git-change-exec/main.go

+		commonBase = append(commonBase, commit)
+		addBase, err := branchHead.MergeBase(commit)
+		if err != nil {
+			log.Printf("finding merge base failed: %v", err)


I guess this will fail in most cases as you will try to find a merge base for all the potential target branches here. If the branch is based on master and you try to find a merge base for -stable - the result is kind of undefined and irrelevant... Maybe we can still call it here, but the error printing is unnecessary, I guess.

I removed the error printing, now it only does debugLog

OhmSpectator · 2024-09-09T15:16:00Z

tools/git-change-exec/main.go

+		}
+
+		for _, st := range commitStats {
+			gce.addActionByPath(st.Name)


I don't think it makes sense to add actions for files that don't exist in the current branch's HEAD.
Also, as far as I understand it, it will run for all the commits between the HEAD and the common baseS, even the oldest, the most irrelevant -stable... It will include tens/hundreds of irrelevant commits, hence - files. Or did I get the algorithm wrong?

It will stop if it finds a hash that is in at least one of the common baseS

OhmSpectator · 2024-09-09T15:22:53Z

tools/git-change-exec/main.go

+			gce.fetchOrigin()
+
+			gce.collectActionsGitTree()
+			gce.collectDirtyGitTree()


I need to read the function code to understand the purpose of these calls or the difference between them. I guess they define the actions to be run, but the names could be more clear.

For the second function, I would make it optional, depending on the flag. If I prepare a PR, I still can have some modified files that will not go into the commits: ./conf/*, for example - and I don't need to check them.

I would prefer to make it opt-out with --ignore-dirty or perhaps:

git-change-exec run test to run as right now

git-change-exec run-branch test to only consider collectActionsGitTree()

But I think the behavior could then be a bit unexpected, f.e. if you change two files in pillar, then no action will be triggered, now you git add one of those files, then the tests for pillar will be triggered, but the tests include the second file that has not been added yet.

OhmSpectator · 2024-09-09T15:26:16Z

tools/git-change-exec/main.go

+			continue
+		}
+
+		fp := filepath.Join(gce.gitPath, file)


Shouldn't we ignore the removed files?

it does an os.Stat here: https://github.com/lf-edge/eve/pull/4202/files#diff-711d61f4f130257efb2bca48eebbdc4d829cf9f3f687f00014abe0c9aafc2ccfR297
and ignores the file if it does not exist.

Ah, sorry. I missed the lines

I, too 🤣
when I tried to find out ...

christoph-zededa · 2024-09-09T17:00:35Z

I added https://github.com/lf-edge/eve/pull/4202/files#diff-159efde8a5a441c430d0d0c512d4dd1553136852df9f0ac2443606d2faf1db5cR38-R46

which was the original motivation to have this as the tests take several minutes

eriknordmark · 2024-09-10T15:42:12Z

tools/git-change-exec/actions.go

+		getDepsTestAction{},
+		bpftraceCompilerExecTest{},
+	},
+	"lint": []action{


I don't see any invocation of the tool with "lint" as the argument. Is that missing or hiding somewhere?

So far I would keep it to be run by the user, but I can also add a Makefile target, perhaps lint.

eriknordmark · 2024-09-10T15:45:59Z

tools/git-change-exec/actions.go

+var actions = map[string][]action{
+	"test": []action{
+		pillarTestAction{},
+		gitChangeExecTest{},
+		getDepsTestAction{},
+		bpftraceCompilerExecTest{},


Seems like all of these are of the form
if match on directory prefix, then run this command

Do we need to put that as golang code or could we e.g., have that in some separate file? E.g., an actions_test file will be used for test, actions_lint file for lint, and actions_foo for a future foo target?
Or have the action filename be an argument to the tool instead of passing e.g., "test" as the argument?

Currently it is in different files actions-test.go and actions-lint.go, but unfortunately it is not possible with reflection to read which structs are there that implement the action interface.

As an experiment I tried to use a golang interpreter: 66803f0#diff-5bf19742ee6060707f406c1625c23226124e161d2049d12a80a8a61ef076ace8R72
But then I thought, what if there are dependencies between actions (like f.e. for linting, one would like first to run the spdx-fixer and then go fmt), so I would have to implement a way to express dependencies.
I still strive for the tool to be simple 😆

But I don't see why we need the generality of golang to run the actions, since the actions do simple matches on paths and then exec code. And it would be a lot more modular and easier to understand if we don't have to put various make invocations in the golang code but keep that in separate files IMHO.

I don't see why we need the generality of golang to run the actions

F.e. for the lint action I need it, because it has to edit files and add the copyright statement.
Of course I could do it that git-change-exec invokes another program to do so, but then I have to add:

finding the path to the other program

serialization

deserialization including error(-string) parsing

eventually implementing a way to save state between match and do and handle a proper cleanup

(the first point makes it harder to develop this tool, because I test it on another clone of the eve repository)

more modular and easier to understand

I changed the code so that it runs make test for the tests, so the Makefile is the abstraction, but I fear that in the future the actions become more complex, so only type action interface will be an abstraction.

since the actions do simple matches on paths

That was just, because I did not add yet to ignore READMEs and markdown files; also pkg/pillar/agentlog/cmd does not need to trigger the go tests. Now it is there.
The lint action has to save the matches in an array to be able to fix them in the do method.

Generally speaking I think match can become a lot more complex in the future once f.e. we only run the tests based on code coverage.

but keep that in separate files

What would these files look like?

Suggestions from my side:

put every action in it's own go-file

enhance dry-run to show the match result of every file

add a doDryRun() to the action interface where every action has to describe what it plans to do

This is needed for git-change-exec. Signed-off-by: Christoph Ostarek <[email protected]>

Sometimes hard tabs come from a copy-paste example and the example should not be modified. Also tabs are better than spaces ;-) Signed-off-by: Christoph Ostarek <[email protected]>

also allow to use this tool directly with a path as a parameter Signed-off-by: Christoph Ostarek <[email protected]>

The option is called `ignore-words` according to https://github.com/codespell-project/codespell?tab=readme-ov-file#using-a-config-file Also add `OptionAll` as is not a misspelled `optionally` Signed-off-by: Christoph Ostarek <[email protected]>

add this to have a somehow similar interface as other parts in the repo to run the tests Signed-off-by: Christoph Ostarek <[email protected]>

This new tool detects if in your git tree changed files (compared to master branch and local-only changed files) and uses this information to run only the specified actions. Here it is used to run pillar's go-tests only if something changed there, same for this tool itself and the get-deps tool. Signed-off-by: Christoph Ostarek <[email protected]>

in order to hope that yetus does not complain anymore Signed-off-by: Christoph Ostarek <[email protected]>

eriknordmark

My fundamental issue is that it is impossible to tell what make does without reading every line of the tool.

Can we make this more modular by splitting it into a git-change tool and a change-exec tool, with a list of files being the output from the first and the input to the second?

And I'm curious whether the git imvocations capture the base commit which the brach is based on, or if they compare against the current state of master. The latter would be quite suboptimal since it means I type make in my workspace once and it does little work, and then without any changes in my workspace or branch I type make 5 minutes later and then it does lots of work because something changed in master.

christoph-zededa · 2024-11-11T13:35:08Z

I didn't have time lately to work on it, so I move it into draft state.

christoph-zededa requested review from deitch and eriknordmark as code owners September 3, 2024 17:12

github-actions bot requested a review from jsfakian September 3, 2024 17:12

christoph-zededa marked this pull request as draft September 3, 2024 17:13

christoph-zededa force-pushed the git_change_detect branch from 97865b3 to 51a141e Compare September 3, 2024 17:21

christoph-zededa closed this Sep 3, 2024

christoph-zededa reopened this Sep 3, 2024

christoph-zededa force-pushed the git_change_detect branch 4 times, most recently from a9877ab to 575f3f0 Compare September 3, 2024 18:19

christoph-zededa force-pushed the git_change_detect branch from 575f3f0 to 2dd51fe Compare September 3, 2024 18:25

christoph-zededa force-pushed the git_change_detect branch 5 times, most recently from 6a6c964 to ea1bf18 Compare September 4, 2024 10:48

christoph-zededa marked this pull request as ready for review September 4, 2024 11:35

OhmSpectator reviewed Sep 4, 2024

View reviewed changes

christoph-zededa force-pushed the git_change_detect branch from ea1bf18 to d55a98b Compare September 4, 2024 13:38

christoph-zededa requested a review from yash-zededa as a code owner September 4, 2024 13:38

github-actions bot requested a review from uncleDecart September 4, 2024 13:39

christoph-zededa force-pushed the git_change_detect branch from d55a98b to 112ed2d Compare September 4, 2024 13:40

OhmSpectator reviewed Sep 9, 2024

View reviewed changes

christoph-zededa force-pushed the git_change_detect branch 3 times, most recently from 4385415 to 486b571 Compare September 9, 2024 09:45

christoph-zededa force-pushed the git_change_detect branch from 486b571 to 8a2f04d Compare September 9, 2024 10:31

OhmSpectator reviewed Sep 9, 2024

View reviewed changes

christoph-zededa force-pushed the git_change_detect branch from 8a2f04d to da15b2f Compare September 9, 2024 10:51

OhmSpectator reviewed Sep 9, 2024

View reviewed changes

christoph-zededa force-pushed the git_change_detect branch 2 times, most recently from 90d00f8 to 3e7e1d2 Compare September 9, 2024 16:54

eriknordmark reviewed Sep 10, 2024

View reviewed changes

christoph-zededa force-pushed the git_change_detect branch from 3e7e1d2 to 132f566 Compare September 20, 2024 12:37

github-actions bot requested a review from eriknordmark September 20, 2024 12:38

christoph-zededa added 6 commits September 20, 2024 15:46

.github: enable fetch-all for go test

353b327

This is needed for git-change-exec. Signed-off-by: Christoph Ostarek <[email protected]>

markdownlint: disable check for hard tabs

91f8b5e

Sometimes hard tabs come from a copy-paste example and the example should not be modified. Also tabs are better than spaces ;-) Signed-off-by: Christoph Ostarek <[email protected]>

tools: spdx-check file parameter

6442e86

also allow to use this tool directly with a path as a parameter Signed-off-by: Christoph Ostarek <[email protected]>

codespellrc: add OptionAll and fix

9ed6b04

The option is called `ignore-words` according to https://github.com/codespell-project/codespell?tab=readme-ov-file#using-a-config-file Also add `OptionAll` as is not a misspelled `optionally` Signed-off-by: Christoph Ostarek <[email protected]>

tools/get-deps: add 'test' to Makefile

66dda8c

add this to have a somehow similar interface as other parts in the repo to run the tests Signed-off-by: Christoph Ostarek <[email protected]>

christoph-zededa force-pushed the git_change_detect branch from 132f566 to ffe1b50 Compare September 20, 2024 13:46

git-change-exec: downgrade go.mod

d6a6fec

in order to hope that yetus does not complain anymore Signed-off-by: Christoph Ostarek <[email protected]>

eriknordmark reviewed Sep 22, 2024

View reviewed changes

christoph-zededa mentioned this pull request Oct 13, 2024

Remove dockerfile from checker lf-edge/eve-build-tools#30

Open

christoph-zededa marked this pull request as draft November 11, 2024 13:34

tools: introduce git-change-exec tool #4202

Are you sure you want to change the base?

tools: introduce git-change-exec tool #4202

Conversation

christoph-zededa commented Sep 3, 2024

christoph-zededa commented Sep 3, 2024

christoph-zededa commented Sep 3, 2024

OhmSpectator commented Sep 4, 2024

christoph-zededa commented Sep 4, 2024

deitch commented Sep 4, 2024

OhmSpectator commented Sep 4, 2024

christoph-zededa commented Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

christoph-zededa Sep 9, 2024 • edited Loading

Choose a reason for hiding this comment

christoph-zededa commented Sep 9, 2024

yash-zededa commented Sep 9, 2024

christoph-zededa commented Sep 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

christoph-zededa commented Sep 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

christoph-zededa Sep 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eriknordmark left a comment

Choose a reason for hiding this comment

christoph-zededa commented Nov 11, 2024

christoph-zededa commented Sep 4, 2024 •

edited

Loading

christoph-zededa Sep 9, 2024 •

edited

Loading

christoph-zededa Sep 10, 2024 •

edited

Loading