feat: tart macOS vm's as job container #2434

ChristopherHX · 2024-08-24T12:00:12Z

adds the tart:// protocol to platform mapping

e.g. -P macos-14=tart://ghcr.io/cirruslabs/macos-sonoma-base:latest if you have a mac.

I have written the code in a payed dedicated host machine that is now deallocated, expenses satisfied by sponsoring of my other act based project while I don't see anything here, as I don't own a mac.

add-path is probably broken, but I ignore the fact as of now.

github-actions · 2024-08-24T12:15:21Z

🦙 MegaLinter status: ✅ SUCCESS

Descriptor	Linter	Files	Errors	Elapsed time
✅ EDITORCONFIG	editorconfig-checker	10	0	0.03s
✅ REPOSITORY	gitleaks	yes	no	2.35s
✅ REPOSITORY	git_diff	yes	no	0.01s
✅ REPOSITORY	grype	yes	no	8.51s
✅ REPOSITORY	secretlint	yes	no	1.09s
✅ REPOSITORY	trivy-sbom	yes	no	0.11s
✅ REPOSITORY	trufflehog	yes	no	4.46s

See detailed report in MegaLinter reports
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

MegaLinter is graciously provided by

codecov · 2024-08-24T12:28:16Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.07%. Comparing base (5a80a04) to head (706746e).
Report is 122 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2434       +/-   ##
===========================================
+ Coverage   61.56%   75.07%   +13.51%     
===========================================
  Files          53       63       +10     
  Lines        9002    10019     +1017     
===========================================
+ Hits         5542     7522     +1980     
+ Misses       3020     1935     -1085     
- Partials      440      562      +122

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ChristopherHX · 2024-08-24T17:58:23Z

pkg/tart/environment_darwin.go

+func (e *Environment) Copy(destPath string, files ...*container.FileEntry) common.Executor {
+	return e.HostEnvironment.Copy(e.ToHostPath(destPath), files...)
+}
+func (e *Environment) CopyTarStream(ctx context.Context, destPath string, tarStream io.Reader) error {
+	return e.HostEnvironment.CopyTarStream(ctx, e.ToHostPath(destPath), tarStream)
+}
+func (e *Environment) CopyDir(destPath string, srcPath string, useGitIgnore bool) common.Executor {
+	return e.HostEnvironment.CopyDir(e.ToHostPath(destPath), srcPath, useGitIgnore)
+}
+func (e *Environment) GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error) {
+	return e.HostEnvironment.GetContainerArchive(ctx, e.ToHostPath(srcPath))
+}


As I'm overriding GetActPath with a relocated vm path and I'm copying into the bind mount those change needs to be undone by ToHostPath before calling the super implementation

ChristopherHX · 2024-08-24T17:59:01Z

pkg/tart/environment_darwin.go

+func (e *Environment) GetRunnerContext(ctx context.Context) map[string]interface{} {
+	rctx := e.HostEnvironment.GetRunnerContext(ctx)
+	rctx["temp"] = e.ToContainerPath(e.TmpDir)
+	rctx["tool_cache"] = e.ToContainerPath(e.ToolCache)
+	return rctx
+}


This is needed so RUNNER_TEMP are valid inside the vm, when used by actions cache

ChristopherHX · 2024-08-24T17:59:26Z

pkg/tart/environment_darwin.go

+func (e *Environment) GetActPath() string {
+	return e.ToContainerPath(e.HostEnvironment.GetActPath())
+}


This is needed so script paths passed to exec are valid inside the vm

ChristopherHX · 2024-08-24T18:00:29Z

pkg/tart/environment_darwin.go

+		return err
+	}
+
+	return e.execRaw(ctx, "ln -sf '/Volumes/My Shared Files/act' /private/tmp/act")


This is needed for working around bugs with paths using spaces, e.g. the shell templates of run steps

ChristopherHX · 2024-08-24T18:02:10Z

pkg/tart/env_darwin.go

+}
+
+func (e Env) VirtualMachineID() string {
+	return e.JobID


I'm reusing the docker container names used by containers, so no prefix and so on

ChristopherHX · 2024-08-24T18:02:43Z

pkg/runner/run_context_other.go

+
+func (rc *RunContext) startTartEnvironment() common.Executor {
+	return func(_ context.Context) error {
+		return fmt.Errorf("You need macOS for tart")


If someone thinks to run this mode on another OS, abort as tart is not available and some syscall part doesn't compile anyway.

ChristopherHX · 2024-08-24T18:04:30Z

pkg/runner/run_context_darwin.go

+		toolCache := filepath.Join(cacheDir, "tool_cache")
+		platImage := rc.runsOnImage(ctx)
+		platURI, _ := url.Parse(platImage)
+		query := platURI.Query()


I thought that I'm smart, but no providing options in the -P <label>=tart://test?... query doesn't actually work so only defaults for now

ChristopherHX · 2024-08-24T18:05:24Z

pkg/runner/run_context_darwin.go

+	"github.com/nektos/act/pkg/tart"
+)
+
+func (rc *RunContext) startTartEnvironment() common.Executor {


This started as a copy of hostenvironment, but is now different

ChristopherHX · 2024-08-24T18:07:04Z

pkg/runner/run_context.go

+func (rc *RunContext) IsTartEnv(ctx context.Context) bool {
+	platform := rc.runsOnImage(ctx)
+	image := rc.containerImage(ctx)
+	return image == "" && strings.HasPrefix(platform, "tart://")


like -self-hosted, but with tart://. I forget this variant has only been used in act_runner as of now, but looks better.

YOU54F · 2024-09-05T10:32:48Z

Just starting to test this out and it looks good on first pass. testing on a mac m1 mbp

if the gh action step fails, the container is left in a running state.

if the container fails to start (say 2 vm limit is reached), the container is left in a stopped state.

I've been using ssh, with tart / parallels / qemu to run jobs with -P ${PLATFORM}=-self-hosted on windows/macos/linux vms for the last year (https://gist.github.com/YOU54F/3ac099e54e48a31a69ac2d671aa878f6 / https://gist.github.com/YOU54F/789c53ae8380a1f90325952615635f8a )

I had a play with your runner.server & github-act-runner to build gh actions in my homelab with a set of vm's on a few diff machines. Cheers for your work

ChristopherHX · 2024-09-05T19:11:40Z

if the gh action step fails, the container is left in a running state.

Does --rm change anything here? Act has a feature for docker containers to keep them running if a step fails so you can debug, but reusing the vm has not been working in my testing..

I had a play with your runner.server & github-act-runner to build gh actions in my homelab with a set of vm's on a few diff machines.

Yeah I have written and use my runner.server prog exactly for such a use case, but mostly with actions/runner as the actual runner.

pkg/tart/environment_darwin.go

pkg/runner/expression.go

allexiusw · 2024-10-19T05:59:42Z

pkg/runner/run_context_darwin.go

+		_, _ = rand.Read(randBytes)
+		miscpath := filepath.Join(cacheDir, hex.EncodeToString(randBytes))
+		actPath := filepath.Join(miscpath, "act")
+		if err := os.MkdirAll(actPath, 0o777); err != nil {


Should 0o777 be added as a constant? Maybe FULL_RWX or FULL_PERMS

pkg/runner/run_context_other.go

+
+func (rc *RunContext) startTartEnvironment() common.Executor {
+	return func(_ context.Context) error {
+		return fmt.Errorf("You need macOS for tart")


allexiusw

I'm seeing not test coverage, do we require them in a PR in order to get merged in this repo?

ChristopherHX · 2024-10-19T07:37:21Z

I'm seeing not test coverage

This requires a self-hosted runner, macos-14 github runner cannot be used due to running already in a Virtual Machine.

Or stubbing tart, but then do we really know if tart works?
I wanted in the past to have a tart emulator that fakes creating VM, but didn't find anything,

This is not like intel macOS with nested virtualization... :(.

mergify · 2024-10-20T10:08:03Z

@ChristopherHX this pull request has failed checks 🛠

mergify · 2024-10-20T13:51:13Z

@ChristopherHX this pull request has failed checks 🛠

ChristopherHX · 2024-10-26T09:42:19Z

Added a test for the error caused by using the tart mapping in linux where it is not supported, now coverage is green.

We have a coverage collection problem for platforms outside of linux anyway.

ChristopherHX added 6 commits August 23, 2024 21:05

wip tart untested code

ec7f288

more tests

0be92a8

fixes

b751b59

fixes

e4d7c63

fixes

2941e25

lint fixes

6c9ee6b

ChristopherHX requested a review from a team as a code owner August 24, 2024 12:00

ChristopherHX linked an issue Aug 24, 2024 that may be closed by this pull request

Request for Support: Integration of Cirruslabs/Tart with Nektos/Act #2105

Open

pull-request-size bot added the size/XL label Aug 24, 2024

ChristopherHX marked this pull request as draft August 24, 2024 12:00

fixes

2b1fefe

ChristopherHX added 3 commits August 24, 2024 14:44

remove sample

538618a

remove offtopic changes

e18951c

remove references to original code

5512718

ChristopherHX marked this pull request as ready for review August 24, 2024 13:23

ChristopherHX commented Aug 24, 2024

View reviewed changes

ChristopherHX changed the title ~~Act tart~~ feat: tart macOS vm's as job ~container~ Aug 24, 2024

ChristopherHX changed the title ~~feat: tart macOS vm's as job ~container~~~ feat: tart macOS vm's as job ~~container~~ Aug 24, 2024

ChristopherHX changed the title ~~feat: tart macOS vm's as job ~~container~~~~ feat: tart macOS vm's as job container Aug 24, 2024

ChristopherHX added the area/image Relating to issues in the Docker image label Aug 24, 2024

ChristopherHX mentioned this pull request Aug 24, 2024

Does nektos/act support macOS VMs? #2431

Open

ChristopherHX commented Sep 5, 2024

View reviewed changes

pkg/tart/environment_darwin.go Show resolved Hide resolved

ChristopherHX added 2 commits September 28, 2024 00:07

cleanup vm on failed to start vm

22d5a80

Merge branch 'master' into act-tart

ae8e5ca

allexiusw reviewed Oct 19, 2024

View reviewed changes

pkg/runner/expression.go Outdated Show resolved Hide resolved

allexiusw reviewed Oct 19, 2024

View reviewed changes

pkg/runner/run_context_other.go

func (rc *RunContext) startTartEnvironment() common.Executor {

return func(_ context.Context) error {

return fmt.Errorf("You need macOS for tart")

This comment was marked as outdated.

Sign in to view

This comment was marked as outdated.

Sign in to view

allexiusw reviewed Oct 19, 2024

View reviewed changes

ChristopherHX added 2 commits October 20, 2024 11:51

revert unrelated change

175ea58

Merge branch 'master' into act-tart

01897b9

mergify bot added the needs-work Extra attention is needed label Oct 20, 2024

ChristopherHX added 2 commits October 20, 2024 15:30

Test tart only supported on macOS

ce5d63f

Update runner_test.go

bb077bc

fix test

706746e

ChristopherHX marked this pull request as draft October 26, 2024 09:32

ChristopherHX marked this pull request as ready for review October 26, 2024 09:40

mergify bot removed the needs-work Extra attention is needed label Oct 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: tart macOS vm's as job container #2434

feat: tart macOS vm's as job container #2434

ChristopherHX commented Aug 24, 2024 •

edited

Loading

github-actions bot commented Aug 24, 2024 •

edited

Loading

codecov bot commented Aug 24, 2024 •

edited

Loading

ChristopherHX Aug 24, 2024

ChristopherHX Aug 24, 2024

ChristopherHX Aug 24, 2024

ChristopherHX Aug 24, 2024

ChristopherHX Aug 24, 2024

ChristopherHX Aug 24, 2024

ChristopherHX Aug 24, 2024

ChristopherHX Aug 24, 2024

ChristopherHX Aug 24, 2024

YOU54F commented Sep 5, 2024 •

edited

Loading

ChristopherHX commented Sep 5, 2024

allexiusw Oct 19, 2024 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

allexiusw left a comment

ChristopherHX commented Oct 19, 2024 •

edited

Loading

mergify bot commented Oct 20, 2024

mergify bot commented Oct 20, 2024

ChristopherHX commented Oct 26, 2024

feat: tart macOS vm's as job container #2434

Are you sure you want to change the base?

feat: tart macOS vm's as job container #2434

Conversation

ChristopherHX commented Aug 24, 2024 • edited Loading

github-actions bot commented Aug 24, 2024 • edited Loading

🦙 MegaLinter status: ✅ SUCCESS

codecov bot commented Aug 24, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YOU54F commented Sep 5, 2024 • edited Loading

ChristopherHX commented Sep 5, 2024

allexiusw Oct 19, 2024 • edited Loading

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

allexiusw left a comment

Choose a reason for hiding this comment

ChristopherHX commented Oct 19, 2024 • edited Loading

mergify bot commented Oct 20, 2024

mergify bot commented Oct 20, 2024

ChristopherHX commented Oct 26, 2024

ChristopherHX commented Aug 24, 2024 •

edited

Loading

github-actions bot commented Aug 24, 2024 •

edited

Loading

codecov bot commented Aug 24, 2024 •

edited

Loading

YOU54F commented Sep 5, 2024 •

edited

Loading

allexiusw Oct 19, 2024 •

edited

Loading

ChristopherHX commented Oct 19, 2024 •

edited

Loading