KEP-2170: Add the manifests overlay for Kubeflow Training V2 #2382

Doris-xm · 2025-01-09T14:54:28Z

What this PR does / why we need it:

This PR adds the manifests overlay for Kubeflow Training V2, allowing to install it within Kubeflow Platform.

Which issue(s) this PR fixes :
Fixes #2381

Checklist:

Docs included if any changes are user facing

Electronic-Waste · 2025-01-09T15:25:45Z

Will review it later.

/cc @kubeflow/wg-training-leads @kubeflow/release-team @saileshd1402
/ok-to-test
/rerun-all

google-oss-prow · 2025-01-09T15:25:50Z

@Electronic-Waste: GitHub didn't allow me to request PR reviews from the following users: kubeflow/release-team, saileshd1402.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Will review it later.

/cc @kubeflow/wg-training-leads @kubeflow/release-team @saileshd1402
/ok-to-test
/rerun-all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Electronic-Waste

Thanks for creating this @Doris-xm! I left some initial comments for you.

Btw, I recommend that you could learn more about the concept of Training V2. This will help you update the manifests overlay in training-operator and kubeflow/manifests:)

FYI: KubeCon 2024 NA Talk by Andrey and Yuki

Electronic-Waste · 2025-01-10T01:49:25Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+    resources:
+      - trainjobs/status
+    verbs:
+      - get


Suggested change

- get

- get

Please add a blank line in the end of file

Thanks for your careful review！

Electronic-Waste · 2025-01-10T01:49:43Z

manifests/v2/overlays/kubeflow/kustomization.yaml

+  - name: training-operator-v2-webhook-cert
+    namespace: kubeflow-system
+    options:
+      disableNameSuffixHash: true


Suggested change

disableNameSuffixHash: true

disableNameSuffixHash: true

Same as above

Electronic-Waste · 2025-01-10T01:54:30Z

manifests/v2/overlays/kubeflow/kustomization.yaml

+kind: Kustomization
+namespace: kubeflow
+resources:
+  - ../../base


There does not exist a kustomization.yaml file under ../../base dir. So we may need to reference those resources in subdir explicitly.

Like:

training-operator/manifests/v2/overlays/only-manager/kustomization.yaml

Lines 5 to 8 in be2e29e

- ../../base/crds

- ../../base/manager

- ../../base/rbac

- ../../base/webhook

Electronic-Waste · 2025-01-10T02:00:54Z

manifests/v2/overlays/kubeflow/kustomization.yaml

+resources:
+  - ../../base
+  - kubeflow-training-roles.yaml
+  - https://github.com/kubernetes-sigs/jobset/releases/download/v0.6.0/manifests.yaml


Please add comments to make this item consistent with:

training-operator/manifests/v2/overlays/standalone/kustomization.yaml

Lines 10 to 11 in be2e29e

# TODO (andreyvelich): JobSet should support kubeflow-system namespace.

- https://github.com/kubernetes-sigs/jobset/releases/download/v0.6.0/manifests.yaml

I am wondering for Kubeflow Platform installation, should we actually deploy JobSet controller in the kubeflow namespace ?
@kannon92 also made it possible: kubernetes-sigs/jobset#719
cc @kubeflow/wg-manifests-leads

Electronic-Waste · 2025-01-10T02:04:23Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+metadata:
+  name: kubeflow-training-admin-v2
+  labels:
+    rbac.authorization.kubeflow.org/aggregate-to-kubeflow-admin: "true"


This label will conflict with:

training-operator/manifests/overlays/kubeflow/kubeflow-training-roles.yaml

Line 6 in be2e29e

rbac.authorization.kubeflow.org/aggregate-to-kubeflow-admin: "true"

Please rename it to rbac.authorization.kubeflow.org/aggregate-to-kubeflow-admin-v2 to avoid conflicts.

Electronic-Waste · 2025-01-10T02:04:36Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+aggregationRule:
+  clusterRoleSelectors:
+    - matchLabels:
+        rbac.authorization.kubeflow.org/aggregate-to-kubeflow-training-admin: "true"


Same as above.

Electronic-Waste · 2025-01-10T02:04:57Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+    rbac.authorization.kubeflow.org/aggregate-to-kubeflow-edit: "true"
+    rbac.authorization.kubeflow.org/aggregate-to-kubeflow-training-admin: "true"


Same as above.

Electronic-Waste · 2025-01-10T02:05:11Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+metadata:
+  name: kubeflow-training-view-v2
+  labels:
+    rbac.authorization.kubeflow.org/aggregate-to-kubeflow-view: "true"


Same as above.

Electronic-Waste · 2025-01-10T02:18:05Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+  - apiGroups:
+      - ""
+    resources:
+      - events
+    verbs:
+      - get
+      - list
+      - watch


I haven't seen any event resources in Training V2. Maybe we should remove them

training-operator/pkg/controller.v2/trainjob_controller.go

Lines 69 to 72 in be2e29e

// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs/status,verbs=get;update;patch

// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs/finalizers,verbs=get;update;patch

training-operator/pkg/controller.v1/pytorch/pytorchjob_controller.go

Lines 106 to 114 in be2e29e

// +kubebuilder:rbac:groups=kubeflow.org,resources=pytorchjobs,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups=kubeflow.org,resources=pytorchjobs/status,verbs=get;update;patch

// +kubebuilder:rbac:groups=kubeflow.org,resources=pytorchjobs/finalizers,verbs=update

// +kubebuilder:rbac:groups="",resources=pods,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;delete

// +kubebuilder:rbac:groups=autoscaling,resources=horizontalpodautoscalers,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups=scheduling.volcano.sh,resources=podgroups,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups=scheduling.x-k8s.io,resources=podgroups,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups="",resources=events,verbs=get;list;watch;create;update;patch;delete

PTAL👀 @kubeflow/wg-training-leads. Do you think we should reserve events resource in the manifests?

Thanks for your review! It seems event resources are required for EventRecorder used in TrainJobReconciler to log events related to training jobs.

training-operator/pkg/controller.v2/trainjob_controller.go

Lines 53 to 58 in be2e29e

type TrainJobReconciler struct {

log logr.Logger

client client.Client

recorder record.EventRecorder

runtimes map[string]jobruntimes.Runtime

}

However, the event resource permissions are not included in the kubebuilder annotations in the trainjob_controller.go

training-operator/pkg/controller.v2/trainjob_controller.go

Lines 69 to 71 in be2e29e

// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs/status,verbs=get;update;patch

// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs/finalizers,verbs=get;update;patch

I'm not sure I understand it correctly. Appreciate your clarification.

SGTM! However, in controller v2, we haven't called r.recorder.Eventf() yet. We just initialize it in:

training-operator/pkg/controller.v2/trainjob_controller.go

Lines 53 to 67 in be2e29e

type TrainJobReconciler struct {

log logr.Logger

client client.Client

recorder record.EventRecorder

runtimes map[string]jobruntimes.Runtime

}

func NewTrainJobReconciler(client client.Client, recorder record.EventRecorder, runtimes map[string]jobruntimes.Runtime) *TrainJobReconciler {

return &TrainJobReconciler{

log: ctrl.Log.WithName("trainjob-controller"),

client: client,

recorder: recorder,

runtimes: runtimes,

}

}

Maybe we could add events resource's permission to the Kubeflow Manifests when we call the function.

Actually, events will be required in the SDK, like for V1: https://github.com/kubeflow/training-operator/blob/master/sdk/python/kubeflow/training/api/training_client.py#L1309
We haven't implemented it yet in V2, but we will do it in the future.

Electronic-Waste · 2025-01-10T02:21:12Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+      - clustertrainingruntimes
+      - trainingruntimes


We could also edit trainingruntimes and clustertrainingruntimes in the Kubeflow platform. I think, their perssmission granularity should be the same with trainjobs:)

Electronic-Waste · 2025-01-10T02:30:03Z

/rerun-all

Electronic-Waste

Basically LGTM. Thank you for your great contributions!

/lgtm
/assign @kubeflow/wg-training-leads @kubeflow/wg-manifests-leads

andreyvelich

Thank you for this great contribution @willb!
/assign @kubeflow/wg-manifests-leads @kubeflow/release-team

andreyvelich · 2025-01-10T16:27:05Z

manifests/v2/overlays/kubeflow/kustomization.yaml

@@ -0,0 +1,20 @@
+apiVersion: kustomize.config.k8s.io/v1beta1


Can we name this overlay: kubeflow-platform to make it consistent with the naming: https://www.kubeflow.org/docs/started/introduction/#what-is-kubeflow-platform.

andreyvelich · 2025-01-10T16:28:31Z

manifests/v2/overlays/kubeflow/kustomization.yaml

+resources:
+  - ../../base
+  - kubeflow-training-roles.yaml
+  - https://github.com/kubernetes-sigs/jobset/releases/download/v0.6.0/manifests.yaml


I am wondering for Kubeflow Platform installation, should we actually deploy JobSet controller in the kubeflow namespace ?
@kannon92 also made it possible: kubernetes-sigs/jobset#719
cc @kubeflow/wg-manifests-leads

andreyvelich · 2025-01-10T16:30:40Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+  - apiGroups:
+      - ""
+    resources:
+      - events
+    verbs:
+      - get
+      - list
+      - watch


Actually, events will be required in the SDK, like for V1: https://github.com/kubeflow/training-operator/blob/master/sdk/python/kubeflow/training/api/training_client.py#L1309
We haven't implemented it yet in V2, but we will do it in the future.

Electronic-Waste

Basically LGTM!

Electronic-Waste · 2025-01-12T09:04:35Z

manifests/v2/overlays/kubeflow-platform/kustomization.yaml

@@ -0,0 +1,20 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: kubeflow


May I ask a question? Since we have defined the namespace for manager and webhook:

training-operator/manifests/v2/base/manager/kustomization.yaml

Lines 3 to 4 in 1dfa40c

# TODO (andreyvelich): Move it to overlays once we copy the JobSet manifests.

namespace: kubeflow-system

Does kubeflow namespace take any effect here? WDYT👀 @kubeflow/wg-training-leads @Doris-xm

Thanks for your reviewing. It seems that namespace: kubeflow in the overlay won't override namespace: kubeflow-system defined for manager and webhook. I'm wondering if we should unify the namespace as kubeflow when installing them within Kubeflow Platform, aligning with other modules.

I think, we should make our namespace configurable and use kubeflow namespace for Kubeflow Platform installation for now.
By default the Kubeflow Training V2 will use the kubeflow-system namespace.

@Doris-xm FYI, I think what @andreyvelich means is that we should address these TODO:

training-operator/manifests/v2/base/manager/kustomization.yaml

Lines 3 to 4 in 1dfa40c

# TODO (andreyvelich): Move it to overlays once we copy the JobSet manifests.

namespace: kubeflow-system

training-operator/manifests/v2/base/webhook/kustomization.yaml

Lines 13 to 14 in 1dfa40c

# TODO (andreyvelich): Move it to overlays once we copy the JobSet manifests.

namespace: kubeflow-system

And specify their namespace in overlays and kubeflow-platform dir. Because we need to make namesapce configurable and install these components in different namspace:

Standalone training-operator: Install all components in kubeflow-system namespace

Kubeflow Platform: Install all components in kubeflow namespace

Thanks for your detailed explanation! I've moved the namespace configuration from base to overlays. Appreciate your time!

Electronic-Waste

Thanks for your great contribution and your patience!

I think, we should also add namespace configuration here:

training-operator/manifests/v2/overlays/only-manager/kustomization.yaml

Lines 1 to 3 in 1dfa40c

    
           apiVersion: kustomize.config.k8s.io/v1beta1 
        
           kind: Kustomization 
        
           resources:

training-operator/manifests/v2/overlays/standalone/kustomization.yaml

Lines 1 to 3 in 1dfa40c

    
           apiVersion: kustomize.config.k8s.io/v1beta1 
        
           kind: Kustomization 
        
           resources:

to install these components in kubeflow-system namespace.

Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

…ace: kubeflow-system Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

Electronic-Waste

LGTM! Thanks for updating this:)

/lgtm
/assign @kubeflow/wg-training-leads @kubeflow/wg-manifests-leads @saileshd1402

google-oss-prow · 2025-01-13T07:22:23Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Electronic-Waste
Once this PR has been reviewed and has the lgtm label, please assign jeffwan for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andreyvelich · 2025-01-13T12:26:32Z

manifests/v2/base/manager/kustomization.yaml

-# TODO (andreyvelich): Move it to overlays once we copy the JobSet manifests.
-namespace: kubeflow-system


How this is going to work since we don't copy JobSet manifests in the Kubeflow Training ?

Currently, the JobSet manifest is referred from a remote URL, so the namespace configuration in the base or overlays cannot override the JobSet manifests, I think.

From the changlog, it seems that the latest release hasn’t included the changes in kubernetes-sigs/jobset/#719. We may need to manually modify the namespace setting for JobSet. I'm not sure if I need to deal with it in this pr.

@andreyvelich I tried it with:

$ kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/v0.7.2/manifests.yaml -n kubeflow-test namespace/jobset-system serverside-applied customresourcedefinition.apiextensions.k8s.io/jobsets.jobset.x-k8s.io serverside-applied clusterrole.rbac.authorization.k8s.io/jobset-manager-role serverside-applied clusterrole.rbac.authorization.k8s.io/jobset-metrics-reader serverside-applied clusterrole.rbac.authorization.k8s.io/jobset-proxy-role serverside-applied clusterrolebinding.rbac.authorization.k8s.io/jobset-manager-rolebinding serverside-applied clusterrolebinding.rbac.authorization.k8s.io/jobset-proxy-rolebinding serverside-applied mutatingwebhookconfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration serverside-applied validatingwebhookconfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration serverside-applied the namespace from the provided object "jobset-system" does not match the namespace "kubeflow-test". You must pass '--namespace=jobset-system' to perform this operation. the namespace from the provided object "jobset-system" does not match the namespace "kubeflow-test". You must pass '--namespace=jobset-system' to perform this operation. the namespace from the provided object "jobset-system" does not match the namespace "kubeflow-test". You must pass '--namespace=jobset-system' to perform this operation. the namespace from the provided object "jobset-system" does not match the namespace "kubeflow-test". You must pass '--namespace=jobset-system' to perform this operation. the namespace from the provided object "jobset-system" does not match the namespace "kubeflow-test". You must pass '--namespace=jobset-system' to perform this operation. the namespace from the provided object "jobset-system" does not match the namespace "kubeflow-test". You must pass '--namespace=jobset-system' to perform this operation. the namespace from the provided object "jobset-system" does not match the namespace "kubeflow-test". You must pass '--namespace=jobset-system' to perform this operation.

It does not work. Maybe we should modify the jobset manifests to make namespace configurable.

And just a question, do we really need to install all components in the same namespace?

Recently @kannon92 made a PR to support other namespaces in JobSet: kubernetes-sigs/jobset#719.
Probably, we should use kubeflow namespace to install the JobSet operator within Kubeflow Platform if @kubeflow/wg-training-leads or @kubeflow/wg-manifests-leads don't have any other suggestions.

However, this PR seems not to be included in the current release of jobset:

kubernetes-sigs/jobset@release-6.0...v0.7.2

In the manifest, the namespace is hardcoded into jobset-system：

https://github.com/kubernetes-sigs/jobset/releases/download/v0.7.2/manifests.yaml

SGTM. For me, I just don't know how to overlay namespace items outside of the .metadata field if we only specify namespace field in kustomization.yaml. Here is the example of which namespace field lies outside of the .metadata field:

We can use the var references, similar to this example. We will tell Kustomize how to override the namespace value.

SGTM. That's what I want to do with the JobSet manifests!

@Doris-xm So now we can wait for the next release of JobSet. And make some appropriate changes to the manifest after that:)

Thank you all for the insightful discussion! I’ve learned a lot from this. Much appreciated!

coveralls · 2025-01-20T12:05:01Z

Pull Request Test Coverage Report for Build 12742381626

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals
Change from base Build 12718184264:	0.0%
Covered Lines:	85
Relevant Lines:	85

💛 - Coveralls

google-oss-prow bot requested review from jinchihe and kuizhiqing January 9, 2025 14:54

google-oss-prow bot added the size/L label Jan 9, 2025

google-oss-prow bot requested a review from a team January 9, 2025 15:25

google-oss-prow bot added the ok-to-test label Jan 9, 2025

Electronic-Waste reviewed Jan 10, 2025

View reviewed changes

google-oss-prow bot added size/M and removed size/L labels Jan 10, 2025

Electronic-Waste approved these changes Jan 10, 2025

View reviewed changes

google-oss-prow bot assigned Electronic-Waste Jan 10, 2025

google-oss-prow bot added the lgtm label Jan 10, 2025

andreyvelich reviewed Jan 10, 2025

View reviewed changes

google-oss-prow bot added size/L and removed lgtm size/M labels Jan 12, 2025

Electronic-Waste reviewed Jan 12, 2025

View reviewed changes

Electronic-Waste reviewed Jan 13, 2025

View reviewed changes

Xinmin Du added 5 commits January 13, 2025 15:16

Add the manifests overlay for Kubeflow Training V2

a46294f

Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

Update manifest: adjust permissions, and format changes

963bfbc

Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

Update manifest: rename overlay, adjust event permissions

1100986

Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

Update manifest: make namespace configurable

a038ef2

Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

Update manifest: move standalone, only-manager installation in namesp…

1f1b0c2

…ace: kubeflow-system Signed-off-by: Xinmin Du <[email protected]> Signed-off-by: Xinmin Du <[email protected]>

Doris-xm force-pushed the add-overlay-manifest-v2 branch from 9518f7b to 1f1b0c2 Compare January 13, 2025 07:17

Electronic-Waste approved these changes Jan 13, 2025

View reviewed changes

google-oss-prow bot added the lgtm label Jan 13, 2025

andreyvelich reviewed Jan 13, 2025

View reviewed changes

andreyvelich changed the title ~~Add the manifests overlay for Kubeflow Training V2~~ KEP-2170: Add the manifests overlay for Kubeflow Training V2 Jan 14, 2025

Doris-xm mentioned this pull request Jan 15, 2025

Add Doris-xm as a member kubeflow/internal-acls#733

Merged

	- ../../base/crds
	- ../../base/manager
	- ../../base/rbac
	- ../../base/webhook

	# TODO (andreyvelich): JobSet should support kubeflow-system namespace.
	- https://github.com/kubernetes-sigs/jobset/releases/download/v0.6.0/manifests.yaml

		rbac.authorization.kubeflow.org/aggregate-to-kubeflow-edit: "true"
		rbac.authorization.kubeflow.org/aggregate-to-kubeflow-training-admin: "true"

	// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs,verbs=get;list;watch;create;update;patch;delete
	// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs/status,verbs=get;update;patch
	// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs/finalizers,verbs=get;update;patch

	type TrainJobReconciler struct {
	log logr.Logger
	client client.Client
	recorder record.EventRecorder
	runtimes map[string]jobruntimes.Runtime
	}

	# TODO (andreyvelich): Move it to overlays once we copy the JobSet manifests.
	namespace: kubeflow-system

	apiVersion: kustomize.config.k8s.io/v1beta1
	kind: Kustomization
	resources:

KEP-2170: Add the manifests overlay for Kubeflow Training V2 #2382

Are you sure you want to change the base?

KEP-2170: Add the manifests overlay for Kubeflow Training V2 #2382

Conversation

Doris-xm commented Jan 9, 2025

Electronic-Waste commented Jan 9, 2025

google-oss-prow bot commented Jan 9, 2025

Electronic-Waste left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Electronic-Waste Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Electronic-Waste commented Jan 10, 2025

Electronic-Waste left a comment

Choose a reason for hiding this comment

andreyvelich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Electronic-Waste left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Electronic-Waste left a comment

Choose a reason for hiding this comment

Electronic-Waste left a comment

Choose a reason for hiding this comment

google-oss-prow bot commented Jan 13, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Electronic-Waste Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Jan 20, 2025

Pull Request Test Coverage Report for Build 12742381626

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

Electronic-Waste Jan 10, 2025 •

edited

Loading

Electronic-Waste Jan 14, 2025 •

edited

Loading