[Feature] Support preserving only the Head pod with RayJob when shutdownAfterJobFinishes=true

[andrewsykim]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

When using RayJob with shutdownAfterJobFinishes=true, it can useful to keep the Head pod around to debug issues or access the Ray dashboard while deleting the workers that are using more expensive resources.

Use case

Allow access to the Ray dashboard after job finishes.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[andremoeller]

Similarly or alternatively: dumping state to persistent storage to allow resuming Dashboard after a Job has finished.

[MortalHappiness]

Should we add a flag to specify whether to preserve the head pod?

[MortalHappiness]

In this case, the RayCluster must be retained and cannot be deleted. There are two possible approaches:

1. Add a field to the RayCluster CR, and when this field is set to true, delete all workers.
2. Force all worker replicas (or max replicas) to be set to 0.

[andrewsykim]

I'm thinking of a new field like deletePolicy or cleanupPolicy, with options like DeleteCluster, DeleteWorkers, and None. But this would overlap with existing fields like shutdownAfterJobFinishes

[kevin85421]

This makes sense to me.

[andrewsykim]

Initial PR is here #2643

[kevin85421]

• [Umbrella] Suspend API for worker groups #2692