[Umbrella] Autoscaler improvements

[kevin85421]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

TODO:

• Define the scope of Autoscaler V2 beta
• List the important issues need to solve (V1 & V2).

This umbrella issue covers two topics:

• Autoscaler V2 towards beta
• Autoscaler stability improvements (V1 + V2)

Reliability

Top priority:

• Autoscaler should not terminate worker Pods with running Actor / Task
• Autoscaler should not crash because of CR spec
• A Job should be able to finish.

• fix token expiration for ray autoscaler ray#48481
• [Ray Core] The node storing the actor will be kill unexpectedly when autoscaler is turned on ray#46172
• [Bug] autoscaler crash when resources.limt not set #2612
• [Autoscaler] Issue with RayCluster autoscaler and hanging non existent tasks ray#48950
• [core][autoscaler] Autoscaler V2 - cluster state version number  ray#35873
• [core][autoscaler][v1] Autoscaler overprovisions nodes when strict placement group is rescheduling ray#40212
• [Feature] Don't submit scale requests if the worker group is suspended #2666 (relies on KubeRay v1.3)
• [Bug] Autoscaler sideacr crashes, bringing down head pod, if request exceeds max pod replicas #2385
• [core][autoscaler] Autoscaler doesn't scale up correctly when the KubeRay RayCluster is not in the goal state ray#48909
• [core][autoscaler] Fuse scaling requests together to avoid overloading the Kubernetes API server ray#49150
• [core][autoscaler] Set timeout for requests sent to K8s API server ray#48513
• [core][autoscaler] Fix incorrectly terminating nodes misclassified as idle in autoscaler v1 ray#48519
• [core][autoscaler] Autoscaler v2 does not honor minReplicas/replicas count of the worker nodes and constantly terminates after idletimeout  ray#48623
• [Core]: Fix ConnectionError on Autoscaler CR lookups in K8s clusters with custom DNS for Kubernetes API. ray#48541

Usability

• [core][autoscaler] Make some Autoscaler configurations configurable ray#49200
• [Core][Autoscaler] Configure idleTimeoutSeconds per node type ray#48813
• [Core][Autoscaler] API for getting max cluster resources ray#49501

Testing

• [Umbrella] Ray Autoscaling tests #2173

Observability / Debuggability

• [core][autoscaler] Health check logs are not visible in the autoscaler container's stdout ray#48905
• De-noise autoscaler logs. Currently, the autoscaler loops with Fetched pod data, outputting the state of the RayCluster even with no changes to the resources requested or allocated. This can make it fairly difficult to debug autoscaler logs. It'd be useful to provide the option to output relevant logs only on Autoscaler updates. (issue to be created) (@ryanaoleary)
• [core][autoscaler] Better observability for request resources  ray#37959 (@ryanaoleary)
• [autoscaler] Refactor ray status output code ray#37856 (@ryanaoleary)
• [autoscaler][core][v2] Support all clouds for instance type name from config  ray#39735 (@ryanaoleary)

Others

• [core][autoscaler] Remove unused deprecated flags for AutoscalerMonitor ray#48919
• [core][autoscaler] Make the head group name consistent with the head Pod's ray.io/group label. ray#48840
• [core][autoscaler] Remove local dependencies on Ray for KubeRay autoscaling e2e tests ray#48566
• Update doc for v2 if v2 is used by default

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[kevin85421]

/assign @ryanaoleary

[ryanaoleary]

TODO: I'll leave a comment outlining the V2 beta scope and remaining issues to solve for V1 & V2.

[kevin85421]

@ryanaoleary thanks! You can compile a list of issues, and we can schedule a meeting to go through them one by one.

[ryanaoleary]

The issues that I think we ought to complete before considering Autoscaler v2 in Beta can be broken down into observability improvements and reliability bug-fixes.

Observability:

• Provide an API to persist worker logs after autoscaler scale-down, this is covered in the export API proposal (this may fall outside the scope of KubeRay v1.3)
• De-noise autoscaler logs. Currently, the autoscaler loops with Fetched pod data, outputting the state of the RayCluster even with no changes to the resources requested or allocated. This can make it fairly difficult to debug autoscaler logs. It'd be useful to provide the option to output relevant logs only on Autoscaler updates. (issue to be created)
• Direct formatting to output ray status: [autoscaler] Refactor ray status output code ray#37856
• Refactor request resources in autoscaler logs: [core][autoscaler] Better observability for request resources  ray#37959
• Generate node IDs: [autoscaler] Autoscaler should avoid using ip address to identify nodes (as node id of node provider) ray#19086
• Expose API to expose autoscaler state: [autoscaler] Able to inspect autoscaler state/cluster config ray#37838

Reliability:

• Maintain correctness of cluster state version: [core][autoscaler] Autoscaler V2 - cluster state version number  ray#35873
• Reserve custom accelerators: [core][autoscaler][v2] Reserve nodes with accelerator nodes.  ray#43079
• Further v2 autoscaler tests in Ray Core and e2e v2 Autoscaler tests using the latest Ray image in KubeRay.
• [autoscaler][core][v2] Support all clouds for instance type name from config  ray#39735

Several of the completed issues mentioned in the issue description fix the main reliability issues found within the v1 Autoscaler, and from my manual testing with multiple CPU and GPU worker-groups I've seen consistent behavior. Additionally, new features in the v2 autoscaler such as configuring idle node timeouts by node type will enable users with more fine-grained control of their workloads and minimize the amount of autoscaling errors we were previously seeing. We should also consider it a requirement to ensure reliable testing for CPUs, GPUs, and custom accelerators before considering v2 beta.

[kevin85421]

TODO: confirm the following two issues.

• [autoscaler] Autoscaler should avoid using ip address to identify nodes (as node id of node provider) ray#19086
• [core][autoscaler][v2] Reserve nodes with accelerator nodes.  ray#43079