Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Independent Vetex/Actor API #40

Open
ashione opened this issue Mar 28, 2022 · 2 comments
Open

Independent Vetex/Actor API #40

ashione opened this issue Mar 28, 2022 · 2 comments
Assignees

Comments

@ashione
Copy link
Member

ashione commented Mar 28, 2022

Mobius/Ray Streaming might need users create independent actors these out of streaming DAG.
In our prevois design, job master will invoke and manage all lifecyle of independent actors, such as Parameter Server actor that supported for training inside.

Actually, different training infra need user-define independent actors so engine dev should design a more general api for creating and managing independent actors/vertices.

ctx = context.build_context(context_config)
ctx.independent_vertex(module_name = 'xx', class_name = 'xx', number = 10, config = {}, resources = {}, lazy=false)

For exmplae, we define a parameter server group:

ctx.independent_vertex(module_name = "penrose.ps_actor", class_name = "PsActor", number  = 10)
@ashione
Copy link
Member Author

ashione commented Apr 1, 2022

@BalaBalaYi we talked offline that api could support linked-declaration likes

ctx.independent_vertex(module_name = "penrose.ps_acotr, class_name = "PsActor")
.with_config({"key" : "value"})
.with_resource({"CPU": 10 , "MEM" : 20})
.set_parallellism(10)
.set_language(PYTHON)
.set_mode(LAZY)

After definition, jobmaster will create 10 parameter servers after dag construction, which also mare use all of workers in DAG can attain ps actor handler then do what by themself.

@BalaBalaYi
Copy link
Collaborator

Still working on it. Will sync the impl to the repo after the internal release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants