[Initiative] Ephemeral sessions

[yuvipanda]

(This was manually migrated from https://github.com/2i2c-org/meta/issues/1105, since that was private and we want this to be public)

This was refined by me, @Gman0909 and @jmunroe. ProductBoard link.

Description

Provide the ability to spin up MyBinder-like ephemeral hubs to allow authenticated and unauthenticated users (based on configuration) access to the contents of an externally published artifact (GitHub repo, Zenodo, etc) animated via an interactive computing session.

User personas

2i2c engineering: sets up the ephemeral session functionality for a community.
Admin: member of the community, sets up permissions globally, such as whether users are allowed to share to unauthenticated individuals, and whether to limit sharing to GitHub repos controlled by their organization. They will also set resource requests and limitations.
Content creator/Sharer: has to be able to package a snapshot of their work to be shared as a link, including all necessary environment configurations to ensure the work can be reproduced and experienced in its full form by the recipient of the link. If the Admin has made unauthenticated access to shared links available, the sharer should have that option as a toggle when creating that link.
Reader: receives a link, clicks it, and is brought into an interactive session that reproduces the work shared by the Content Creator/Sharer

Desired features

2i2c engineering

Everything we build should be upstreamable as much as possible

Admin

• Can determine users and user groups who can have access to the infrastructure
  • Includes ability to provide fully unauthenticated users
• Can determine what kinda content can be run on the ephemeral hubs
  • Restricted by binderhub provider specs - (eg. GitHub orgs / list of repos, zenodo DOIs, etc)
• Basic usage monitoring dashboard
• Can decide to deploy or not deploy dask-gateway

Implementation note: All admin actions will have to currently go through 2i2c support.

Creator

• Ability to create a link for sharing their work
• Specify the environment their work needs
• Specify the content of their work, if it's separate from where the environment is hosted
• Ability to select what kinda resources the Reader receives when they click the link

Reader

• Receives a link, clicks it, and is brought into an interactive session that reproduces the work shared by the Content Creator/Sharer

Why is it important

Compliant with our Value Proposition's support of creating and sharing knowledge, we want to empower our communities to more easily permit access to their works, in the form of interactive computing sessions, to authorized and unauthorized users, with zero-setup on the part of the recipient of that shared work. This is also a commonly requested feature we have committed to delivering to the likes of NASA HPOSS and Project Pythia

Measures and Definition of Done

• Communities are able to easily share work to authorized and unauthorized users via a link leading to a zero-setup interactive compute session containing said work.

Tasks

Beta Give feedback

1. Remove the binderhub chart from our infrastructure #4085
   +0
   
2. [Spike] Determine what needs to happen for binderhub-service to expose the binderhub UI #4086
   +0
   

   
3. [New Hub] binderhub-ui-demo.2i2c.cloud #4133
   +0
   
4. Decide on the 'home page' flow when deploying ephemeral hubs with binderhub-service showing the binder-ui #4168
   +0
5. Provide a "clean" logged out front page for binderhub-ui-demo #4197
   +0
   
6. Write a runbook for spinning up an ephemeral hub that can launch sessions #4169
   +0
   
7. Handle healthchecks hubs with binderhub-service set up to launch #4167
   3 of 3
   +0
   
8. Allow disabling all subsections of hub home page #4195
   +0
   
9. [New Hub] binder.pythia.2i2c.cloud #4134
   +0
   
10. Explicitly define scope of binderhub-service helm chart binderhub-service#95
    +0
    
11. Explicitly determine when jupyterhub-fancy-profiles should be used vs the binderhub UI yuvipanda/jupyterhub-fancy-profiles#37
    +0
    
12. Coalesce Daskhub into basehub #4242
    4 of 4
    +0
    
13. Run a knowledge sharing session about binderhub technical architecture #4113
    +0
    
14. [Spike] Determine what is needed to make dask-gateway a toggleable option in basehub #4087
    +0
    
15. Do not put name of release in resource names by default binderhub-service#57
    +0
    
16. [New Hub] [Deadline July 7] HPOSS 'Small' Binder #4246
    +0
    
17. [New Hub] [Deadline: July 7] HPOSS 'Big' Binder #4247
    +0
    
18. Enable dask-gateway on opensci big binder #4301
    +0
19. [Spike] [90min *max*] Understand what is needed to enable eventlog aggregation for hubs with BinderHub UI #4309
    +0
    
20. Design a simple system where users can ask for access to HPOSS Binders #4310
    +0
    
21. [EPIC] Collect analytics information from all our binderhub ui installations #4365
    5 of 5
    +0
22. Bring latest version of binderhub-service chart into our repository #4370
    +0
    
23. Copy mybinder.org's build resource limits to all our binderhub-service deployments #4371
    +0
    
24. Spin up binder build pods on appropriately sized nodes #4241
    +0
    
25. Retain binderhub analytics events #4423
    +0
    

[yuvipanda]

Porting some relevant comments from the internal issue

From @yuvipanda, at https://github.com/2i2c-org/meta/issues/1105#issuecomment-2111573907

I had a nice and productive scoping session with @consideRatio today. I'll create an epic in the infrastructure repository later, but wanted to dump the first set of tasks we have scoped out so far:

1. Remove all of binderhub chart
2. Spike to figure out what is needed in binderhub-service to expose classical binderhub UI including ability to launch in a way that makes it composable. Constraint is that binderhub-service should not know what is jupyterhub, but can have extensibility that is generic added to it. Outcome should primarily be: documentation changes, and possibly extra extensibility in binderhub-service.
3. Explicitly specify constraints for use of fancy-profiles vs binderhub UI. Persistent home directory is an important one here.
4. Explicitly specify scope for binderhub-service.
5. Spike for figuring out how dask-gateway can be an optional add on in basehub rather than daskhub.

These all need to be cleaned up and expanded, a task for tomorrow.

---

From @yuvipanda in https://github.com/2i2c-org/meta/issues/1105#issuecomment-2113090855

I've updated the issue body with the tasks from https://github.com/2i2c-org/meta/issues/1105#issuecomment-2111573907. They're also prioritized in the order I believe they should be done. This should allow us to make progress on a lot of these goals! While I don't believe we'll have a full deployment once these are done, it would allow us to spec out a 'new hub' issue that would allow us to get there by the end of the sprint I believe.

The one thing that's currently explicitly out of scope is 'resource selection' and 'GPU' use, which are tied together. This would involve reviving these upstream conversations:

• Select pod resources from binder UI jupyterhub/binderhub#731
• Query parameter naming scheme? jupyterhub/binderhub#712
• Custom resource requests jupyterhub/binderhub#759

---

From @Gman0909 in https://github.com/2i2c-org/meta/issues/1105#issuecomment-2114375304

Thanks Yuvi. Given the likely time lag that is involved in any upstream involvement, should we not prioritize getting some of these conversations going?

Also, if we expect that the current task list will lead to the speccing out of a new issue that will allow us to meet the DoD, I feel we should record that in the task list, even as a draft if need be.

[yuvipanda]

Me and @jmunroe have refined #4133 to be the first demo hub to be deployed with this functionality, and we believe it's fully refined now. Our goal was to make sure that if we can deploy this, we can deploy what we promised project pythia. I've prioritized it above our dask-gateway work.

[yuvipanda]

I have spoken with @jmunroe and @Gman0909, and we're going to split off the following tasks into their own initiative:

• [EPIC] Allow selection of resources via URL parameters  #4151 (this is not currently required by the three grants that drive this inititative)
• Shepherd a cleanup of the BinderHub UI #4152 (this shall be moved to be a part of completing our GESIS work as well as upcoming NASA VEDA work)

[yuvipanda]

#4246 and #4247 need further refinement, and I'd like them to be done by someone who wasn't the original set of people working on this.

[Gman0909]

Thanks @yuvipanda. I'm assuming that, funding-centered priorities notwithstanding, the plan is to get these two done contiguously to the rest of the work on Ephemeral Hubs?

[GeorgianaElena]

and I'd like them to be done by someone who wasn't the original set of people working on this.

@yuvipanda, the second iterartion of https://infrastructure.2i2c.org/howto/features/binderhub-ui/ is now ready so this should unblock other people doing those deployments

[yuvipanda]

@Gman0909 yes. I've spoken to @sgibson91 and she'll pick those up (after a session as part of #4113)

[yuvipanda]

#3655 is a big milestone, as it's now counted as 'tech support' than product dev. Upon completion of that without issues, I'd say we can then proceed to add this to our single source of truth about the kind of hubs we can deploy! I'll work with @Gman0909 on that next week.

[yuvipanda]

I don't think there are any more tasks to do for this initiative - once the two remaining ones are completed, we are good to go! We need to add this to the canonical list of things we can provide, and then mark it as done! \o/

[yuvipanda]

All tasks are complete here.

I've added BinderHub UI to https://docs.google.com/spreadsheets/d/1qy_ozom5QBSiDDedzmxHUsBWlEot44gl3qPWfvMYiEA/edit?gid=0#gid=0 which IIRC is our 'canonical' location for various components we 'support'.

With that, I decree this initiative completed!

Congratulations to all of us :)