09-collaborative-data-science.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "71c7114a-89c5-498f-ae05-2fc82d04fa70",
   "metadata": {},
   "source": [
    "# Collaborative data science\n",
    "\n",
    "In this notebook, we'll learn about some ways to work on data science projects in a collaborative setting.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cbe5f6b-69ad-4297-8751-cabde78885f7",
   "metadata": {},
   "source": [
    "In the real world, we often work with a team of people on data science tasks. It's beneficial to quickly share your work with relevant people, and incorporate timely feedback.\n",
    "\n",
    "At minimum, you may want to share files (notebooks, scripts, etc.), environments, and dashboards. Let's look at one convenient way (we're biased!) to do this. :)\n",
    "\n",
    "Also, you have all been _collaborating_ through the Jitsi chat already!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc8862a6-3857-4bc5-a583-f78153026527",
   "metadata": {},
   "source": [
    "## Share files\n",
    "\n",
    "There is a `shared` folder present by default at the root of the file browser.\n",
    "\n",
    "Files and directories in this folder can be viewed by everyone in your \"group\", in our case the group has all the SciPy participants. :)\n",
    "\n",
    "### 💻 Your turn: Create a new text file with your name in the title and share it with the person next to you!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb0c491e-6807-427e-ab3d-5c1e64cda3a3",
   "metadata": {},
   "source": [
    "## Share environments\n",
    "\n",
    "We can use and share `environment.yaml` or `requirements.txt` files with team members, however environments created with these files aren't always reproducible, especially if library versions aren't pinned or if you make changes to your environment as you work.\n",
    "\n",
    "The safest way to ensure reproducibility is by pinning exact versions of libraries in the dependency tree, in conda you can do this with lockfiles. Here is the lockfile for the `data-of-unusual-size` environment we have been using today: https://scipy.quansight.dev/conda-store/api/v1/build/3/lockfile/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fea75666-a591-4a66-9ffd-e6e8a351bc95",
   "metadata": {},
   "source": [
    "### (Optional) Demonstrate environment creation\n",
    "\n",
    "If we have time, the instructors will demonstrate how to create and share a new environment with a tool called `conda-store`.\n",
    "\n",
    "Instructors, ensure the new environment has `numpy`, `pandas`, and `xarray`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a624d5a8-83ea-4700-9a15-702951ee44b9",
   "metadata": {},
   "source": [
    "### (Optional) 💻 Your turn: Change your kernel to use the newly created environment to run the following cell "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c7b1990-4504-4a19-a3de-3f9071b5dcbf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import xarray as xr\n",
    "\n",
    "data = xr.DataArray(np.random.randn(2, 3), dims=(\"x\", \"y\"), coords={\"x\": [10, 20]})\n",
    "data.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ece71639-a677-4889-95e7-6513a23cc2fa",
   "metadata": {},
   "source": [
    "## Share dashboards\n",
    "\n",
    "You can create interactive dashboards as web applications using libraries like Panel (as we saw earlier!), streamlit, viola, etc.\n",
    "\n",
    "To share your dashboards:\n",
    "\n",
    "1. Make sure they are \"servable\" (for Holoviz tools, there is a `.servable(` method)\n",
    "2. From JupyterLab, click on \"File\" in the menu bar and select the \"Hub Control Panel\"\n",
    "3. Click on \"Dashboards\" in the top menu and then click \"New Dashboard\"\n",
    "4. Select the appropriate file, framework, and environment and hit \"Save\"\n",
    "\n",
    "### 💻 (Optional) Your turn: Share `02-intro-to-hvplot` as a dashboard with the person next to you!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcb3500b-c990-4aad-a3fc-1e95c08ff730",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Data science platforms\n",
    "\n",
    "### JupyterHub\n",
    "\n",
    "A project by the Jupyter team that provides deploy-able Jupyter Notebook/Lab IDEs for collaboration.\n",
    "\n",
    "> JupyterHub brings the power of notebooks to groups of users. It gives users access to computational environments and resources without burdening the users with installation and maintenance tasks. Users - including students, researchers, and data scientists - can get their work done in their own workspaces on shared resources which can be managed efficiently by system administrators.\n",
    "> \n",
    "> ~ [Project Jupyer](https://jupyter.org/hub)\n",
    "\n",
    "### Nebari\n",
    "\n",
    "[Nebari](https://nebari.dev) is an open source project that makes deploying your own JupyterHub faster, removes the need for DevOps expertise, and ships with sane defaults (Dask, dashboard sharing, environment sharing, etc.). As you already guessed/know, we have been running this tutorial on Nebari.\n",
    "\n",
    "If you're considering a platform for your team, you can check out Nebari, as well as our friends in the ecosystem: [2i2c](https://2i2c.org/service/), [Anaconda](https://www.anaconda.com/products/enterprise), [Coiled](https://www.coiled.io/), [Domino Data Lab](https://www.dominodatalab.com/product/domino-enterprise-mlops-platform), [SaturnCloud](https://saturncloud.io/), and more!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d050ed1-197e-42d9-88aa-54863fef399f",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Next →\n",
    "\n",
    "[Conclusion](./10-conclusion.ipynb) :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "076c10a3-d2b2-4719-a9bf-c20acf8aca17",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "scipy-scipy-data-of-unusual-size",
   "language": "python",
   "name": "conda-env-scipy-scipy-data-of-unusual-size-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}