training/neuron_parallel_compile/neuron_parallel_compile.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SageMaker Training Toolkit Support for neuron_parallel_compile\n",
    "\n",
    "This is a step by step guide on how to use ahead-of-time compilation to speed up SageMaker training jobs running on Amazon EC2 Trn1 (AWS Trainium) Instances by up to 10x using neuron_parallel_compile utility."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ML Frameworks such as PyTorch and Tensorflow, leverage compilers that take high-level descriptions of machine learning models, often in the form of computational graphs, and translate them into lower-level representations that can be efficiently executed on specific hardware architectures (ex. GPUs, TPUs). These optimizations may include parallelization, vectorization, and other techniques to make better use of the available hardware resources. Precompiling models in advance can result in training jobs running up to 10x faster.\n",
    "\n",
    "Trn1 (AWS Trainium) EC2 instances leverage the Neuron SDK, the software stack that includes the Neuron hardware driver, user tools, framework integration, and compiler. Before you are able to train your model on Trn1 (AWS Trainium) EC2 instances you must leverage the Neuron compiler to complete a compilation step which converts your model from the standard ML framework-level model to a Neuron Executable File Format (NEFF) binary. The Neuron Compiler accepts Machine Learning models in various formats (TensorFlow, MXNet, PyTorch, XLA HLO) and optimizes them to run on Neuron devices. To provide some benchmarks, the compilation process for Llama2 7B with a parallel compilation of 16 nodes using trn1.32xlarge for sequences of 4k tokens takes approximately 3 minutes, and with 4 nodes using trn1.32xlarge , it takes around 5 minutes. \n",
    "\n",
    "\n",
    "The Neuron compiler has 2 compilation methods:\n",
    "\n",
    "* just-in-time (JIT) compilation (default)\n",
    "* ahead-of-time compilation with neuron_parallel_compile\n",
    "\n",
    "PyTorch Neuron defaults to just-in-time (JIT) compilation of graphs during execution, this is where at every step, a graph is traced. If the traced graph varies from the previous executions, it is compiled by the neuron compiler. JIT compilation can be helpful to speed up developer workflow, however when using JIT, graphs are compiled sequentially which can lead to much longer compilation times than compared to neuron_parallel_compile. \n",
    "\n",
    "To reduce this compilation time during execution, the neuron_parallel_compile utility is provided as part of PyTorch Neuron installation. The neuron_parallel_compile utility will extract graphs from a trial run of your script, perform parallel pre-compilation of the graphs, and populate the Neuron Cache on disk or AWS S3 URL location with the compiled graphs. This pre-compilation run should be limited to a few training steps (eg. <100), enough for the utility to explore all code branches present in your training script to extract the different graphs needed for full execution. Once, the neuron_parallel_compile finishes compilation of all graphs, it will copy all the compilation results into the Neuron Cache directory (which can be a specified S3 location). Therefore you are then able to specify the location of the Neuron Cache directory in subsequent training runs so that the precompiled graphs will be used - avoiding recompilation.\n",
    " \n",
    "With recent versions of the Neuron Deep Learning Containers (DLCs), you are now able to leverage neuron_parallel_compile utility with SageMaker training jobs by setting the `RUN_NEURON_PARALLEL_COMPILE = \"1\"` environment variable within the SageMaker Estimator class. \n",
    "\n",
    "### The specific steps to enable this workflow would be the following:\n",
    "\n",
    "1. Ahead-of-time compilation SageMaker training run:<br>\n",
    "    1. The 1st training run will use ahead-of-time compilation by setting the RUN_NEURON_PARALLEL_COMPILE = \"1\" environment variable in the SageMaker Estimator class. Please note the values that are outputted from the training script when using neuron_parallel_compile are placeholder values and should be disregarded (ex: loss_value=0, etc.).<br>\n",
    "    \n",
    "    2. You will also specific an S3 URL location to store your Neuron Persistant Cache files by using the NEURON_COMPILE_CACHE_URL environment variable in the SageMaker Estimator class. The Neuron SDK will check the specified S3 location for available Neuron Persistent Cache files as well as upload Neuron Persistent Cache files once the training job is complete.<br>\n",
    "    \n",
    "    3. To minimize the number of training steps you can set the max_steps hyperparameters to <100 steps. You want to ensure you set the max_steps to a minimum number of steps that are enough for the neuron_parallel_compile utility to explore and extract the different graphs needed for full execution. In most cases 100 steps would suffice.<br>\n",
    "    \n",
    "2. Subsequent SageMaker training runs leveraging the precompiled neuron persistent cache files located in S3:<br>\n",
    "\n",
    "    1. You would run subsequent training jobs without setting the RUN_NEURON_PARALLEL_COMPILE = \"1\" environment variable but still making sure to pass in the NEURON_COMPILE_CACHE_URL. The Neuron SDK will check the specified S3 URL location and download the Neuron Persistent Cache which will have all of the precompiled graphs to be used for subsequent training runs. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "_*Note:* we only install the required libraries from Hugging Face and AWS. You also need PyTorch or Tensorflow, if you haven´t it installed_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install \"sagemaker>=2.48.0\"  --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Permissions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "sess = sagemaker.Session()\n",
    "# sagemaker session bucket -> used for uploading data, models and logs\n",
    "# sagemaker will automatically create this bucket if it not exists\n",
    "sagemaker_session_bucket=None\n",
    "if sagemaker_session_bucket is None and sess is not None:\n",
    "    # set to default bucket if a bucket name is not given\n",
    "    sagemaker_session_bucket = sess.default_bucket()\n",
    "\n",
    "try:\n",
    "    role = sagemaker.get_execution_role()\n",
    "except ValueError:\n",
    "    iam = boto3.client('iam')\n",
    "    role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']\n",
    "\n",
    "sess = sagemaker.Session(default_bucket=sagemaker_session_bucket)\n",
    "\n",
    "print(f\"sagemaker role arn: {role}\")\n",
    "print(f\"sagemaker bucket: {sess.default_bucket()}\")\n",
    "print(f\"sagemaker session region: {sess.boto_region_name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Set the parameters for the training job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_name = \"roberta-base\"\n",
    "env_var_options = \"\"\n",
    "task_name = \"mrpc\"\n",
    "batch_size = 8\n",
    "max_seq_length = 128\n",
    "learning_rate = 2e-05\n",
    "num_train_epochs = 1\n",
    "model_base_name = model_name\n",
    "max_train_samples = 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating an Estimator and start a training job with Neuron_Parallel_Compile\n",
    "\n",
    "In this example we will use the run_glue.py training script provided by Hugging Face library in this github repo [here](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.pytorch import PyTorch\n",
    "\n",
    "# hyperparameters, which are passed into the training job\n",
    "hyperparameters={\n",
    "    'model_name_or_path': model_name,\n",
    "    'task_name': task_name,\n",
    "    'do_train': True,\n",
    "    # 'do_eval': True,\n",
    "    'max_seq_length': max_seq_length,\n",
    "    'per_device_train_batch_size': batch_size,\n",
    "    'learning_rate': learning_rate,\n",
    "    'max_train_samples': max_train_samples,\n",
    "    'num_train_epochs': num_train_epochs,\n",
    "    'max_steps': 200,\n",
    "    'output_dir': '/opt/ml/model',\n",
    "}\n",
    "\n",
    "# configuration for running training on smdistributed Data Parallel\n",
    "distribution={\"torch_distributed\": {\"enabled\": True} }\n",
    "\n",
    "# instance configurations\n",
    "instance_type='ml.trn1.32xlarge'\n",
    "instance_count=2\n",
    "volume_size=450\n",
    "\n",
    "# metric definition to extract the results\n",
    "metric_definitions=[\n",
    "     {\"Name\": \"train_runtime\", \"Regex\": \"train_runtime.*=\\D*(.*?)$\"},\n",
    "     {'Name': 'train_samples_per_second', 'Regex': \"train_samples_per_second.*=\\D*(.*?)$\"},\n",
    "     {'Name': 'epoch', 'Regex': \"epoch.*=\\D*(.*?)$\"},\n",
    "     {'Name': 'f1', 'Regex': \"f1.*=\\D*(.*?)$\"},\n",
    "     {'Name': 'exact_match', 'Regex': \"exact_match.*=\\D*(.*?)$\"}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Specify the neuronx container image\n",
    "training_image = \"763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-training-neuronx:1.13.1-neuronx-py310-sdk2.15.0-ubuntu20.04\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Using ahead-of-time compilation by setting RUN_NEURON_PARALLEL_COMPILE == \"1\" environment variable\n",
    "# Please note the values that are outputted from the training script when using neuron_parallel_compile \n",
    "# are placeholder values and should be disregarded (ex: loss_value=0, etc.). \n",
    "\n",
    "pytorch_estimator = PyTorch(\n",
    "    entry_point='run_glue.py', #training script file\n",
    "    source_dir='./', #source directory for your training script \n",
    "    metric_definitions=metric_definitions, #A list of dictionaries that defines the metric(s) used to evaluate the training jobs.\n",
    "    instance_type=instance_type, #type of EC2 instance to use for training\n",
    "    instance_count=instance_count, #number of Amazon EC2 instances to use for training\n",
    "    volume_size=volume_size, #size in GB of the storage volume to use for storing input and output data during training\n",
    "    role=role, #An AWS IAM role to be used by Amazon SageMaker training jobs and APIs \n",
    "    transformers_version='4.31.0', #specifies version of the transformer library to be used\n",
    "    pytorch_version='1.13.1', #specifies version of the PyTorch library to be used\n",
    "    py_version='py39', #specifies version of Python to be used\n",
    "    distribution=distribution, #configuration for running training on smdistributed Data Parallel\n",
    "    image_uri=training_image, #specifies the Docker image to use for training\n",
    "    environment={\n",
    "        \"RUN_NEURON_PARALLEL_COMPILE\": \"1\", # runs neuron precompile step if equal to \"1\"\n",
    "        \"FI_EFA_FORK_SAFE\": \"1\", #Older Linux (<5.15) kernels require environment variable FI_EFA_FORK_SAFE to be set to 1 for the libfabric to operate correctly.\n",
    "        \"NEURON_COMPILE_CACHE_URL\": f\"s3://{sess.default_bucket()}/s3-neuron-cache\" #specifies s3 path location to store neuron persistent cache\n",
    "    },\n",
    "    hyperparameters = hyperparameters) #hyperparameters to be passed into the training script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# starting the train job\n",
    "pytorch_estimator.fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rerun Training Job Without Neuron_Parallel_Compile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Subsequent training run without setting RUN_NEURON_PARALLEL_COMPILE == \"1\" environment variable\n",
    "#Please note that you will need to rerun the above compilation step if you change the training script or the model.\n",
    "pytorch_estimator = PyTorch(\n",
    "    entry_point='run_glue.py', #training script file\n",
    "    source_dir='./', #source directory for your training script \n",
    "    metric_definitions=metric_definitions, #A list of dictionaries that defines the metric(s) used to evaluate the training jobs.\n",
    "    instance_type=instance_type, #type of EC2 instance to use for training\n",
    "    instance_count=instance_count, #number of Amazon EC2 instances to use for training\n",
    "    volume_size=volume_size, #size in GB of the storage volume to use for storing input and output data during training\n",
    "    role=role, #An AWS IAM role to be used by Amazon SageMaker training jobs and APIs \n",
    "    transformers_version='4.31.0', #specifies version of the transformer library to be used\n",
    "    pytorch_version='1.13.1', #specifies version of the PyTorch library to be used\n",
    "    py_version='py39', #specifies version of Python to be used\n",
    "    distribution=distribution, #configuration for running training on smdistributed Data Parallel\n",
    "    image_uri=training_image, #specifies the Docker image to use for training\n",
    "    environment={\n",
    "        \"FI_EFA_FORK_SAFE\": \"1\", #Older Linux (<5.15) kernels require environment variable FI_EFA_FORK_SAFE to be set to 1 for the libfabric to operate correctly.\n",
    "        \"NEURON_COMPILE_CACHE_URL\": f\"s3://{sess.default_bucket()}/s3-neuron-cache\" #specifies s3 path location to store neuron persistent cache\n",
    "    },\n",
    "    hyperparameters = hyperparameters) #hyperparameters to be passed into the training script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "pytorch_estimator.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "interpreter": {
   "hash": "c281c456f1b8161c8906f4af2c08ed2c40c50136979eaae69688b01f70e9f4a9"
  },
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}