Environments

Files

In addition to the files common to all assets, the following are used for environments.

environment.yaml

An environment config file provides configuration specific to environments.

image:                          # Image configuration
  name: azureml/curated/my_env  # Name of the environment's Docker image
  os: linux                     # OS type, either linux or windows
  context:                      # Docker build context information
    dir: context                # Directory containing the build context, relative to the environment config file
    dockerfile: Dockerfile      # Dockerfile location, relative to the build context directory. Defaults to Dockerfile if unspecified.
    template_files:             # Optional list of files that contain template tags that should be resolved before building the image
    - Dockerfile                # Template tags in this file will be replaced
  publish:                      # Publishing settings for the image
    location: mcr               # Location to which the image will be published. Must be set to mcr.
    visibility: public          # Visibility of published image. Options are public, internal, staging, or unlisted.

Note that the image name will need to be registered with MCR before the environment is added to azureml-assets. Otherwise, the environment will fail to create in any registry. The image name should largely match the environment name and be prefixed with azureml/curated/.

Template tags

The following template tags are supported:

Tag	Normally used in	Resolves to	Example
{{latest-image-tag}}	Dockerfile	Either an image tag that with the same digest as latest, or the digest itself if a tag is not found	FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:{{latest-image-tag}}
{{latest-image-tag:<some_regex>}}	Dockerfile	The most recent (descending lexicographical sort) tag that matches the provided regular expression	FROM mcr.microsoft.com/azureml/aifx/stable-ubuntu2004-cu113-py38-torch1110:{{latest-image-tag:monthly\.\d{6}}}
{{latest-pypi-version}}	requirements.txt or conda_dependencies.yaml	The latest version of a package in PyPI	azureml-core=={{latest-pypi-version}}
{{latest-pypi-version:pre}}	requirements.txt or conda_dependencies.yaml	The latest version of a package in PyPI, including prerelease ones	azureml-core=={{latest-pypi-version:pre}}

Any of the above tags are resolved if they appear in any of the files listed in an environment config's template_files property.

spec.yaml

An Azure CLI v2 spec file. An environment example would look like this:

$schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json

description: >-
  A sample environment.

name: "{{asset.name}}"
version: "{{asset.version}}"

build:
  path: "{{asset.repo.url}}#{{asset.repo.commit_hash}}:{{asset.repo.build_context.path}}"
  dockerfile_path: "{{image.dockerfile.path}}"

os_type: linux

tags:
  PyTorch: "1.11"
  GPU: Cuda11
  OS: Ubuntu20.04
  Training: ""
  Preview: ""

Template tags

Spec files associated with assets in this repository may contain the following template tags:

Tag	Resolves to
{{asset.name}}	Name of the asset, from the asset config
{{asset.version}}	Version of the asset, from the asset config
{{asset.repo.build_context.path}}	Path to the environment's Docker build context in the release branch
{{asset.repo.commit_hash}}	Git commit hash of the commit being released
{{asset.repo.url}}	Git URL for this respository
{{image.name}}	Name of the environment's Docker image
{{image.context.path}}	Path to the Docker build context, relative to the spec file
{{image.dockerfile.path}}	Path to the Dockerfile, relative to the Docker build context root
{{image.publish.hostname}}	Hostname of the container registry to which the image will be published

Note that some of the tags above are common to all assets and some are environment specific.

Asset tags

The tags dictionary contains a list of asset tag names/values that are displayed in the UI. Empty values result in just the tag name being shown. Asset tags that apply to environments are:

Name	Example value	Description	Required
Cuda	11.1.1	CUDA version
CuDnn	8.0.5.39	cuDNN version
DeepSpeed	0.7.3	DeepSpeed version
Inferencing	<empty>	Supports AzureML inferencing
Nccl	2.8.4	NCCL version
OnnxRuntime	1.12	ONNX Runtime version
OpenMpi	4.1.0	Open MPI version
OS	Ubuntu20.04	Operating system name and version	X
Preview	<empty>	Public preview
Python	3.9	Python version	X
PyTorch	1.12	PyTorch version
ScikitLearn	1.1	Scikit-learn version
TensorFlow	2.8	TensorFlow version

Testing environment image builds

To test your environment image builds locally, you can do something like the following:

cd <build_context_dir>
docker build . --file <dockerfile>

This isn't going to work if you're using template tags (and you should). An approach that handles template tags is:

cd <azureml-assets_repo>
pip install -e scripts/azureml-assets
cp -r <build_context_dir> /tmp/context
scripts/azureml-assets/azureml/assets/environment/pin_versions.py -i /tmp/context/<dockerfile>
# Repeat the above step for other files you need to update
docker build /tmp/context --file <dockerfile>