Initial Commit

.dockerignore

@@ -0,0 +1,2 @@
+config
+*.sif

.gitattributes

@@ -0,0 +1,2 @@
+# Auto detect text files and perform LF normalization
+* text=auto

.gitignore

@@ -0,0 +1,131 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# PyCharm
+.idea/
+
+# do not ignore any files in doc
+!doc/**

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 Scientific Machine Learning (SciML) Group (STFC, RAL)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

MANIFEST.in

@@ -0,0 +1 @@
+recursive-include sciml_bench *

README.md

@@ -0,0 +1,121 @@
+<img src="./doc/resources/logo.png" alt="logo" width="500"/>
+
+
+
+# SciML-Bench: Table of Contents
+- [1. Synopsis](#1-synopsis)
+- [2. Benchmark Suite](#2-benchmark-suite)
+  * [2.1 Organisation](#21-organisation)
+  * [2.2 Features](#22-features)
+  * [2.3 Benchmarks and Datasets](#23-benchmarks-and-datasets)
+- [3. Installation and Usage](#3-installation-and-usage)
+- [4. Citation](#4-citation)
+- [5. Acknowledgments](#5-acknowledgments)
+
+
+
+
+# 1. Synopsis
+
+Benchmarking is a well known topic of Computer Science, primarily aimed at comparing software and/or hardware systems.  With the rise of machine learning and AI, the overall efforts on AI benchmarking in constantly increasing. The SciMLBench is aimed at the AI for Science domain, as opposed to generic AI benchmarking.  
+
+With an ever growing number of machine learning models and algorithms, a range of scientific problems, and the proliferation of AI systems, developing cutting-edge ML/AI algorithms requires a detailed understanding of the interactions between these aspects. 
+
+The SciMLBench is aimed at providing an answer to this open question. It is an open-source initiative, and covers a range of scientific problems from various domains of science, including  material, life, and earth sciences, particle physics and astronomy.  The benchmarks are implemented in  Python, relying on one or more  machine learning frameworks, such as TensorFlow, PyTorch or SciKit-Learn. 
+
+The overarching purpose of this initiative is many-fold, including, supporting benchmarking of machine learning models, AI systems, and supporting the AI for Science community for developing better solutions.
+
+
+
+# 2. Benchmark Suite 
+
+
+
+## 2.1 Organisation
+
+The suite has three components, namely, 
+
+1. **Benchmarks**: The benchmarks are machine learning applications performing a specific scientific task, written in Python. These are included as part of this package, and can be found inside the ``./sciml_bench/benchmarks`` directory. In the scale of *micro-apps*, *mini-apps*, and *apps*, these are full-fledged applications. 
+
+2. **Datasets**: Each benchmark in (1) relies on one or more datasets, for example for training and/or inferencing. These datasets are open, task- or domain-specific, and FAIR compliant. Most of these datasets being large, they are hosted separately,  on one of the servers (or mirrors), and are automatically or explicitly downloaded on demand. The framework (see (3)), supports manual downloading of these datasets. 
+
+3. **Framework**:  The framework serves two purposes: first, at the user level, it facilitates an easier approach to benchmarking, logging and reporting of the results. Secondly, at the developer level, it provides a coherent API for unifying and simplifying the development of AI benchmarks. This can be found in ``./sciml_bench/core`` directory. 
+
+The source tree, which captures these aspects,  is organised as follows:
+
+```bash
+├── README.md                  <This file>
+├── config                     <Container configuration files>
+│   ├── Dockerfile
+│   ├── ompi4.def
+│   └── sciml-bench.def
+├── doc                        <User documentation>
+│   ├── benchmarks_datasets.md <List of benchmarks / datasets>
+│   ├── contributing.md        <How to contribute>
+│   ├── credits.md
+│   ├── demo_output__MNIST_torch
+│   │   └──                    <Sample benchmark outputs> 
+│   ├── resources
+│   │   └──                    <Various resources> 
+│   └── usage.md               <Usage documentation> 
+├── requirements.txt
+├── sciml_bench
+    ├── benchmarks             <Benchmark sources> 
+    │   ├── registration.yml   <Key registration file>
+    │   └── template
+    │       └── template.py    <Benchmark Template>
+    ├── core                   <Core scripts> 
+    │   ├── messages           <Display messages>
+    └── sciml_bench_config.yml <Directory configurations>
+
+```
+
+We have annotated the purpose of each folder/directory witin `<>`.  
+
+## 2.2 Features 
+
+A typical user-base for the benchmarking framework may include a number of user communities, such as system manufacturers and integrators (for assessing system performance), scientists (for developing new algorithms), and ML enthusiasts (for understanding the basics of various machine learning models and algorithms). It is a challenging task to design for and cover all these requirements in a single framework. Here, with SciMLBench, we have attempted to cover these requirements through the following set of features: 
+
+* Very flexible, customisable and lightweight framework,
+* Powerful logging and monitoring capabilities, 
+* Support for multiple machine learning frameworks (Tensorflow, PyTorch, and SciKit-Learn), 
+* Simplified application programming interface (API), to support easier development of benchmarks, 
+* Fully customisable installation, 
+* Simplified use of framework encouraging a wide range of users, and
+* Fully decoupled,  on-demand, and user-initiated data downloads. 
+
+
+
+## 2.3 Benchmarks and Datasets 
+
+The number of datasets and benchmarks may vary with every release. Please consult the [Benchmarks](./doc/benchmarks_datasets.md) document for this. A number authors have contributed towards the development of the benchmarks,  and these can be see in the [Credits](./doc/credits.md). If you are thinking of contributing towards the benchmarks or datasets, please see the [Contributing Datasets & Benchmarks](./doc/contributing.md).
+
+
+
+
+# 3. Installation and Usage
+
+Please consult the [Installation & Usage](./doc/usage.md) file, placed inside the `doc` folder,  for getting started. 
+
+
+
+# 4. Citation 
+
+Cite this benchmark suite as follows:
+
+    ```
+    @misc{scimlbench:2021,
+        title  = {SciMLBench: A Benchmarking Suite for AI for Science},
+        author = {Jeyan Thiyagalingam, Kuangdai Leng, Samuel Jackson,  Juri Papay, Mallikarjun Shankar, Geoffrey Fox,  Tony Hey},
+        url    = {https://github.com/stfc-sciml/sciml-bench},
+        year   = {2021}
+    }
+    ```
+
+
+# 5. Acknowledgments
+
+This benchmarking programme is supported by [1] Wave I of the UKRI Strategic Priorities Fund under the EPSRC grant (EP/T001569/1), particularly the *AI for Science* theme in that grant and the Alan Turing Institute, and [2] the Benchmarking for AI for Science at Exascale (BASE), EPSRC ExCALIBUR Phase I grant (EP/V001310/1). 
+
+<div style="text-align: right">◼︎</div>
+

config/Dockerfile

@@ -0,0 +1,54 @@
+FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
+
+ENV PYTHON_VERSION=3.8
+ENV TENSORFLOW_VERSION=2.3.0
+ENV PYTORCH_VERSION=1.7.1+cu101
+
+RUN apt-get -y update && apt-get install -y --allow-downgrades --allow-change-held-packages --no-install-recommends \
+        build-essential \
+        cmake \
+        git \
+        curl \
+        vim \
+        wget \
+        libcudnn8 \
+        libnccl2 \
+        libnccl-dev \
+        ca-certificates \
+        libjpeg-dev \
+        libpng-dev \
+        python${PYTHON_VERSION} \
+        python${PYTHON_VERSION}-dev \
+        python${PYTHON_VERSION}-distutils
+
+RUN ln -sf /usr/bin/python${PYTHON_VERSION} /usr/bin/python
+
+RUN curl -O https://bootstrap.pypa.io/get-pip.py && \
+    python get-pip.py && \
+    rm get-pip.py
+
+
+# Install TensorFlow, Keras and PyTorch
+RUN pip install torch==${PYTORCH_VERSION} torchvision==0.8.2+cu101 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html
+RUN pip install mxnet-cu101 tensorflow-gpu==${TENSORFLOW_VERSION} keras h5py filelock matplotlib
+
+# Install Horovod, temporarily using CUDA stubs
+RUN ldconfig /usr/local/cuda-10.1/targets/x86_64-linux/lib/stubs && \
+    HOROVOD_GPU_ALLREDUCE=NCCL HOROVOD_WITH_MXNET=1 HOROVOD_WITH_TENSORFLOW=1 HOROVOD_WITH_PYTORCH=1 pip install --no-cache-dir horovod && \
+    ldconfig
+
+
+RUN echo NCCL_DEBUG=INFO >> /etc/nccl.conf && \
+    echo NCCL_SOCKET_IFNAME=^docker0 >> /etc/nccl.conf
+
+
+# Copy over sciml-bench files
+COPY sciml_bench /sciml-benchmarks/sciml_bench
+COPY requirements.txt /sciml-benchmarks/requirements.txt
+COPY MANIFEST.in /sciml-benchmarks/MANIFEST.in
+COPY setup.py /sciml-benchmarks/setup.py
+COPY doc /sciml-benchmarks/doc
+
+RUN cd /sciml-benchmarks && pip install .
+
+ENTRYPOINT [ "sciml-bench" ]