Gradient Starvation

.gitignore

@@ -26,3 +26,12 @@ docs/site/
 # committed for packages, but should be committed for applications that require a static
 # environment.
 Manifest.toml
+
+# VS Code 
+.vscode/
+
+# Local History for Visual Studio Code
+.history/
+
+# Built Visual Studio Code Extensions
+*.vsix

Project.toml

@@ -1,9 +1,13 @@
-name = "STMOZOO"
-uuid = "8e7dc78f-7190-4116-bf88-3d148ae9f088"
-authors = ["michielstock <[email protected]>"]
-version = "0.1.0"
+name = "GradientStarvation"
+uuid = "b11d05d9-c3d3-46bc-b9c9-1875ab7cf247"
+authors = ["Peter Merseburger <[email protected]>"]
+version = "0.0.99"
 
 [deps]
-Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
-LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
-Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
+DataStructures = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
+Flux = "587475ba-b771-5e3f-ad9e-33799f191a9c"
+MLDatasets = "eb30cadb-4394-5ae3-aed4-317e484a6458"
+PlotlyJS = "f0f68f2c-4968-5e81-91da-67840de0976a"
+Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
+ProgressMeter = "92933f4c-e287-5a05-a399-4b506db050ca"
+ScikitLearn = "3646fa90-6ef7-5e7e-9f22-8aca16db6324"

README.md

@@ -1,14 +1,10 @@
-# STMO-ZOO
+# Gradient Starvation 🍴💀
 
-Welcome to the STMO zoo! This is your final assignment for the course Selected Topics in Mathematical Optimization. Your goal is to implement an optimization method in Julia and contribute this to this repository. To pass, you have to:
+Welcome to my STMO project!
 
-- fork this repo and create a pull request;
-- add a module to `src` with **at least one function**
-- add at least one unit test to the folder `test`;
-- document all your functions and add a page to the documentation page;
-- make a notebook in [Pluto](https://github.com/fonsp/Pluto.jl) and add it to `notebooks`;
-- perform a small code review of two other students.
+I wanted to explore the findings of Pezeshki et al. presented in their publication [Gradient Starvation: A Learning Proclivity in Neural Networks](https://arxiv.org/abs/2011.09468).
+The goal was to explain the problem of gradient starvation and to investigate its effects and a proposed remedy on a simple 2D classification task.
 
-Depending on the project you choose some of these individual assignments might be really minimalistic, with other parts larger. For example, if you want to develop an application, say solving the graph coloring problem with Tabu Search, you might have only a single function in the source code (e.g., generating an instance) but have a fairly large notebook with a tutorial. On the other hand, if you work on a method, e.g., implementing Bee Colony Optimization, you might have many functions in the source code, while your notebook is only a demonstration on the test functions. 
+The notebook and the code of my project are separate! You can find them in their respective subfolders `notebook` and `src`.
 
-[![Build Status](https://travis-ci.org/MichielStock/STMOZOO.svg?branch=master)](https://travis-ci.org/MichielStock/STMOZOO)[![Coverage Status](https://coveralls.io/repos/github/MichielStock/STMOZOO/badge.svg?branch=master)](https://coveralls.io/github/MichielStock/STMOZOO?branch=master)
+All experiments conducted, including the related figures, can be simply repeated and re-created by running `include("src\\gradientstarvation.jl")` from a Julia REPL after checking out the project on your local machine.

docs/Project.toml

@@ -1,3 +1,4 @@
 [deps]
+DataStructures = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
-STMOZOO = "8e7dc78f-7190-4116-bf88-3d148ae9f088"
+GradientStarvation = "b11d05d9-c3d3-46bc-b9c9-1875ab7cf247"

docs/make.jl

@@ -1,17 +1,37 @@
 using Documenter
+using GradientStarvation.Data
+using GradientStarvation.Figures
+using GradientStarvation.Experiments
+using GradientStarvation.NeuralNetwork
+using GradientStarvation.PlotLib
+using GradientStarvation.PlotUtils
+using GradientStarvation.Regularization
 
-using STMOZOO
-using STMOZOO.Example
-
-makedocs(sitename="STMO ZOO",
+makedocs(
+    sitename = "Gradient Starvation",
     format = Documenter.HTML(),
-    modules=[Example], # add your module
-    pages=Any[
-        "Example"=> "man/example.md",  # add the page to your documentation
-    ])
+    modules = [
+        GradientStarvation.Data, 
+        GradientStarvation.Figures, 
+        GradientStarvation.Experiments, 
+        GradientStarvation.NeuralNetwork, 
+        GradientStarvation.PlotLib, 
+        GradientStarvation.PlotUtils,
+        GradientStarvation.Regularization
+    ],
+    pages = Any[
+        "Data" => "man/data.md",
+        "Figures" => "man/figures.md",
+        "Experiments" => "man/experiments.md",
+        "Neural Network" => "man/neuralnetwork.md",
+        "Plots" => "man/plotlib.md",
+        "Plot Utils" => "man/plotutils.md",
+        "Regularization" => "man/regularization.md"
+    ]
+)
 
 #=
 deploydocs(
-            repo = "github.com/michielstock/STMOZOO.jl.git",
+            repo = "github.com/justinsane1337/GradientStarvation.git",
         )
 =#

docs/src/index.md

@@ -1,7 +1,4 @@
-# STMO ZOO documentation
-
-This is the documentation page for the STMO ZOO packages (edition 2020-2021). This is the place for the **documentation**
-of your function. So explain the main functionality of your module and list the documentation of your funtions.
+# Gradient Starvation documentation
 
 ```@contents
 ```

docs/src/man/data.md

@@ -0,0 +1,20 @@
+## Data
+
+This module provides basic functionality to obtain training-ready `Flux.DataLoader`s.
+
+Following methods are exported:
+
+```@docs
+get_moon_data_loader
+get_nmist_data_loader
+```
+
+These helper functions are private to the module:
+
+```@docs
+generate_moons_from_publication
+generate_moons
+apply_moon_offset
+moon_class_1
+moon_class_2
+```

docs/src/man/experiments.md

@@ -0,0 +1,7 @@
+## Experiments
+
+The Experiments module provides a single function to run experiments as defined by a `DataStructures.OrderedDict` containing experiments according to the `Experiment` struct.
+
+```@docs
+run_experiments
+```

docs/src/man/figures.md

@@ -0,0 +1,7 @@
+## Figures
+
+This module collects all functions to create any special figures appearing in the notebook. This does not include plots that are produced along during the processing of experiments.
+
+```@docs
+create_figure_1
+```

docs/src/man/neuralnetwork.md

@@ -0,0 +1,14 @@
+# Neural Network
+
+Here, we basically have the main module of the whole project. The neural network architecture is defined in this module, as well as functions to train it, to retrieve an optimization strategy or obtain information about training loss and accuracy.
+
+```@docs
+get_loss_and_accuracy
+train
+```
+
+Private helper methods:
+```@docs
+neural_network
+get_optimizer
+```

docs/src/man/plotlib.md

@@ -0,0 +1,10 @@
+# Plot Library
+
+To avoid naming confusion with `Plots`, my plot module was coined `PlotLib` and contains solely functions that are concerned with the creation of plots. Thus, all of these return ready to display `PlotlyJS` plots. In contrast to the functionality provided in `Figures`, functions presented here are more raw or general purpose in the sense that they plot a certain aspect of the neural network and the training process.
+
+```@docs
+plot_train_and_test_data
+plot_loss_and_accuracy
+plot_features
+plot_decision_boundary
+```

docs/src/man/plotutils.md

@@ -0,0 +1,9 @@
+# Plot Utils
+
+Addtional functionality related to plotting. These functions are helper functions that do not directly produce plots.
+
+```@docs
+get_custom_rdbu_scale
+get_plot_resolution
+map_bool_to_color
+```

docs/src/man/regularization.md

@@ -0,0 +1,7 @@
+## Regularization
+
+This module contains custom-implemented regularizations like spectral decoupling.
+
+```@docs
+spectral_decoupling
+```