IPOP and BIPOP restart mechanisms

doc/optimizers.md

@@ -778,6 +778,69 @@ optimizer2.Optimize(f, coordinates);
  * [SGD in Wikipedia](https://en.wikipedia.org/wiki/Stochastic_gradient_descent)
  * [SGD](#standard-sgd)
 
+## BIPOPCMAES
+
+*An optimizer for [separable functions](#separable-functions).*
+
+BIPOP CMA-ES (Bi-Population CMA-ES) extends the idea of IPOP CMA-ES by using
+two intertwined restart strategies: one with an increasing population size and
+another maintaining a smaller, variable population size. This strategy allows
+BIPOP CMA-ES to adaptively balance exploration and exploitation across the
+fitness landscape, and can outperform IPOP. The larger population restarts aim
+to explore broadly, improving global search capabilities, while the smaller
+populations intensify the search in promising regions.
+
+#### Constructors
+
+ * `BIPOPCMAES<`_`CMAESType`_`>()`
+ * `BIPOPCMAES<`_`CMAESType`_`>(`_`CMAES`_`)`
+ * `BIPOPCMAES<`_`CMAESType`_`>(`_`CMAES, maxRestarts`_`)`
+ * `BIPOPCMAES<`_`CMAESType`_`>(`_`lambda, transformationPolicy`_`)`
+ * `BIPOPCMAES<`_`CMAESType`_`>(`_`lambda, transformationPolicy, batchSize`_`)`
+ * `BIPOPCMAES<`_`CMAESType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy`_`)`
+ * `BIPOPCMAES<`_`CMAESType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize`_`)`
+ * `BIPOPCMAES<`_`CMAESType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize, maxRestarts`_`)`
+
+The _`CMAESType`_ template parameter is the CMA-ES type used for the optimization. 
+Currently, either CMAES or ActiveCMAES can be used.
+
+#### Attributes
+
+| **type** | **name** | **description** | **default** |
+|----------|----------|-----------------|-------------|
+| `CMAESType` | **`cmaes`** | The CMAES object used for optimization. | `CMAESType()` |
+| `size_t` | **`maxRestarts`** | Maximum number of restarts. | `9` |
+
+Attributes of the optimizer may also be changed via the member methods
+`CMAES()`, `MaxRestarts()`.
+
+#### Examples:
+
+<details open>
+<summary>Click to collapse/expand example code.
+</summary>
+
+```c++
+RosenbrockFunction f;
+arma::mat coordinates = f.GetInitialPoint();
+
+// IPOP Active CMA-ES
+BoundaryBoxConstraint<> b(-1, 1);
+CMAES cmaes(0, b, 32, 200, 1e-4);
+BIPOPCMAES optimizer(cmaes, 5);
+Optimizer.Optimize(f, coordinates);
+```
+
+</details>
+
+#### See also:
+
+ * [CMAES](#cmaes)
+ * [ActiveCMAES](#activecmaes)
+ * [IPOP-CMAES](#ipopcmaes)
+ * [Benchmarking a BI-Population CMA-ES on the BBOB-2009 Function Testbed](https://dl.acm.org/doi/pdf/10.1145/1570256.1570333)
+ * [Evolution strategy in Wikipedia](https://en.wikipedia.org/wiki/Evolution_strategy)
+
 ## Coordinate Descent (CD)
 
 *An optimizer for [partially differentiable functions](#partially-differentiable-functions).*
@@ -1484,6 +1547,73 @@ optimizer.Optimize(f, coordinates);
  * [HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent](https://arxiv.org/abs/1106.5730)
  * [Sparse differentiable separable functions](#sparse-differentiable-separable-functions)
 
+## IPOPCMAES
+
+*An optimizer for [separable functions](#separable-functions).*
+
+IPOP CMA-ES (Increasing Population Size CMA-ES) is an extension of the
+Covariance Matrix Adaptation Evolution Strategy (CMA-ES). It introduces a 
+restart mechanism that progressively increases the population size. This 
+approach is beneficial for optimizing multi-modal functions,
+characterized by numerous local optima. The restart mechanism is designed to 
+improve the adaptability of CMA-ES by improving the likelihood of escaping
+local optima, thus increasing the chances of discovering the global optimum.
+
+#### Constructors
+
+ * `IPOPCMAES<`_`CMAESType`_`>()`
+ * `IPOPCMAES<`_`CMAESType`_`>(`_`CMAES`_`)`
+ * `IPOPCMAES<`_`CMAESType`_`>(`_`CMAES, populationFactor`_`)`
+ * `IPOPCMAES<`_`CMAESType`_`>(`_`CMAES, populationFactor, maxRestarts`_`)`
+ * `IPOPCMAES<`_`CMAESType`_`>(`_`lambda, transformationPolicy`_`)`
+ * `IPOPCMAES<`_`CMAESType`_`>(`_`lambda, transformationPolicy, batchSize`_`)`
+ * `IPOPCMAES<`_`CMAESType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy`_`)`
+ * `IPOPCMAES<`_`CMAESType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize`_`)`
+ * `IPOPCMAES<`_`CMAESType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize, populationFactor`_`)`
+ * `IPOPCMAES<`_`CMAESType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize, populationFactor, maxRestarts`_`)`
+
+The _`CMAESType`_ template parameter is the CMA-ES type used for the optimization. 
+Currently, either CMAES or ActiveCMAES can be used.
+
+#### Attributes
+
+| **type** | **name** | **description** | **default** |
+|----------|----------|-----------------|-------------|
+| `CMAESType` | **`cmaes`** | The CMAES object used for optimization. | `CMAESType()` |
+| `double` | **`populationFactor`** | The factor by which population increases after each restart. | `2` |
+| `size_t` | **`maxRestarts`** | Maximum number of restarts. | `9` |
+
+Attributes of the optimizer may also be changed via the member methods
+`CMAES()`, `MaxRestarts()`, `PopulationFactor()`.
+
+#### Examples:
+
+<details open>
+<summary>Click to collapse/expand example code.
+</summary>
+
+```c++
+RosenbrockFunction f;
+arma::mat coordinates = f.GetInitialPoint();
+
+// IPOP Active CMA-ES
+BoundaryBoxConstraint<> b(-1, 1);
+CMAES cmaes(0, b, 32, 200, 1e-4);
+IPOPCMAES optimizer(cmaes, 2, 5);
+Optimizer.Optimize(f, coordinates);
+```
+
+</details>
+
+#### See also:
+
+ * [CMAES](#cmaes)
+ * [ActiveCMAES](#activecmaes)
+ * [BIPOPCMAES](#bipopcmaes)
+ * [A Restart CMA Evolution Strategy With Increasing Population Size](http://www.cmap.polytechnique.fr/~nikolaus.hansen/cec2005ipopcmaes.pdf)
+ * [Evolution strategy in Wikipedia](https://en.wikipedia.org/wiki/Evolution_strategy)
+
+
 ## IQN
 
 *An optimizer for [differentiable separable functions](#differentiable-separable-functions).*

include/ensmallen.hpp

@@ -103,6 +103,8 @@
 #include "ensmallen_bits/bigbatch_sgd/bigbatch_sgd.hpp"
 #include "ensmallen_bits/cmaes/cmaes.hpp"
 #include "ensmallen_bits/cmaes/active_cmaes.hpp"
+#include "ensmallen_bits/cmaes/ipop_cmaes.hpp"
+#include "ensmallen_bits/cmaes/bipop_cmaes.hpp"
 #include "ensmallen_bits/cd/cd.hpp"
 #include "ensmallen_bits/cne/cne.hpp"
 #include "ensmallen_bits/de/de.hpp"

include/ensmallen_bits/cmaes/bipop_cmaes.hpp

@@ -0,0 +1,147 @@
+/**
+ * @file bipop_cmaes.hpp
+ * @author Marcus Edel
+ * @author Benjami Parellada
+ * 
+ * Definition of the BIPOP Covariance Matrix Adaptation Evolution Strategy 
+ * as proposed by N. Hansen in "Benchmarking a BI-population CMA-ES on the
+ * BBOB-2009 function testbed".
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+#ifndef ENSMALLEN_CMAES_BIPOP_CMAES_HPP
+#define ENSMALLEN_CMAES_BIPOP_CMAES_HPP
+
+#include "cmaes.hpp"
+
+namespace ens {
+
+/**
+ * BI-Population CMA-ES is a variant of the stochastic search algorithm
+ * CMA-ES - Covariance Matrix Adaptation Evolution Strategy.
+ * It implements a dual restart strategy with varying population sizes: one 
+ * increasing and one with smaller, varied sizes. This BI-population approach
+ * is designed to optimize performance on multi-modal function testbeds by 
+ * leveraging different exploration and exploitation dynamics.
+ *
+ * For more information, please refer to:
+ *
+ * @code
+ * @inproceedings{hansen2009benchmarking,
+ *   title={Benchmarking a BI-population CMA-ES on the BBOB-2009 function testbed},
+ *   author={Hansen, Nikolaus},
+ *   booktitle={Proceedings of the 11th annual conference companion on genetic and evolutionary computation conference: late breaking papers},
+ *   pages={2389--2396},
+ *   year={2009}}
+ * @endcode
+ *
+ * BI-Population CMA-ES can efficiently handle separable, multimodal, and weak
+ * structure functions across various dimensions, as demonstrated in the 
+ * comprehensive results of the BBOB-2009 function testbed. The optimizer
+ * utilizes an interlaced multistart strategy to balance between broad 
+ * exploration and intensive exploitation, adjusting population sizes and 
+ * step-sizes dynamically.
+ *
+ * @tparam CMAESType The type of CMA-ES used for optimization. Options include standard CMAES and variants like ActiveCMAES.
+ */
+template<typename CMAESType = CMAES<>>
+class BIPOPCMAES
+{
+ public:
+  /**
+   * Construct the BIPOP CMA-ES optimizer with the given CMAES object,
+   * and maximum number of restarts. The defaults 
+   * here are not necessarily good for the given problem, so it is
+   * suggested that the values used be tailored to the task at hand.  The
+   * maximum number of iterations refers to the maximum number of points that
+   * are processed (i.e., one iteration equals one point; one iteration does not
+   * equal one pass over the dataset).
+   *
+   * @param CMAES The CMAES object.
+   * @param populationFactor The factor by which population increases
+   *      after each restart.
+   * @param maxRestarts Maximum number of restarts.
+   */
+  BIPOPCMAES(const CMAESType& CMAES = CMAESType(),
+             const size_t maxRestarts = 9);
+
+  /**
+   * Construct the BIPOP CMA-ES optimizer with the given function and parameters. 
+   * The defaults here are not necessarily good for the given problem, so it 
+   * is suggested that the values used be tailored to the task at hand.  The 
+   * maximum number of iterations refers to the maximum number of points that 
+   * are processed (i.e., one iteration equals one point; one iteration does 
+   * not equal one pass over the dataset). 
+   * 
+   * @param lambda The initial population size (0 use the default size).
+   * @param transformationPolicy Instantiated transformation policy used to
+   *    map the coordinates to the desired domain.
+   * @param batchSize Batch size to use for the objective calculation.
+   * @param maxIterations Maximum number of iterations allowed(0 means no
+         limit).
+   * @param tolerance Maximum absolute tolerance to terminate algorithm.
+   * @param selectionPolicy Instantiated selection policy used to calculate the
+   *     objective.
+   * @param stepSize Starting sigma/step size (will be modified).
+   * @param populationFactor The factor by which population increases
+   *     after each restart.
+   * @param maxRestarts Maximum number of restarts.
+   */
+BIPOPCMAES(const size_t lambda = 0,
+          const typename CMAESType::transformationPolicyType&
+                transformationPolicy = typename CMAESType::transformationPolicyType(),
+          const size_t batchSize = 32,
+          const size_t maxIterations = 1000,
+          const double tolerance = 1e-5,
+          const typename CMAESType::selectionPolicyType&
+                selectionPolicy = typename CMAESType::selectionPolicyType(),
+          double stepSize = 0,
+          const size_t maxRestarts = 9);
+
+  /**
+   * Optimize the given function using BIPOP CMA-ES. The given starting point will be
+   * modified to store the finishing point of the algorithm, and the final
+   * objective value is returned.
+   *
+   * @tparam SeparableFunctionType Type of the function to be optimized.
+   * @tparam MatType Type of matrix to optimize.
+   * @tparam CallbackTypes Types of callback functions.
+   * @param function Function to optimize.
+   * @param iterate Starting point (will be modified).
+   * @param callbacks Callback functions.
+   * @return Objective value of the final point.
+   */
+  template<typename SeparableFunctionType,
+      typename MatType,
+      typename... CallbackTypes>
+      typename MatType::elem_type Optimize(SeparableFunctionType& function,
+          MatType& iterate,
+          CallbackTypes&&... callbacks);
+
+  //! Get the maximum number of restarts.
+  size_t MaxRestarts() const { return maxRestarts; }
+  //! Modify the maximum number of restarts.
+  size_t& MaxRestarts() { return maxRestarts; }
+
+  //! Get the CMAES object.
+  const CMAESType& CMAES() const { return cmaes; }
+  //! Modify the CMAES object.
+  CMAESType& CMAES() { return cmaes; }
+
+ private:
+  //! Maximum number of restarts.
+  size_t maxRestarts;
+
+  //! The CMAES object used for optimization.
+  CMAESType cmaes;
+};
+
+} // namespace ens
+
+// Include implementation.
+#include "bipop_cmaes_impl.hpp"
+
+#endif