Nelder-Mead simplex algorithm with adaptive parameters #231
Conversation
| @@ -1246,7 +1246,7 @@ can be paired with the `Lookahead` optimizer. | |||
| | `BaseOptimizerType` | **`baseOptimizer`** | Optimizer for the forward step. | Adam | | |||
| | `double` | **`stepSize`** | Step size for each iteration. | `0.5` | | |||
| | `size_t` | **`k`** | The synchronization period. | `5` | | |||
| | `size_t` | **`max_iterations`** | Maximum number of iterations allowed (0 means no limit). | `100000` | | |||
| | `size_t` | **`maxIterations`** | Maximum number of iterations allowed (0 means no limit). | `100000` | | |||
| | `double` | **`alpha`** | The reflection parameter. | `` | | ||
| | `double` | **`beta`** | The expansion parameter. | `` | | ||
| | `double` | **`gamma`** | The contraction parameter. | `` | | ||
| | `double` | **`delta`** | The shrink step parameter. | `` | |
There was a problem hiding this comment.
Should we mention that the defaults here are dependent on the dimension of the problem, according to Equation 4.1 of the paper? https://www.webpages.uidaho.edu/~fuchang/res/ANMS.pdf
| #### See also: | ||
|
|
||
| * [Nelder-Mead in Wikipedia](https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method) | ||
| * [Arbitrary functions](#arbitrary-functions) |
There was a problem hiding this comment.
Should we also update the documentation for arbitrary functions to reference Nelder-Mead?
| @@ -103,6 +103,7 @@ | |||
| #include "ensmallen_bits/parallel_sgd/parallel_sgd.hpp" | |||
| #include "ensmallen_bits/pso/pso.hpp" | |||
| #include "ensmallen_bits/rmsprop/rmsprop.hpp" | |||
| #include "ensmallen_bits/nelder_mead/nelder_mead.hpp" | |||
There was a problem hiding this comment.
It's up to you, but the rest of these seem to be sorted alphabetically, so maybe we should do the same with this one. :)
| * @param function Function to optimize. | ||
| */ | ||
| template<typename FunctionType, typename MatType> | ||
| void Simplex(MatType& simplex, |
There was a problem hiding this comment.
I suppose it does not make a difference for the API you are using, but you could mark this static if you wanted.
| // Shift values by one to the right and swap the last with the first. | ||
| const size_t fLowest = order(dim); | ||
| for (size_t d = dim; d > 0; --d) | ||
| order(d) = order(d - 1); |
There was a problem hiding this comment.
Here too you could use subvec() to copy all at once, or, if you use a max-heap this step may not be needed. (I believe std::priority_queue is a max-heap.)
| else | ||
| { | ||
| // Inside contraction. | ||
| const MatType xic = (1.0 - gamma) * M + gamma * |
There was a problem hiding this comment.
Should this have alpha multiplied too? I did the algebra in my head and I think it should, but I didn't double check and I have a headache right now... 😄
| // Sorting can be slow, however there is overwhelming evidence that shrink | ||
| // transformations almost never happen in practice. See "Efficient | ||
| // Implementation of the Nelder–Mead Search Algorithm" by S. Singer and S. | ||
| // Singer. |
There was a problem hiding this comment.
I thought for sure this had to be a typo, but no, there are actually two S. Singers, and they collaborated on this. 😄
| for (size_t d = 1; d < dim; ++d) | ||
| { | ||
| simplex.col(order(d)) = simplex.col(order(0)) + delta * | ||
| (simplex.col(order(d)) - simplex.col(order(0))); |
There was a problem hiding this comment.
It could be possible to express this as two operations (first on columns with index less than order(0), and second on columns with index greater than order(0)), using repmat(), but it's up to you if you want to rewrite it like that. I believe it would be more efficient, but I don't know how much of a bottleneck it would be. The Evaluate() would still need to be called over all new points in a loop though.
| // Singer. | ||
| order = arma::sort_index(fx); | ||
| fx = fx.cols(order); | ||
| simplex = simplex.cols(order); |
There was a problem hiding this comment.
Is it necessary to reorder simplex at the end of the loop? (And fx too?) Inside the main loop of the optimization, those two members are always accessed with order(), so I think it is fine to leave them in the order they're in.
|
This issue has been automatically marked as stale because it has not had any recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions! 👍 |
Nelder-Mead simplex algorithm with adaptive parameters as proposed in "Implementing the Nelder-Mead simplex algorithmwith adaptive parameters" by F Gao and L. Han. The implementation outperforms the standard Nelder-Mead method (for high dimensional problems).