Don't reset the objective estimate on the last iteration

[rcurtin]

I noticed while working with RNNs in mlpack that the estimated objective would be 0 if I was training sequences of different length one iteration at a time... digging into it, I discovered that any time you optimize a separable function with SGD-like optimizers and the number of iterations you use is an exact multiple of the number of functions (e.g. number of sequences given to the RNN), then the returned objective estimate is 0 unless you specify exactObjective.

This is because the objective estimate gets reset after every epoch, but if it is the last epoch... it still resets it anyway, it doesn't return the estimate.

So I fixed that for all optimizers I could find that had the condition, and then I updated the documentation to be a little clearer about what does get returned for separable functions.

A test program:

#include <ensmallen.hpp>

using namespace ens;
using namespace ens::test;

int main()
{
  GeneralizedRosenbrockFunction f(11);
  std::cout << "Number of functions: " << f.NumFunctions() << ".\n";

  // With one epoch only.
  StandardSGD s(0.001, 1, f.NumFunctions(), 1e-5);
  arma::mat coordinates = f.GetInitialPoint();
  double result = s.Optimize(f, coordinates);
  std::cout << "Optimized to result: " << result << ".\n";

  // With not an even number of epochs.
  s.MaxIterations() = f.NumFunctions() - 1;
  coordinates = f.GetInitialPoint();
  result = s.Optimize(f, coordinates);
  std::cout << "Optimized to result: " << result << ".\n";
}

Compile with:

g++ -O3 -DENS_PRINT_INFO -DENS_PRINT_WARN -I../include/ -o out_test out_test.cpp

or similar.

Output before this PR:

$ ./out_test
Number of functions: 10.
SGD: iteration 10, objective 2257.11.
SGD: maximum iterations (10) reached; terminating optimization.
Optimized to result: 0.
SGD: maximum iterations (9) reached; terminating optimization.
Optimized to result: 2399.38.

Output after this PR:

$ ./out_test 
Number of functions: 10.
SGD: iteration 10, objective 2257.11.
SGD: maximum iterations (10) reached; terminating optimization.
Optimized to result: 2257.11.
SGD: maximum iterations (9) reached; terminating optimization.
Optimized to result: 2399.38.

[rcurtin]

This is the same note I used for all optimizers with the exactObjective option. Happy to accept any suggestions that can make it more concise or clearer without breaking the meaning.

[github-actions]

Second approval provided automatically after 24 hours. 👍