-
Notifications
You must be signed in to change notification settings - Fork 224
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GSoC: Test Function Suite #223
Comments
Note that CUTEst.jl is now available from https://github.com/JuliaOptimizers/CUTEst.jl in more useable form. A lot of the CUTEst problems are also available in AMPL format from https://github.com/mpf/Optimization-Test-Problems and you can read them (once decoded) using https://github.com/JuliaOptimizers/AmplNLReader.jl. If you want to change the size of an AMPL model, you must edit the
and subsequently decode with
CUTEst.jl doesn't currently support passing extra options to |
Thank you for your reply. It was indeed the JuliaOptimizers version of CUTEst I installed, I should have made that clear. I only used the blog-post to install the homebrew-stuff. Thanks for the other pointers! |
Hi, thanks for using my post, but it isn't updated to the last version of the packages. Also, I'm interested in the load problems you had. |
A bindeps version would be wonderful, as the whole homebrew thing was a bit, well, annoying :) I would love to see what the following does on your installation: https://gist.github.com/pkofod/5dda61027b6829e91c40dccbe6f38f9b . On my computer, all of those problems throw an error, so |
Did something go wrong?
What is the error? Did you try from the command line? |
It's started here. Haven't tested it in a while, but it was working.
If you're on linux, I agree with you. Notice how bindeps uses linux-cutest, which is a simpler alternative, although very underdeveloped (no tests, no reinstall, etc.) |
Tried it, and they all throw an error, but that's because those problems are not on
|
No, nothing went wrong, but as a linux user, homebrew is not really something I've ever used for anything before. It all worked out (after I found the blog post), but it would of course be preferable if
Okay, my bad then! The error was probably just that I didn't have them. I thought MASTSIF had all the CUTEst problems.Thanks for the grep. |
So, the revised test set is 173 unconstrained problems with the following dimensions:
Quite a few big problems, and maybe a bit too few in the range 10-50 (I suspect a lot of users might be interested in these. I'm running stuff, but it takes a while, that's why there's been no update :) |
What are you measuring in this profile? Number of objective evals? |
That's instructive. The two plots together tell a more complete story. Armijo remains amazingly robust given how simple it is. I'll have to look at the details of your interpolating linesearch. |
I would still say hz is dominating if we take into consideration the actual minimum they end up reporting. Edit: this was supposed to be just after the image not a response to you :) |
I should compare these with LBFGS. thanks again for the CUTEst package :) the newest version is even easier to install ! |
I'm just spamming at this point. Any participants are welcome to unsubscribe if they're annoyed :) This is actually quite surprising to me! (I took the upper tau to 20 for clarity, the picture remains the same with other taus bars). Conjugate Gradient is doing remarkably bad compared to Gradient descent here! I've cherry-picked the best line search algorithms for each "solver" or overall algorithm. Edit Hm, I think the between solver results are wrong, but within each solver it should be correct. Will be back with an update :) |
I didn't really follow along here, but just scrolling over the images made me curious. Are these for the Julia implementation in this package? if so what library did you use for plotting? |
Yes they are. Plots.jl with the Plotly backend |
Almost surely a bug in CG, perhaps in the convergence criteria? You already noticed one typo, there may be others! This is what's so wonderful about having a good test suite! 💯 |
the dominating gradient descent was definitely an error on my part, but now I can't see the cgs in the plot. In a car, will check later. I agree that the test suite is going to be very helpful. The current test set is 155 functions, and it excludes the largest ones. Not a bad start I think. |
@pkofod, where is the code you're using to run these experiments? |
I just mean that I'll post the code I used to produce these figures. It's only available if you're setting next to me right now. :) I'll post a gist or something. I just need to organize it a bit better. |
As an example, this code produces the figures posted in #220 , that I repeat here: Do produce that figure, I first load the module, Plots, and Optim. Then I construct a vector of names, and a vector of using Plots, OptimBenchmarks, Optim
names = ["NelderMead with fixed parameters and unit step",
"NelderMead with fixed parameters and relative_step (0.05)",
"NelderMead with fixed parameters and relative step (0.1)",
"NelderMead with fixed parameters and relative step (0.50)",
"NelderMead with adaptive parameters and relative step (0.05)",
"NelderMead with adaptive parameters and relative step (0.1)",
"NelderMead with adaptive parameters and relative step (0.50)"]
opt_cases =[Optim.NelderMead(parameters = Optim.fixed_parameters, initial_step = Optim.unit_step),
Optim.NelderMead(parameters = Optim.fixed_parameters, initial_step = x->Optim.relative_step(x; s = 0.05)),
Optim.NelderMead(parameters = Optim.fixed_parameters, initial_step = x->Optim.relative_step(x; s = 0.1)),
Optim.NelderMead(parameters = Optim.fixed_parameters, initial_step = x->Optim.relative_step(x; s = 0.5)),
Optim.NelderMead(parameters = Optim.adaptive_parameters, initial_step = x->Optim.relative_step(x; s = 0.05)),
Optim.NelderMead(parameters = Optim.adaptive_parameters, initial_step = x->Optim.relative_step(x; s = 0.1)),
Optim.NelderMead(parameters = Optim.adaptive_parameters, initial_step = x->Optim.relative_step(x; s = 0.5))] then I run the benchmarks. It uses try/catches, as an out of memory error or line search error would otherwise ruin everything. We should make line searches soft quit imo. out = benchmark(names, opt_cases, Optim.OptimizationOptions(iterations = 20000); max_dim = 1000)
save("nelder_mead_benchmark.csv", out...) Then, load the data as a dataframe (I just saved it to keep a copy), and plot it dff = read_profile_data("nelder_mead_benchmark.csv")
tau = 1:0.5:20
profiles = performance_profile(names, dff, tau)
plot(tau, hcat(profiles...),
label = hcat(names...),
ls=[:solid :dash :dash :dash :dot :dot :dot],
size =(800,500),
ylims = (0,1)) the module is in https://gist.github.com/pkofod/82ba6021e73cf9ee2289f35b08a09f2e (some things could be done more cleanly...) (note to self, I should wait with the call to aggregate until the |
This work is now in OptimTests, thanks for the feedback. |
I figured I should update details in separate issues instead of the one big issue #200 .
I am planning to start off by testing on CUTEst unconstrained functions, through the CUTEst.jl package. I installed it by following http://abelsiqueira.github.io/blog/installing-cutest-and-cutest.jl/ . The develop branch is necessary, as this has a
grad!
function defined already. The master branch also segfaulted on me quite often.I had originally chosen the 216 unconstrained problems in CUTEst, but one seemed to have constraints, and CUTEst.jl had problems loading quite a few (I need to file an issue upstream), so now there are closer to 160.
An interesting feature of the test suite is how broadly we're testing in terms of dimensionality of x (call it N). It seems that a lot of functions are 2- or 3- dimensional, but also values of around 3000 and 5000 seem popular. 11 functions have N = 10000, and 2 have N = 20000. Some of the original formulations allow for a user defined N (like in the Rosenbrock case), but I'm not sure of CUTEst allows the user to specify this.
Anyway, this is a much broader test bed than we already have, and I'll try to post some initial benchmarks as soon as possible.
The text was updated successfully, but these errors were encountered: