Tools for debugging

[pnevyk]

Analyzing why the solver can't find the solution or optimizer can't find the minimum is quite difficult process (at least for not so experienced people, like me) and often feels like black magic. gomez should provide ways how to help with this task.

Currently, there are some log statements which can be enabled by injecting a global logger, but this is not perfect for number of reasons (it's not possible to switch different types of logs (log level is not a good approximation) and it gets very messy in a multi-threaded environment).

In some cases, it is especially hard to understand what's happening. An evidence is some non-standard things (1, 2, 3) inside gomez, which were often discovered by accident/bug, but proved to work much better than correct code in my cases. I would like to understand these things and come up with the real solutions or explain why they work.

Nonetheless, I don't really have a perfect idea what should be done in this regard. I will use this issue to collect ideas and would also appreciate any help/advice/thoughts.

[pnevyk]

First and foremost, the solution must be opt-in and have zero cost (in terms of performance) if not opted in. That means compile-time flags such as features or cfg flags.

And here is a few random things that I would somehow want to know form a debug run and post-processing. Important to note that the main task in my application is solving the equations, not optimization, so the following ideas are biased towards that problem and described from that point of view. But a lot of them should be transferable to function optimization.

• Collect what kind of actions was the algorithm doing during the process and what was their distribution or sequence patterns. By kinds of actions I mean for instance type of step in trust region (dogleg, Newton, Cauchy, Levenberg-Marquardt) or simplex changes in Nelder-Mead (reflection, expansion, contraction, shrinking).
• Using some heuristics and the data from the previous point, come up with some possible explanations. For example, a lot of Levenberg-Marquardt means that the Jacobian matrix is often singular – and what does that mean?
• In case of singular Jacobian matrix, what variables are causing the problem? Something similar for gradient and Hessian.
• In case of no/bad derivative information (singular Jacobian, zero gradient), is the problem in calculating the approximation (bad step size ­– why?) or the system/function itself? Could it be somehow fixed (improving estimated magnitudes, using different relative epsilon, different step size computation rule, ...?
• Which equations seem to be troublesome to solve (consistently large residual).
• Which constraints (in case of constrained domain) are causing "troubles"?
• Were there any/many invalid values (infinity, NaN) during the process? What caused it?
• Does it seem that the solver is jumping "randomly"? Why?
• Somehow analyze the system/function and give descriptions. What variables are non-linear? Does it have a lot of peaks? Valleys?