Max iterations based on validation_loss?

[jmrichardson]

Hi, is it possible to set the maximum number if iterations per learner without validation loss improvement? I know there is a early_stop setting but it appears to be based on convergence. Thanks for any guidance

[sonichi]

Do you mean stop trying a learner after a given number of iterations without validation loss improvement? There is no such option in the current library.

[jmrichardson]

Hi @sonichi

Yes, that would be great to be able to stop after N number of iterations without validation loss improvement. In my case, I set a budget but often times I find that after N number of iterations, it is unlikely a better parameter set will be found and thus could be stopped. Any plans to implement?

[qingyun-wu]

Hi @jmrichardson,
That's an interesting early stopping idea, which is currently not supported but can be added presumably easily. I have several clarification questions:

1. "N" is the number of consecutive iterations where there is no improvement in the validation performance and the actual value of "N" is supposed to be specified by the user. Are these understandings accurate?
2. In addition to the number of iterations without validation loss improvement, do you think using the amount of wall-clock-time (e.g., in seconds) without validation loss improvement is also a good indicator for early stopping?
3. Can you please share more details about your use case and if possible code examples?

Thank you!

[jmrichardson]

Hi @qingyun-wu

1: Yes that is correct
2: Great idea! I think this would be very valuable
3: Essentially I have millions of observations time series data with a low signal to noise ratio (financial data). I do partition the data in multiple ways such as rolling windows, every nth observation, consecutive, and others where each partition of the data requires tuning. I have been doing it from scratch each time and now looking into the warm start aspect of FLAML. However, this feature would help streamline the process a bit as currently I just set the budget very high and kill the process manually while keeping the best config from the log. I noticed that generally speaking with time the number of iterations between improvement gets very large or never gets better so it would be good to just stop especially if the performance is better than the baseline.

Thank you for considering this feature request :)

[qingyun-wu]

Hi @jmrichardson,

Thank you for sharing your use case. Do you want to define the desired API in a PR? If you prefer not to do it, we will define the API and ask you to review it. Please let me know your preference.

Thank you!

[jmrichardson]

Hi @qingyun-wu

I would prefer to review if possible. Thank you so much!

[qingyun-wu]

I've created an issue for this request. We do not have an ETA for now but will try to find some time to resolve it. Please feel free to make your proposal if this is urgent to you.

Thank you!