Small updates to the evaluation environment

[gasse]

Dear participants,

We've addressed a few issues with the evaluation environment, which led to a few update in the ml4co-competition repo as well as a new minor release for the Ecole library. Those minor changes will most likely not affect you a lot, however please note that we now use Ecole version 0.7.3 to evaluate your submissions, so please update the ecole version in your training environments and in your submissions (ecole=0.7.3 in your conda.yaml file) !

The exact changes are detailed below:

1. we fixed a few bugs in our config baseline implementation, it should now work properly.

2. we now impose a memory limit of 19GiB to SCIP in the evaluation environment, so that we can prevent and recover OOM errors on the machines we use for evaluation (which have a hard limit of 20GiB). In the event where SCIP reaches this memory limit, the environment simply stops and the task's metric (primal integral, dual integral or primal-dual integral) is completed by considering that neither the primal or dual bounds can improve in the remaining time window T. Note that hard OOM errors are still possible if the code in your agent also requires a large amount of memory (>=1GiB), so that the total memory of the process exceeds 20GiB. We are studying whether and how we could recover from such a situation but the solution does not appear to be trivial, so for now when this happens we will simply consider it as a code failure and communicate with you to try to find a solution.

3. we added a new option to the ecole.observation.Khalil2016 observation function, which allows to extract those features for either pseudo branching candidate variables (SCIPgetPseudoBranchCands) or LP branching candidate variables (SCIPgetPseudoBranchCands). This feature has been requested by several users (see this discussion as well as the the pull request), and does not introduce any breaking change. This is the only change introduced in ecole v0.7.3 (release notes here).

Again, we wish you good luck and fun with your submissions !

Best,
The organizing team

[superboom]

"Note that hard OOM errors are still possible if the code in your agent also requires a large amount of memory (>=1GiB)"
According to your description, test instances are evaluation with multiple processes on one machine?

[gasse]

Hi @superboom ,

No, evaluation codes are run one at a time, one per machine, with the same machine specifications for everyone (see our hardware description). We use a scheduling system (SLURM) for practical reasons, where each job occupies an entire machine, and consist in a single process that installs your environment, and then runs your code on the test instances sequentially, one at a time. Still, the machines we run your codes on are limited in memory and compute capacity, so you should pay attention to memory usage in your codes, both regarding CPU RAM (20G) and GPU RAM (8G). If OOM errors (or errors of any other kind) happen during the evaluation, we will come back to you to help you figure out a solution.

Best,
Maxime