Updates

[gasse]

Dear participants,

Please find below a few updates regarding the competition.

Important dates

We have updated the important dates section on the competition's website. The virtual NeurIPS timeline will require us to wrap up the competition and present the results in a video recording early before the actual event (no precise date yet). Because at the moment we have almost 100 teams registered already (:tada:), we have decided to move the submission deadline a bit earlier in order to be on the safe side to handle the workload at the end of October (reception and processing of all submissions). Please note that the final deadline is now October 21st (it was October 31st before)

• Oct 14th '21 - registration deadline, no new team is accepted past this date
• Oct 21st '21 - submission deadline, no submission is accepted past this date
• Nov 14th '21 - final result annoucement and declaration of the winners
• December '21 - virtual event at NeurIPS with poster session for all participants
• February '22 - official report with winner contributions, to be published in PMLR

Ranking rules

We have updated the rules section on the competition's website, to include a description of the rule we will use to declare a single winner in each task, which had been a bit brushed under the carpet until now. Indeed, since participants compete on three benchmark datasets in each task, a formula has to be used to produce an aggregated overall ranking of the teams. What we have decided to opt for is a geometric mean of the rankings of each team across all three benchmarks (multiplication), which somewhat favors teams that rank 1st in at least one benchmark (for example: 1x2x4=8 < 2x2x3=12). This choice is arbitrary, and of course any formula has its drawbacks, but the most important we though was to make this formula clear early enough to avoid any ambiguity, which is now done.

• Each team will be evaluated separately on each of the three benchmark datasets according the task's metric, i.e., primal integral, dual integral, or primal-dual integral. This metric will be averaged across all instances in the test dataset, resulting in an individual ranking for each dataset. For example, for the primal task, team A might obtain ranks 3, 8, 1 respectively in the three problem benchmark, while team B obtains ranks 2, 2, 15.
• Then, for each team those three ranks will be multiplied together to obtain the overall score of the team. For example, team A will obtain a score of 3x8x1=24 while team B obtains 2x2x15=60.
• The team with the lowest overall score wins.

Results

We have had a few hiccups in processing the first submissions we received so far, due to technical issues with CUDA on our evaluation cluster. This has now been fixed and the submissions have been successfully processed, so the first (partial) results should be made public very soon. We will of course make a new announcement when this is done.

We wish you good luck with your submissions, and most importantly, a lot of fun !

Best,
The organizing team