Question about Leaderboard metrics

[msublee]

What is the difference between dual integral and cumulative reward?
In common/evaluate.py, we can see how we get the cumulative reward, but I wonder how you calculate the dual integral on the leaderboard.

[gasse]

Hi @msublee ,

This is a bit confusing indeed, but really these two measures are equivalent. One is simply the negated (and potentially shifted) version of the other. You're not the first to wonder, so I'll refer you to this discussion and also to the text accompanying the leaderboard:

Note: for each challenge and each benchmark, we report two performance measures:

• the average evaluation metric (primal integral, dual integral, or primal-dual integral) obtained over the test instances, which is to be minimized. This measure is described in detail in the metrics section, and corresponds to the negated cumulated reward computed by our evaluation script, plus a constant term that depends on the optimal solution value of each instance. Because for some instances it is hard to obtain the optimal solution value, and because this constant term is the same for every participant and does not impact the rankings, we compute an adequate estimate to substitute cTx* when the optimal solution value is not available, based on the best primal and dual bounds obtained after some time limit.

• the average cumulated reward (cum. reward) obtained over the test instances, which is to be maximized. We report this measure for information purposes, as it is exactly the one computed by our evaluation script. This measure can be interpreted as the negated and unshifted version of the evaluation metric (it does not depend on cTx*), and it produces the exact same ranking.

Let us know if things remain unclear.

Best,
Maxime

[msublee]

Thank you very much for your answer.

But I'm still confused.
I wonder why their (dual integral and cumulative reward) absolute values are not equal in the primal and dual task (they have the same absolute values in the configuration task). If the one is simply the negated version, I think they should have the same absolute value.

Best,
Minsub.

[gasse]

Hi @msublee ,

In the config task yes, one is the negated version of the other. In the primal and dual task however, we also add a constant to the cumulated reward, and then we take the negated value. This shift is basically -Tcx* or +Tcx*, with cx* the optimal solution value of the instance, as explained in the Metrics section of the competition's webpage. You can compute this shift for each team in the leaderboard, you should get the same value for each of the benchmarks.

Best,
Maxime

[msublee]

Thank you very much. It helped me a lot.

Best,
Minsub.