understanding video.bench and video.eval_model

[amarzullo24]

Hi!
I would like to use the utils scripts to make a distortion-rate plot comparing x264 and ssf2020 on my own dataset. When I use the video.bench, it generates one single value for each metric:

{
  "name": "x264-medium-tune-psnr",
  "description": "x264 medium, psnr, ffmpeg version 3.4.8-0ubuntu0.2",
  "results": {
    "qp": [
      32
    ],
    "ms-ssim-rgb": [
      0.9757549166679382
    ],
    "bitrate": [
      4185.242424242424
    ],
    "psnr-rgb": [
      26.23819639661245
    ],
   ...

This obviously results in a plot with one single point rather than a curve. Am I doing something wrong or I am misunderstanding the meaning of the plot?

Furthermore, I would like to evaluate the models on a dataset of videos (ideally, I would obtain a mean rate-distortion curve with the corresponding standard deviation). However, I have never seen such a plot in all the papers I have read so far. In your experience, how such a situation is usually handled?

Thanks for your help!

[fracape]

Hi,
you edited the first message, I guess you found the solution. I can see you ran ffmpeg on a qcif file (176x144) 30Hz and output a .yuv with dimensions 480x480 and 25Hz, which failed to create what you were expecting I suspect.
For bench/eval, you can provide multiple quality levels in your command line which result in a json file that contains all the results, like in the provided json files.
python3 -m compressai.utils.video.bench x265 --help
If you use a grid with a separate scheduler, you can also run your jobs separately for each QP and re-run the same command with all qps after they are finished. Then, since the results are already present in your output folder, bench/eval will skip encode/decode and just collect the results in a single .json file
hope this helps