Support for variables in aws pipeline (#75)

* feat: aws pipeline qvbr job * chore: npm aws s3 presigner package * feat: aws bucket generate presigned url * feat: qvbr ffprobe metadata upload * feat: read analysis data and bitrate * fix: set correct file extension for QVBR metadata * fix: metadata upload path * fix: choose correct bitrate file for pairing * fix: vmaf and metadata paths * chore: wip Signed-off-by: Gustav Grusell <[email protected]> * fix: export to csv from local files now works Signed-off-by: Gustav Grusell <[email protected]> * chore: add cli option for selecting concurrency mode Signed-off-by: Gustav Grusell <[email protected]> * chore: add cli option for skipping vmaf analysis Signed-off-by: Gustav Grusell <[email protected]> * feat: extract qvbr for csv export and combine * feat: separate vmaf model data output * chore: improve log format Signed-off-by: Gustav Grusell <[email protected]> * feat(export-csv): parse all variables from filename and include in csv Signed-off-by: Gustav Grusell <[email protected]> * feat: support for calculating statistics for a proposed bitrate ladder Also support for importing csv data into sqlite database Signed-off-by: Gustav Grusell <[email protected]> * fix: broken tests * chore: add eslint Signed-off-by: Gustav Grusell <[email protected]> * chore: fix lint errors Signed-off-by: Gustav Grusell <[email protected]> * feat: add support for setting acceleration mode when transcoding in AWS Signed-off-by: Gustav Grusell <[email protected]> * fix: include both targetBitrate and actualBitrate in csv output Signed-off-by: Gustav Grusell <[email protected]> * fix: illegal chars in tag values for ECS tasks are now replaced Signed-off-by: Gustav Grusell <[email protected]> * chore: fix linting problems Signed-off-by: Gustav Grusell <[email protected]> * chore: npm audit fix Signed-off-by: Gustav Grusell <[email protected]> * fix: remove functionality for exporting to db and print ladder while this functionality may be useful, it would be better suited for an external tool to avoid bloating autovmaf Signed-off-by: Gustav Grusell <[email protected]> * fix: fix type error Signed-off-by: Gustav Grusell <[email protected]> * ci: update publish workflow to use node 18 Node 14 seemed to cause some build error 'Cannot find module '@aws-sdk/abort-controller' or its corresponding type declarations' Signed-off-by: Gustav Grusell <[email protected]> --------- Signed-off-by: Gustav Grusell <[email protected]> Co-authored-by: oshinongit <[email protected]>
Eyevinn · Sep 6, 2024 · 28202d6 · 28202d6
1 parent f858bc6
commit 28202d6
Show file tree

Hide file tree

Showing 35 changed files with 11,442 additions and 4,310 deletions.
diff --git a/.eslintrc b/.eslintrc
@@ -0,0 +1,18 @@
+{
+  "root": true,
+  "env": {
+    "node": true,
+    "browser": true
+  },
+  "extends": [
+    "eslint:recommended",
+    "plugin:@typescript-eslint/recommended",
+    "prettier"
+  ],
+  "parser": "@typescript-eslint/parser",
+  "plugins": ["@typescript-eslint", "prettier"],
+  "rules": {
+    "prettier/prettier": "error"
+  },
+  "ignorePatterns": ["node_modules/*", "dist/*",  "examples/*", "local-types/*"]
+}
diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml
@@ -12,7 +12,7 @@ jobs:
       - uses: actions/checkout@v1
       - uses: actions/setup-node@v1
         with:
-          node-version: 14
+          node-version: '18.x'
           registry-url: https://registry.npmjs.org/
       - run: |
           npm ci

diff --git a/.gitignore b/.gitignore
@@ -4,3 +4,4 @@ node_modules
 coverage
 dist
 docs
+.idea
diff --git a/README.md b/README.md
@@ -160,6 +160,8 @@ npm install -g @eyevinn/autovmaf
 
 ### Environments variables
 
+These are only needed if you are running transcodes and VMAF measurements locally
+
 - `EASYVMAF_PATH` - needs to point to the file `easyVmaf.py` from your
   easyVmaf installation.
 - `FFMPEG_PATH` - only needs to be set if ffmpeg is not in your path.
@@ -203,16 +205,18 @@ Options:
                     key1=value1:key2=value2                             [string]
 ```
 
+### Running transcode and analysis
+
 Output files will be stored in a folder corresponding to the argument given to the `--name` option.
 If resolutions and/or bitrates are not specified default values will be used, [See above](#generate-vmaf-measurements).
 
-### Providing job definition in a json or yaml file
+#### Providing job definition in a json or yaml file
 
 With the `--job` option, a path to a yaml or json file with a job definition can be passed to to the cli. The values
 defined in the file can be overridden with other commandline options. For instance the `reference` video defined
 in the job file can be overridden by passing a source file on the command line.
 
-#### Using variables in the job definition
+##### Using variables in the job definition
 
 It is possible to iterate over other variables than bitrate and resolutions when running a local encode. For
 instance, to run transcode and vmaf analysis with x265 in CRF mode for a number of CRF values, a job definition
@@ -233,7 +237,7 @@ pipeline:
   ffmpegOptions:
     '-pix_fmt': 'yuv420p'
     '-preset': 'veryslow'
-    '-x265-params': 'crf=%CRF%:scenecut=0:keyint=50:min-keyint=50:open-gop=0'
+    '-x265-params': 'crf=${CRF}:scenecut=0:keyint=50:min-keyint=50:open-gop=0'
   easyVmafExtraArgs:
     '-threads': 20
 pipelineVariables:
@@ -245,19 +249,64 @@ pipelineVariables:
 ```
 
 This will run transcode and vmaf analysis for CRF values 22,26,30, and 34. Variables are used in the ffmpeg options
-by insterting `%VARIABLENAME%`. This string will then be substituted with a value from the list of values from
+by inserting `${VARIABLENAME}`. This string will then be substituted with a value from the list of values from
 `pipelineVariables.VARIABLENAME`. Note that when running CRF encode or other non-ABR mode, `skipDefaultOptions` must
 be set to avoid injecting bitrate options to ffmpeg. Also note that the cli needs to be run with the `--probe-bitrate`
 option to get the correct bitrate from the transcoded files.
 
-### Generate VMAF measurements example
+It is also possible to use pipelineVariables with the AWSPipeline. The following example will run transcode and vmaf analysis using the AWS MediaCOnvert QVBR levels 6, 7, 8 and 9.
+
+```yaml
+name: job-name
+pipeline: pipeline.yml
+encodingProfile: encoding-profile.json
+reference: reference.mp4
+models:
+  - HD
+resolutions:
+  - width: 1920
+    height: 1080
+bitrates:
+  - 0
+pipelineVariables:
+  QVBR:
+    - 6
+    - 7
+    - 8
+    - 9
+```
+
+#### Generate VMAF measurements example
 
 ```bash
 autovmaf --resolutions 1920x1080,1280x720,960x540 --bitrates 500k,800k,1200k,1600k,2000k,3000k,4000k --name my-autovmaf-test1 my-source-video.mp4
 ```
 
 With the above command, when the run is finished transcoded files will be available in the folder `my-autovmaf-test1`, and vmaf-data in the folder `my-autovmaf-test1/HD`.
 
+### Exporting results to csv
+
+To export results to csv, use the `export-csv` command.
+
+```
+autovmaf export-csv <folder>
+
+Export Vmaf results as csv
+
+Positionals:
+  folder  Folder with vmaf measurement results               [string] [required]
+
+Options:
+  --version       Show version number                                  [boolean]
+  --help          Show help                                            [boolean]
+  --probeBitrate  Read bitrate of transcoded file with ffprobe
+                                                      [boolean] [default: false]
+  --variables     List of variables to include as columns in csv        [string]
+```
+
+If your job uses variables ([See above](#using-variables-in-the-job-definition)), the variables that should
+be included in the csv data should be specified with the `--variables` option.
+
 ## Development
 
 ### Run tests

diff --git a/examples/aws/aws-job-qvbr.yml b/examples/aws/aws-job-qvbr.yml
@@ -0,0 +1,17 @@
+name: job-name
+pipeline: pipeline.yml
+encodingProfile: encoding-profile-qvbr.json
+reference: reference.mp4
+models:
+  - HD
+resolutions:
+  - width: 1920
+    height: 1080
+bitrates:
+  - 0
+pipelineVariables:
+  QVBR:
+    - 6
+    - 7
+    - 8
+    - 9
diff --git a/examples/aws/encoding-profile-qvbr.json b/examples/aws/encoding-profile-qvbr.json
@@ -0,0 +1,45 @@
+{
+  "Inputs": [
+    {
+      "TimecodeSource": "ZEROBASED",
+      "VideoSelector": {},
+      "FileInput": "${INPUT}"
+    }
+  ],
+  "OutputGroups": [
+    {
+      "Name": "File Group",
+      "OutputGroupSettings": {
+        "Type": "FILE_GROUP_SETTINGS",
+        "FileGroupSettings": {
+          "Destination": "${OUTPUT}"
+        }
+      },
+      "Outputs": [
+        {
+          "VideoDescription": {
+            "CodecSettings": {
+              "Codec": "H_264",
+              "H264Settings": {
+                "RateControlMode": "QVBR",
+                "QvbrSettings": {
+                  "QvbrQualityLevel": "${QVBR}"
+                },
+                "CodecProfile": "HIGH"
+              }
+            },
+            "Width": "${WIDTH}",
+            "Height": "${HEIGHT}"
+          },
+          "ContainerSettings": {
+            "Container": "MP4",
+            "Mp4Settings": {}
+          }
+        }
+      ]
+    }
+  ],
+  "TimecodeConfig": {
+    "Source": "ZEROBASED"
+  }
+}
diff --git a/examples/aws/encoding-profile.json b/examples/aws/encoding-profile.json
@@ -3,7 +3,7 @@
     {
       "TimecodeSource": "ZEROBASED",
       "VideoSelector": {},
-      "FileInput": "$INPUT"
+      "FileInput": "${INPUT}"
     }
   ],
   "OutputGroups": [
@@ -12,7 +12,7 @@
       "OutputGroupSettings": {
         "Type": "FILE_GROUP_SETTINGS",
         "FileGroupSettings": {
-          "Destination": "$OUTPUT"
+          "Destination": "${OUTPUT}"
         }
       },
       "Outputs": [
@@ -22,12 +22,12 @@
               "Codec": "H_264",
               "H264Settings": {
                 "RateControlMode": "CBR",
-                "Bitrate": "$BITRATE",
+                "Bitrate": "${BITRATE}",
                 "CodecProfile": "HIGH"
               }
             },
-            "Width": "$WIDTH",
-            "Height": "$HEIGHT"
+            "Width": "${WIDTH}",
+            "Height": "${HEIGHT}"
           },
           "ContainerSettings": {
             "Container": "MP4",

diff --git a/examples/local/local-job-crf.yaml b/examples/local/local-job-crf.yaml
@@ -13,7 +13,7 @@ pipeline:
   ffmpegOptions:
     '-pix_fmt': 'yuv420p'
     '-preset': 'veryslow'
-    '-x265-params': 'crf=%CRF%:scenecut=0:keyint=50:min-keyint=50:open-gop=0'
+    '-x265-params': 'crf=${CRF}:scenecut=0:keyint=50:min-keyint=50:open-gop=0'
   easyVmafExtraArgs:
     '-threads': 20
 pipelineVariables: