Graph validation test revisions (#100)

* Support for "Skipped Tests" added to the codes.yaml and code base; 
* Add two higher levels of indexing - 'target' and 'test' - into the message catalogue of the **ValidationReporter**
* Refactored 'add_messages' to consume a full "MESSAGES_BY_TARGET" blob of messages, indexed by target and test, ...not just a single core info.. etc. block
* Remove target string from report header ('target' now embedded inside the report)
* Substantial implementation of **get_all_messages_of_type()** to aggregate all messages of a given type from all test and target contexts of a ValidationReporter.
* Allow for a 'case' data entry to use 'predicate_id' in lieu of 'predicate'

CHANGELOG.md

@@ -2,14 +2,20 @@
 
 The Reasoner Validator package is evolving along with progress in TRAPI and Biolink Model standards within the NCATS Biomedical Knowledge Translator.
 
+## 4.0.0
+The major release changes relate to adapting the reasoner-validator to the new (2023-24) Translator testing infrastructure:
+- Two higher levels of indexing are added to the message catalog of the **ValidationReporter**: indexing by target (endpoint url) and indexing by test identifier.
+- Support for "Skipped Tests" added to the codes.yaml and code base
+- The ValidationReporter 'prefix' field converted into a 'default_target' field and accessor methods provided, with a new basic unit test
+- OpenTelemetry features in API disabled for the time being, pending future update
+
 ## 3.9.5
 - pydantic updated to 1.10.13
 - Small clean-up of trapi_validator.py CLI help text
 - gitignore JSON files in scripts subfolder to allow local validation of said JSON files without committing them to the repo
 - silenced warnings about missing workflow schema defined 'parameters' and 'runner_parameters' (workflow schema allows these to be null but...) - but keeping the validation codes in the codes.yaml (for now)
 - fixed subtle bug with id prefix mapping validation
 
-
 ## 3.9.4
 - Demote 'knowledge_graph.node.name.missing' validation error into a warning.
 - Restore TRAPI 'workflow' sanitization to fix null 'parameters' or 'runner_parameters', pending future repairs to the workflow schema (i.e. 'nullable: true' directives?)

DEVELOPER_NOTES.md

@@ -28,11 +28,11 @@ Steps to properly issue a new project release:
 2. If the **codes.yaml** were revised, regenerated the associate code documentation by running the reasoner_validator/validation_codes.py module as a script from the CLI.
 3. Run the unit test suite to ensure that nothing fails. Iterate to fix failures (in the code or in terms of revised unit tests to reflect fresh code designs)
 4. Document release changes in the **CHANGELOG.md**
-5. Update the **`[Tool Poetry]version =`** field in the **pyprojects.yaml**, e.g. "3.9.5"
+5. Update the **`[Tool Poetry]version =`** field in the **pyprojects.yaml**, e.g. "4.0.0"
 6. Run **`poetry update`** (preferably within  **`poetry shell`**)
 7. Commit or pull request merge all files (including the **poetry.lock** file) to **master**
-8. Add the equivalent Git **tag** to **master**. This should be the Semantic Version string from step 4 with an added 'v' prefix, i.e. "v3.9.5".
+8. Add the equivalent Git **tag** to **master**. This should be the Semantic Version string from step 4 with an added 'v' prefix, i.e. "v4.0.0".
 9. Push **master** to remote (if not already done with by a pull request in step 6).
 10.  Check if Git Actions for testing and documentation complete successfully.
-11. Create the release using the same release tag, i.e. "v3.9.5".
-12. Check if Git Action for package deployment is successful and check if the new version (i.e. "3.9.5") is now visible on **[pypy.org](https://pypi.org/search/?q=reasoner-validator)**
+11. Create the release using the same release tag, i.e. "v4.0.0".
+12. Check if Git Action for package deployment is successful and check if the new version (i.e. "4.0.0") is now visible on **[pypy.org](https://pypi.org/search/?q=reasoner-validator)**

README.md

@@ -53,7 +53,7 @@ poetry install --all-extras
 
 A local script **`trapi_validator.py`** is available to run TRAPI Response validation against either a PK (UUID)
 indexed query result of the Biomedical Knowledge Translator "Autonomous Relay System" (ARS), a local JSON Response
-text file or a locally triggered _ad hoc_ query Request against an directly specified TRAPI endpoint.
+text file or a locally triggered _ad hoc_ query Request against a directly specified TRAPI endpoint.
 
 Note that it is best run within a **`poetry shell`** created by **`poetry install`**.
 
@@ -184,8 +184,8 @@ As an example of the kind of output to expect, if one posts the following TRAPI
 
 ```json
 {
-  "trapi_version": "1.4.1",
-  "biolink_version": "3.5.0",
+  "trapi_version": "1.4.2",
+  "biolink_version": "4.1.5",
   "response": {
       "message": {
         "query_graph": {
@@ -227,37 +227,39 @@ one should typically get a response body something like the following JSON valid
 
 ```json
 {
-  "trapi_version": "1.4.1",
-  "biolink_version": "3.2.1",
   "messages": {
-    "_comment": "some categories of messages may be absent, hence, empty dictionaries",
-    "critical": {},
-    "errors": {
-      "error.knowledge_graph.node.category.missing": {
-          "_comment": "source scope of the validation error ('global' or some knowledge source path string)",
-          "global": {
-              "_comment": "this message template does not have any additional parameters other than identifier hence it just has the unique identifier  value as a dictionary key, with associated value None",
-               "MONDO:0005148": null            
+    "Validate TRAPI Response": {
+      "Standards Test": {
+        "info": {
+          "info.query_graph.edge.predicate.mixin": {
+            "global": {
+              "biolink:treats": [
+                {
+                  "edge_id": "drug[biolink:Drug]--['biolink:treats']->type-2 diabetes[None]"
+                }
+              ]
+            }
           }
-        }
-    },
-    "warnings": {
-      "_comment": "validation code",
-      "warning.knowledge_graph.node.unmapped_prefix": {
-          "global": {
-              "_comment": "template identifier field value",
-              "CHEBI:6801": [  
-                  {
-                    "_comment": "additional message template field values, if applicable",
-                    "categories": "['biolink:Drug']"  
-                  }
-              ]       
+        },
+        "skipped": {},
+        "warning": {},
+        "error": {
+          "error.query_graph.edge.predicate.invalid": {
+            "global": {
+              "biolink:treats": [
+                {
+                  "edge_id": "drug[biolink:Drug]--['biolink:treats']->type-2 diabetes[None]"
+                }
+              ]
+            }
           }
-
-        }
-    },
-    "information": {},
-  }
+        },
+        "critical": {}
+      }
+    }
+  },
+  "trapi_version": "v1.4.2",
+  "biolink_version": "4.1.5"
 }
 ```
 
@@ -268,6 +270,8 @@ To minimize redundancy in validation messages, messages are uniquely indexed in
 
 ### OpenTelemetry and Jaeger
 
+NOTE: OpenTelemetry is temporarily disabled in this code release (to be updated later)
+
 The web service may be monitored for OpenTelemetry by setting an environment variable **TELEMETRY_ENDPOINT**  to a suitable trace collecting endpoint in an application like [Jaeger](https://www.jaegertracing.io/) (see also the [Translator SRI Jaeger-Demo](https://github.com/TranslatorSRI/Jaeger-demo)).
 
 **Note:** the current system Docker (Compose) design only supports OpenTemplate tracing using the internal Jaeger container and may require further refinements to enable use of an external telemetry collector.

api/main.py

@@ -15,8 +15,8 @@
 from reasoner_validator.versioning import get_latest_version
 from reasoner_validator.validator import TRAPIResponseValidator
 
-telemetry_endpoint: Optional[str] = getenv("TELEMETRY_ENDPOINT")
-print(f"Telemetry endpoint: {str(telemetry_endpoint)}", file=stderr)
+# telemetry_endpoint: Optional[str] = getenv("TELEMETRY_ENDPOINT")
+# print(f"Telemetry endpoint: {str(telemetry_endpoint)}", file=stderr)
 
 SERVICE_NAME = "reasoner-validator"
 
@@ -25,15 +25,15 @@
 
 app = FastAPI(title=SERVICE_NAME)
 
-if telemetry_endpoint:
-    from api.rvtelemetry import instrument
-
-    # call instrumentation code
-    instrument(
-        app=app,
-        service_name=SERVICE_NAME,
-        endpoint=telemetry_endpoint
-    )
+# if telemetry_endpoint:
+#     from api.rvtelemetry import instrument
+#
+#     # call instrumentation code
+#     instrument(
+#         app=app,
+#         service_name=SERVICE_NAME,
+#         endpoint=telemetry_endpoint
+#     )
 
 
 # Dictionary of validation context identifying the  ARA and KP

docker-compose.yaml

@@ -7,11 +7,11 @@ services:
       - TELEMETRY_ENDPOINT=http://jaeger:4318/v1/traces
     ports:
     - "80:80"
-  jaeger:
-    image: jaegertracing/all-in-one:latest
-    ports:
-      - "16686:16686"
-      - "4318:4318"
-      - "6831:6831"
-    environment:
-      - LOG_LEVEL=debug
+#  jaeger:
+#    image: jaegertracing/all-in-one:latest
+#    ports:
+#      - "16686:16686"
+#      - "4318:4318"
+#      - "6831:6831"
+#    environment:
+#      - LOG_LEVEL=debug

docs/validation_codes_dictionary.md

@@ -1146,3 +1146,13 @@
 
 **Description:** TRAPI Messages in Knowledge Graphs can have 'mixin' attribute type identifiers, when the mode of validation is 'non-strict'.
 
+## Skipped Test
+
+### skipped.test
+
+**Message:** For reason indicated in the identifier
+
+**Context:** identifier
+
+**Description:** Test was skipped for the reason documented in the 'identifier' string value.
+