Test Suite

The test suite is software that can execute, validate and measure the compliance of an evaluator over a set of test cases.

A test case consists of the following:

• A Policy
• The state of the world
• The expected compliance report
• Description of the test case
• (optional) A Request (in case of an Access Control Scenario)

How does it work

First, it loads in the test cases. A test case has a description in RDF which instantiated looks like the following:

<urn:uuid:68023514-bfd3-435f-bf2b-e1ef93e3015e> a ex:TestCase;
    dct:title "Title of the test case.";
    ex:policy <PolicyIdentifier> ;
    ex:request <RequestIdentifier> ;
    ex:sotw <StateOfTheWorldIdentifier> ;
    ex:expectedReport <ReportIdentifier> .

In data/test_cases/ examples can be found of an instantiated test case.

The function loadTestCases is responsible for this task.

Next, each test case is evaluated using an Evaluator. That is the evaluator is given a policy, a request and the state of the world. The result (a compliance report) is then compared against the expected compliance report.

During this process, count is tracked of (i) the amount of test cases, (ii) the total amount of successful validations and (iii) the total amount of expected vs evaluated compliance report equality.

Note: currently, the state of the world and the expected compliance report are added in the serialization of a test case.

How to run the test-suite

The test-suite.ts contains the code that runs the test suite.

The directories for the loadTestCases are hardcoded constants which expect the following structure:

data/
└── ODRL
    ├── policies
    ├── requests
    ├── sotw
    └── test_cases

First, the packages need to be installed

npm install

Then, run the test suite:

npx ts-node demo/test-suite.ts

Note: only Nodejs v18.18.2 and npm 9.8.1 is tested

Following code shows how to run the test suite.

import { ComplianceReportComparator, runTestSuite, TestCase, TestCaseEvaluation, TestCaseEvaluator} from "./src";
import { ODRLEvaluator } from "./ODRL-Evaluator";

async function main() {

    const testCases: TestCase[] = []; // a list of Test Cases conforming to the TestCase interface
    const odrlEvaluator = new ODRLEvaluator();
    const comparison = ComplianceReportComparator.simple; // A comparison metric used to compare an expected and evaluated compliance report
    const testCaseEvaluator = new TestCaseEvaluator(odrlEvaluator, comparison); 

    // running the test suite, which does both the evaluations on all test cases and the comparison
    const evaluations: TestCaseEvaluation[] = await runTestSuite(testCases, testCaseEvaluator);
}
main()

Result of the test suite

Engine: ODRLEngineMultipleSteps
Evaluator: ODRLEvaluator
Comparison metric: Rule-Activation-similarity
There were 67 ODRL evaluations correct compared to the Expected Compliance Report out of 68.

Number	Test case ID	Scenario	Comparison status	ODRL Evaluator Support
001	urn:uuid:e2123eb7-0707-4f24-bcc0-9d61dd9088a9	Any request results into yes (Alice Request).	true	true
002	urn:uuid:bffd9ee8-63a6-4b50-ab68-26709ce95bdc	Any request results into yes (Bob Request).	true	true
003	urn:uuid:debea3a5-056b-464c-bcca-95ab4678a27b	Any request results into yes (Bob Request Write).	true	true
004	urn:uuid:00a20e06-50db-4ae4-a53e-18ffeccde8c8	Any request results into no (Alice Request).	true	true
005	urn:uuid:a1c58c22-c170-483f-b05a-5b6ae4e632f5	Any request results into no (Bob Request).	true	true
006	urn:uuid:f588a33b-9346-4ea5-9d3a-22fd215e50a6	Any request results into no (Bob Request Write).	true	true
007	urn:uuid:b68298ee-f9d5-4f1c-a856-7a011cc2af69	Any request to use results into yes (Alice Request).	true	true
008	urn:uuid:dfe139c2-02c9-40b1-a8ae-e61cc82a768a	Any request to use results into yes (Bob Request).	true	true
009	urn:uuid:f2e067fb-c1df-4266-9b23-6c60ccdbfcb0	Any request to use results into yes (Bob Request Write).	true	true
010	urn:uuid:5e23604d-f6a0-464f-a6b3-eb04fe97b908	Any request to use results into yes (Alice Request Sell).	true	true
011	urn:uuid:f152953e-c7c6-4a47-a7e5-2522210d5627	Any request to use results into no (Alice Request).	true	true
012	urn:uuid:d9c381f3-76db-4e1e-801b-380c8e47722d	Any request to use results into no (Bob Request).	true	true
013	urn:uuid:eed06a7c-055e-4789-bdbc-40365b2a26d4	Any request to use results into no (Bob Request Write).	true	true
014	urn:uuid:687781a9-409d-4ae6-89e5-3a64bdb5bda7	Any request to use results into no (Alice Request Sell).	true	true
015	urn:uuid:a03acb38-73b3-4b09-a1f9-d9dd21bae0ef	Any request from Alice to use returns into yes (Alice Request).	true	true
016	urn:uuid:296a3e9d-af68-4d1c-8acd-f2958c3d7a8a	Any request from Alice to use returns into yes (Bob Request).	true	true
017	urn:uuid:ab740e02-2f2c-4f50-bccd-9884e96b679e	Any request from Alice to use returns into yes (Alice Request Sell).	true	true
018	urn:uuid:95dd195c-0e5e-440f-8517-f4298a64a019	Any request from Bob to use returns into no (Alice Request).	true	true
019	urn:uuid:790f07c5-2dc2-4782-abcc-410340abad94	Any request from Bob to use returns into no (Bob Request).	true	true
020	urn:uuid:c309ff91-5c6e-427a-bee4-4923f5f38844	Any request from Bob to use returns into no (Bob Request Sell).	true	true
021	urn:uuid:84cdbf46-afc2-42f2-9ed5-b9e87063292c	Read request from Alice returns into yes (Alice Request).	true	true
022	urn:uuid:9cbe9ea0-61aa-4e92-a711-65d677b54f55	Read request from Alice returns into yes (Alice Request Sell).	true	true
023	urn:uuid:0b40edb2-8d1c-4c0e-91a0-1c19486686bd	Read request from Alice returns into yes (Alice Request Read Y).	true	true
024	urn:uuid:623d29bc-f4e2-4f2a-80ec-2cbb2e8c6a2a	Read request from Alice returns into yes (Alice Request Write X).	true	true
025	urn:uuid:c6cc07c2-7c97-44d2-a85d-578710efefca	Read request from Alice returns into yes (Bob Request Read X).	true	true
026	urn:uuid:3ce32312-1c36-4839-b12d-e3d6884d89f0	Read request from Alice to resource X returns into yes (Alice Request Read X).	true	true
027	urn:uuid:113d9a92-fee3-42b2-b347-e8bbc8316b6f	Read request from Alice to resource X returns into yes (Alice Request Write X).	true	true
028	urn:uuid:97e3adc5-16f1-45d2-87b9-e8e50b5a61d8	Read request from Alice to resource X returns into yes (Alice Request Read Y).	true	true
029	urn:uuid:2e5cac06-0f27-4265-884c-9a60d5a64a61	Read request from Alice to resource X returns into yes (Bob Request Write Y).	true	true
030	urn:uuid:dde40097-f237-40cf-bb1e-03f632f6fd62	Read request from Alice to resource X returns into yes (temporal eq) (Alice Request Read X).	true	true
031	urn:uuid:f86ae64f-a207-4f5e-824d-c71253419232	Read request from Alice to resource X returns into yes (temporal eq - past) (Alice Request Read X).	true	true
032	urn:uuid:ea199623-277b-4976-9cf5-1bbccb334da3	Read request from Alice to resource X returns into yes (temporal eq - future) (Alice Request Read X).	true	true
033	urn:uuid:b55e6773-1149-4149-86c2-d5133885f99e	Read request from Alice to resource X returns into yes (temporal neq) (Alice Request Read X).	true	true
034	urn:uuid:c58bab8c-a0a7-400b-9f28-1f753482a041	Read request from Alice to resource X returns into yes (temporal neq - past) (Alice Request Read X).	true	true
035	urn:uuid:420152ac-946b-4187-9b0d-f1facebc983e	Read request from Alice to resource X returns into yes (temporal neq - future) (Alice Request Read X).	true	true
036	urn:uuid:e0cbcbbd-00de-4fb1-b4c0-fc86a1b7ec9d	Read request from Alice to resource X returns into yes (temporal lt) (Alice Request Read X).	true	true
037	urn:uuid:22ee88e3-a923-4ee0-b690-65710cf9480f	Read request from Alice to resource X returns into yes (temporal lt - past) (Alice Request Read X).	true	true
038	urn:uuid:d8c0aec1-a8d1-41dc-8847-775669a264e3	Read request from Alice to resource X returns into yes (temporal lt - future) (Alice Request Read X).	true	true
039	urn:uuid:251ab936-e085-44f1-aca6-ecbd704d1128	Read request from Alice to resource X returns into yes (temporal lteq) (Alice Request Read X).	true	true
040	urn:uuid:778fb3a6-aefe-4fd5-91e8-6455b6409707	Read request from Alice to resource X returns into yes (temporal lteq - past) (Alice Request Read X).	true	true
041	urn:uuid:44b34091-7f68-43b2-8a1b-529da71e2237	Read request from Alice to resource X returns into yes (temporal lteq - future) (Alice Request Read X).	true	true
042	urn:uuid:55606dca-592f-49da-95c3-8795e99cafca	Read request from Alice to resource X returns into yes (temporal gt) (Alice Request Read X).	true	true
043	urn:uuid:8a50fa6b-468a-4959-94d3-33458ec30cc9	Read request from Alice to resource X returns into yes (temporal gt - past) (Alice Request Read X).	true	true
044	urn:uuid:431d8e84-236b-4045-a3f2-cc36bf5a21c2	Read request from Alice to resource X returns into yes (temporal gt - future) (Alice Request Read X).	true	true
045	urn:uuid:0cf68c7f-97ba-4bc5-8f57-53e99dc7e989	Read request from Alice to resource X returns into yes (temporal gteq) (Alice Request Read X).	true	true
046	urn:uuid:2535a606-e982-49d4-b6a0-b35b81ec518d	Read request from Alice to resource X returns into yes (temporal gteq - past) (Alice Request Read X).	true	true
047	urn:uuid:43a06637-af8a-4db3-a29d-1ac8916d3291	Read request from Alice to resource X returns into yes (temporal gteq - future) (Alice Request Read X).	true	true
048	urn:uuid:1f435d44-1ba1-477c-b07e-79ee56abee34	Read request from Alice to resource X returns into yes (temporal + And (year 2024)) (Alice Request Read X).	true	true
049	urn:uuid:f7e559c1-e9e2-403a-a335-aa0ec71d2531	Read request from Alice to resource X returns into yes (temporal + And (year 2024)) (Alice Request Read X - past).	true	true
050	urn:uuid:7a7afb6f-f4f8-4bb7-90e0-4b913e58cc94	Read request from Alice to resource X returns into yes (temporal + And (year 2024)) (Alice Request Read X - future).	true	true
051	urn:uuid:11a9dfb5-a3a9-43a5-9c18-e92df3057e76	Read request from people in party collection to resource X returns into yes (Alice Request Read X).	true	true
052	urn:uuid:57856edb-2ff9-4ff8-a2b2-f56ceb8d5825	Read request from people in party collection to resource X returns into yes (Bob Request Read X).	true	true
053	urn:uuid:459e47f0-fcb8-4880-9225-5f6a3b4c155f	Read request from Alice to resource in asset collection returns into yes (Alice Request Read X).	true	true
054	urn:uuid:1e735c45-0c7a-41fc-a267-784894e6fddd	Read request from Alice to resource in asset collection returns into yes (Alice Request Read Y).	true	true
055	urn:uuid:ada99890-79ab-4893-9969-69678fe3756c	Read request from people in party collection to resource in asset collection returns into yes (Alice Request Read X).	true	true
056	urn:uuid:8c94c52e-76e4-4b36-96e5-b2782fcbf767	Read request from people in party collection to resource in asset collection returns into yes (Alice Request Read Y).	true	true
057	urn:uuid:9f6b34db-8bee-4de4-a313-afdae6f8cc3b	Read request from people in party collection to resource in asset collection returns into yes (Bob Request Read X).	true	true
058	urn:uuid:94703acc-1dfd-4689-87d2-607c81974a4c	Read request from people in party collection to resource in asset collection returns into yes (Bob Request Write Y).	true	true
059	urn:uuid:1c47df55-a53d-4740-93a8-614fc3ea9a1b	Read request from Alice to resource X returns into yes, if duty is not violated (Alice Request Read X - duty unknown).	true	false
060	urn:uuid:211654bf-86e1-42f6-8f20-cd08e3c3992e	Read request from Alice to resource X returns into yes, if duty is not violated (Alice Request Read X- duty fulfilled).	true	false
061	urn:uuid:5b372c04-ceb8-48bd-b4b8-cb6380683374	Read request from Alice to resource X returns into yes, if duty is not violated (Alice Request Read X - duty violated).	false	false
062	urn:uuid:9c052f5d-0788-4352-b957-322b432b1bfa	Read request from Alice to resource X on weekdays from 9-17 in 2024 returns into yes (Alice Request Read X).	true	true
063	urn:uuid:8677b86e-4a09-421c-a75b-f50874c8ec58	Read request from Alice to resource X on weekdays from 9-17 in 2024 returns into yes (Alice Request Read X - out of office).	true	true
064	urn:uuid:32ac9d9d-dddf-434a-a309-14bb9224ef2e	Read request from Alice to resource X on weekdays from 9-17 in 2024 returns into yes (Alice Request Read X - past).	true	true
065	urn:uuid:5ab9d056-dcb8-47f6-a4af-dd5d9320f958	Read request from people in party collection to resource X in 2024 returns into yes (Alice Request Read X).	true	false
066	urn:uuid:f9648615-bb2f-4d74-8f87-e2da0d327932	Read request from people in party collection to resource X in 2024 returns into yes (Alice Request Read X - past).	true	false
067	urn:uuid:2c997aef-a337-4503-aec7-3a93675f7b1a	Read request from people in party collection to resource X in 2024 returns into yes (Alice Request Read X - past).	true	false
068	urn:uuid:7844a3ce-40ec-4d6d-90f4-d0be27bba69c	Read request from people in party collection to resource X in 2024 returns into yes (Bob Request Write Y - past).	true	false