Generate Scenarios for isolation testing of rules. #38

brysonjbest · 2024-10-07T20:54:43Z

Add multiselect handling to klamm inputs.
Update CSV exports to UTF-8 to properly render text
Update CSV field formatting to handle multiselect option rendering.
Create a scenario generation function. This handles the generation of scenarios using the validation information from klamm. This takes in the klamm inputs and rule content, and returns a csv document of these tests with their outputs.

Implementation of this functionality required the creation of sub functions to:

Generate possible values for each field. This is currently set to manually generate 10 different options for fields that are number or date ranges. This should be enough for testing purposes, but this has been left as a hardcoded value due to potential memory issues in the deployment environment. If a user selects values on the frontend, these are passed directly to these options, and no additional options are generated. This allows users to define specific fields that should not change.
Generate possible combinations of these fields. This uses the cartesian product formula to find all possible combinations of arrays. A limit has been introduced here as well to handle possible memory concerns.
Possible combinations are then mapped to their respective fields. Additional randomness is introduced in this step to produce a wide variety of combinations instead of linearly creating them - as that was leading to some combinations being completely cut from the end result if the selection of requested items was not long enough. Duplicates are removed after generation.
The final result runs all generated scenarios through the decision of the rulecontent presented, creating a csv file with all scenarios up to the limit requested by the end user. Currently this is capped at 1000 due to memory concerns.

timwekkenbc

Looks great! Not gonna pretend I understood the underworking of every function but I get the basic idea and it runs real slick. I've noted a few minor possible adjustments

timwekkenbc · 2024-10-08T16:12:35Z

src/api/scenarioData/scenarioData.controller.ts

+      // UTF- 8 encoding with BOM
+      const bom = '\uFEFF';
+      const utf8FileContent = bom + fileContent;
+      res.setHeader('Content-Type', 'text/csv; charset=utf-8');
+      res.setHeader('Content-Disposition', `attachment; filename=${filepath.replace(/\.json$/, '.csv')}`);
+      res.status(HttpStatus.OK).send(utf8FileContent);


This is almost the same code as in two other places in this file, so you could throw it in a function

timwekkenbc · 2024-10-08T17:48:04Z

src/api/scenarioData/scenarioData.service.ts

+        const decisionResult = await this.decisionsService.runDecision(
+          ruleContent,
+          filepath,
+          formattedVariablesObject,
+          {
+            trace: true,
+          },
+        );


Probably outside the scope of this, but this has me thinking we may eventually want to update decisionsService to run decisions on bulk (like many different contexts for the same ruleContent). However, you should update this to make use of Promise.all so that it doesn't have to wait for each decision before running the next one.

timwekkenbc · 2024-10-08T18:03:12Z

src/api/scenarioData/scenarioData.service.ts

+
+    const combinations = this.generateCombinations(ruleSchema, simulationContext, testScenarioCount).slice(
+      0,
+      testScenarioCount || 100,


Better to have testScenarioCount: number = 100, in the parameters

timwekkenbc · 2024-10-08T18:04:56Z

src/api/scenarioData/scenarioData.service.ts

+    const ruleRunResults: RuleRunResults = await this.generateTestScenarios(
+      filepath,
+      ruleContent,
+      simulationContext,
+      testScenarioCount,
+    );
+
+    const keys = {
+      inputs: extractUniqueKeys(ruleRunResults, 'inputs'),
+      expectedResults: extractUniqueKeys(ruleRunResults, 'expectedResults'),
+      result: extractUniqueKeys(ruleRunResults, 'result'),
+    };
+
+    const headers = [
+      'Scenario',
+      'Results Match Expected (Pass/Fail)',
+      ...this.prefixKeys(keys.inputs, 'Input'),
+      ...this.prefixKeys(keys.expectedResults, 'Expected Result'),
+      ...this.prefixKeys(keys.result, 'Result'),
+    ];
+
+    const rows = Object.entries(ruleRunResults).map(([scenarioName, data]) => [
+      this.escapeCSVField(scenarioName),
+      `n/a`,
+      ...this.mapFields(data.inputs, keys.inputs),
+      ...this.mapFields(data.expectedResults, keys.expectedResults),
+      ...this.mapFields(data.result, keys.result),
+    ]);
+
+    return [headers, ...rows].map((row) => row.join(',')).join('\n');


Can we wrap this whole thing in a try/catch, make sure we're throwing errors in the above functions it uses, and then throw an error here too. This way we could return the proper error to the user in scenarioData.controller if we wanted.

timwekkenbc · 2024-10-08T18:09:47Z

src/utils/csv.ts

+ * @param arrays The arrays to generate the product from.
+ * @returns The generated product.
+ */
+export const cartesianProduct = (arrays: any[][]): any[][] => {


Should be able to use a generic type "" instead of "any". <T>(arrays: T[][]): T[][]

timwekkenbc · 2024-10-08T18:10:35Z

src/utils/csv.ts

+ * @param limit The maximum number of combinations to generate.
+ * @returns The generated product.
+ */
+export const complexCartesianProduct = (arrays: any[][], limit: number = 3000): any[][] => {


Should be able to use a generic type "" instead of "any".

export const complexCartesianProduct = <T>(arrays: T[][], limit: number = 3000): T[][] => { const result: T[][] = [];

brysonjbest added 6 commits October 1, 2024 16:36

Draft scenario generation of test cases.

8fb7614

Add multiselect handling to test generation.

2de7d77

Update for nested scenarios test generation.

34afa3a

Add initial tests for new test generation functions.

1b753f4

Update with tests for all csv testing functions.

6e58bc5

Update with new filepath and testing.

ec2c0a1

brysonjbest changed the title ~~Feature/generate scenarios~~ Generate Scenarios for isolation testing of rules. Oct 7, 2024

brysonjbest mentioned this pull request Oct 7, 2024

Generate scenarios for isolation testing. bcgov/brm-app#35

Merged

5 tasks

brysonjbest requested a review from timwekkenbc October 7, 2024 21:04

timwekkenbc approved these changes Oct 8, 2024

View reviewed changes

brysonjbest added 2 commits October 8, 2024 13:55

Consolidate CSV processing into new functions.

1bd2b4d

Update with error handling and typing.

8470f3c

brysonjbest merged commit 949f04f into dev Oct 8, 2024
2 checks passed

brysonjbest deleted the feature/generate-scenarios branch December 19, 2024 23:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate Scenarios for isolation testing of rules. #38

Generate Scenarios for isolation testing of rules. #38

brysonjbest commented Oct 7, 2024 •

edited

Loading

timwekkenbc left a comment

timwekkenbc Oct 8, 2024

timwekkenbc Oct 8, 2024

timwekkenbc Oct 8, 2024

timwekkenbc Oct 8, 2024

timwekkenbc Oct 8, 2024

timwekkenbc Oct 8, 2024

Generate Scenarios for isolation testing of rules. #38

Generate Scenarios for isolation testing of rules. #38

Conversation

brysonjbest commented Oct 7, 2024 • edited Loading

timwekkenbc left a comment

Choose a reason for hiding this comment

timwekkenbc Oct 8, 2024

Choose a reason for hiding this comment

timwekkenbc Oct 8, 2024

Choose a reason for hiding this comment

timwekkenbc Oct 8, 2024

Choose a reason for hiding this comment

timwekkenbc Oct 8, 2024

Choose a reason for hiding this comment

timwekkenbc Oct 8, 2024

Choose a reason for hiding this comment

timwekkenbc Oct 8, 2024

Choose a reason for hiding this comment

brysonjbest commented Oct 7, 2024 •

edited

Loading