PR: correct discrepancies with the old bot

.prettierignore

@@ -0,0 +1,2 @@
+output.html
+tests/__mocks__/results/output-reward-split.html

README.md

@@ -1,12 +1,11 @@
 # `@ubiquibot/conversation-rewards`
 
-This is intended to be the proper implementation of comment incentives, based on our learnings from the first go-around. 
+As of 28 February: test driven development to aggregate all necessary information based on a URL to an issue.
 
-As of 28 February: test driven development to aggregate all necessary information based on a URL to an issue. 
-- pass in closed as complete issue URL and receive all the timeline events and activities of all humans who helped close the issue as complete. 
-- most importantly: this can inherit bot authentication and link pull requests to issues in private repositories. 
+- pass in closed as complete issue URL and receive all the timeline events and activities of all humans who helped close the issue as complete.
+- most importantly: this can inherit bot authentication and link pull requests to issues in private repositories.
 
-Be sure to review all `*.test.*` files for implementation details. 
+Be sure to review all `*.test.*` files for implementation details.
 
 ## Data structure
 
@@ -36,8 +35,9 @@ Be sure to review all `*.test.*` files for implementation details.
           },
           "reward": 0.8,
           "relevance": 0.5
+        }
       }
-    }]
+    ]
   }
 }
 ```
@@ -48,70 +48,80 @@ Reward formula: `((count * wordValue) * (score * formattingMultiplier) * n) * re
 
 Here is a possible valid configuration to enable this plugin. See [these files](./src/configuration) for more details.
 
-
 ```yaml
 plugin: ubiquibot/conversation-rewards
 with:
-    evmNetworkId: 100
-    evmPrivateEncrypted: "encrypted-key"
-    erc20RewardToken: "0xe91D153E0b41518A2Ce8Dd3D7944Fa863463a97d"
-    dataCollection:
-      maxAttempts: 10
-      delayMs: 10000
-    incentives:
-      requirePriceLabel: true
-      contentEvaluator:
-      userExtractor:
-        redeemTask: true
-      dataPurge:
-      formattingEvaluator:
-        scores:
-          br: 0
-          code: 1
-          p: 1
-          em: 0
-          img: 0
-          strong: 0
-          blockquote: 0
-          h1: 1
-          h2: 1
-          h3: 1
-          h4: 1
-          h5: 1
-          h6: 1
-          a: 1
-          li: 1
-          td: 1
-          hr: 0
+  evmNetworkId: 100
+  evmPrivateEncrypted: "encrypted-key"
+  erc20RewardToken: "0xe91D153E0b41518A2Ce8Dd3D7944Fa863463a97d"
+  dataCollection:
+    maxAttempts: 10
+    delayMs: 10000
+  incentives:
+    requirePriceLabel: true
+    contentEvaluator:
+      multipliers:
+        - select: [ISSUE_SPECIFICATION]
+          relevance: 1
+        - select: [PULL_AUTHOR]
+          relevance: 1
+        - select: [PULL_ASSIGNEE]
+          relevance: 1
+        - select: [PULL_COLLABORATOR]
+          relevance: 1
+        - select: [PULL_CONTRIBUTOR]
+          relevance: 1
+    userExtractor:
+      redeemTask: true
+    dataPurge:
+    formattingEvaluator:
+      scores:
+        br: 0
+        code: 1
+        p: 1
+        em: 0
+        img: 0
+        strong: 0
+        blockquote: 0
+        h1: 1
+        h2: 1
+        h3: 1
+        h4: 1
+        h5: 1
+        h6: 1
+        a: 1
+        li: 1
+        td: 1
+        hr: 0
         multipliers:
-          - select: [ ISSUE_SPECIFICATION ]
+          - select: [ISSUE_SPECIFICATION]
             formattingMultiplier: 1
             wordValue: 0.1
-          - select: [ ISSUE_AUTHOR ]
+          - select: [ISSUE_AUTHOR]
             formattingMultiplier: 1
             wordValue: 0.2
-          - select: [ ISSUE_ASSIGNEE ]
+          - select: [ISSUE_ASSIGNEE]
             formattingMultiplier: 0
             wordValue: 0
-          - select: [ ISSUE_COLLABORATOR ]
+          - select: [ISSUE_COLLABORATOR]
             formattingMultiplier: 1
             wordValue: 0.1
-          - select: [ ISSUE_CONTRIBUTOR ]
+          - select: [ISSUE_CONTRIBUTOR]
             formattingMultiplier: 0.25
             wordValue: 0.1
-          - select: [ PULL_SPECIFICATION ]
+          - select: [PULL_SPECIFICATION]
             formattingMultiplier: 0
             wordValue: 0
-          - select: [ PULL_AUTHOR ]
+          - select: [PULL_AUTHOR]
             formattingMultiplier: 2
             wordValue: 0.2
-          - select: [ PULL_ASSIGNEE ]
+          - select: [PULL_ASSIGNEE]
             formattingMultiplier: 1
             wordValue: 0.1
-          - select: [ PULL_COLLABORATOR ]
+          - select: [PULL_COLLABORATOR]
             formattingMultiplier: 1
             wordValue: 0.1
-          - select: [ PULL_CONTRIBUTOR ]
+          - select: [PULL_CONTRIBUTOR]
             formattingMultiplier: 0.25
             wordValue: 0.1
       permitGeneration:

src/configuration/content-evaluator-config.ts

@@ -1,5 +1,40 @@
 import { Static, Type } from "@sinclair/typebox";
+import { commentType } from "./formatting-evaluator-config";
 
-export const contentEvaluatorConfigurationType = Type.Object({});
+export const contentEvaluatorConfigurationType = Type.Object({
+  /**
+   * Multipliers applied to different types of comments
+   */
+  multipliers: Type.Array(
+    Type.Object({
+      select: Type.Array(commentType),
+      relevance: Type.Optional(Type.Number()),
+    }),
+    {
+      default: [
+        {
+          select: ["ISSUE_SPECIFICATION"],
+          relevance: 1,
+        },
+        {
+          select: ["PULL_AUTHOR"],
+          relevance: 1,
+        },
+        {
+          select: ["PULL_ASSIGNEE"],
+          relevance: 1,
+        },
+        {
+          select: ["PULL_COLLABORATOR"],
+          relevance: 1,
+        },
+        {
+          select: ["PULL_CONTRIBUTOR"],
+          relevance: 1,
+        },
+      ],
+    }
+  ),
+});
 
 export type ContentEvaluatorConfiguration = Static<typeof contentEvaluatorConfigurationType>;

src/configuration/formatting-evaluator-config.ts

@@ -1,7 +1,7 @@
 import { Static, Type } from "@sinclair/typebox";
 import { CommentAssociation, CommentKind, CommentType } from "./comment-types";
 
-const type = Type.Union(
+export const commentType = Type.Union(
   Object.keys(CommentKind).flatMap((kind) =>
     Object.keys(CommentAssociation).map((association) => Type.Literal(`${kind}_${association}` as CommentType))
   )
@@ -13,7 +13,7 @@ export const formattingEvaluatorConfigurationType = Type.Object({
    */
   multipliers: Type.Array(
     Type.Object({
-      select: Type.Array(type),
+      select: Type.Array(commentType),
       formattingMultiplier: Type.Number(),
       wordValue: Type.Number(),
     }),

src/data-collection/collect-linked-pulls.ts

@@ -14,7 +14,8 @@ export async function collectLinkedMergedPulls(issue: IssueParams) {
     // Works on multiple linked issues, and matches #<number> or URL patterns
     const linkedIssueRegex =
       /\b(?:Close(?:s|d)?|Fix(?:es|ed)?|Resolve(?:s|d)?):?\s+(?:#(\d+)|https?:\/\/(?:www\.)?github\.com\/(?:[^/\s]+\/[^/\s]+\/(?:issues|pull)\/(\d+)))\b/gi;
-    const linkedPrUrls = event.source.issue.body.match(linkedIssueRegex);
+    // We remove the comments as they should not be part of the linked pull requests
+    const linkedPrUrls = event.source.issue.body.replace(/<!--[\s\S]+-->/, "").match(linkedIssueRegex);
     if (!linkedPrUrls) {
       return false;
     }
@@ -33,6 +34,7 @@ export async function collectLinkedMergedPulls(issue: IssueParams) {
             linkedRepo.owner === issue.owner;
         }
       }
+      if (isClosingPr) break;
     }
     return isGitHubLinkEvent(event) && event.source.issue.pull_request?.merged_at && isClosingPr;
   });

src/parser/content-evaluator-module.ts

@@ -9,13 +9,37 @@ import {
 import { IssueActivity } from "../issue-activity";
 import { GithubCommentScore, Module, Result } from "./processor";
 import { Value } from "@sinclair/typebox/value";
+import { commentEnum, CommentType } from "../configuration/comment-types";
+import logger from "../helpers/logger";
+import openAiRelevanceResponseSchema, { RelevancesByOpenAi } from "../types/openai-type";
 
 /**
  * Evaluates and rates comments.
  */
 export class ContentEvaluatorModule implements Module {
   readonly _openAi = new OpenAI({ apiKey: OPENAI_API_KEY });
   readonly _configuration: ContentEvaluatorConfiguration | null = configuration.incentives.contentEvaluator;
+  private readonly _fixedRelevances: { [k: string]: number } = {};
+
+  _getEnumValue(key: CommentType) {
+    let res = 0;
+
+    key.split("_").forEach((value) => {
+      res |= Number(commentEnum[value as keyof typeof commentEnum]);
+    });
+    return res;
+  }
+
+  constructor() {
+    if (this._configuration?.multipliers) {
+      this._fixedRelevances = this._configuration.multipliers.reduce((acc, curr) => {
+        return {
+          ...acc,
+          [curr.select.reduce((a, b) => this._getEnumValue(b) | a, 0)]: curr.relevance,
+        };
+      }, {});
+    }
+  }
 
   get enabled(): boolean {
     if (!Value.Check(contentEvaluatorConfigurationType, this._configuration)) {
@@ -48,66 +72,91 @@ export class ContentEvaluatorModule implements Module {
 
   async _processComment(comments: Readonly<GithubCommentScore>[], specificationBody: string) {
     const commentsWithScore: GithubCommentScore[] = [...comments];
-    const commentsBody = commentsWithScore.map((comment) => comment.content);
-    const relevance = await this._evaluateComments(specificationBody, commentsBody);
 
-    if (relevance.length !== commentsWithScore.length) {
-      console.error("Relevance / Comment length mismatch! Skipping.");
-      return [];
+    // exclude comments that have fixed relevance multiplier. e.g. review comments = 1
+    const commentsToEvaluate: { id: number; comment: string }[] = [];
+    for (let i = 0; i < commentsWithScore.length; i++) {
+      const currentComment = commentsWithScore[i];
+      if (!this._fixedRelevances[currentComment.type]) {
+        commentsToEvaluate.push({
+          id: currentComment.id,
+          comment: currentComment.content,
+        });
+      }
     }
 
-    for (let i = 0; i < relevance.length; i++) {
+    const relevancesByAI = await this._evaluateComments(specificationBody, commentsToEvaluate);
+
+    if (Object.keys(relevancesByAI).length !== commentsToEvaluate.length) {
+      console.error("Relevance / Comment length mismatch! \nWill use 1 as relevance for missing comments.");
+    }
+
+    for (let i = 0; i < commentsWithScore.length; i++) {
       const currentComment = commentsWithScore[i];
-      const currentRelevance = relevance[i];
+      let currentRelevance = 1; // For comments not in fixed relevance types and missed by OpenAI evaluation
+
+      if (this._fixedRelevances[currentComment.type]) {
+        currentRelevance = this._fixedRelevances[currentComment.type];
+      } else if (!isNaN(relevancesByAI[currentComment.id])) {
+        currentRelevance = relevancesByAI[currentComment.id];
+      }
+
       const currentReward = new Decimal(currentComment.score?.reward || 0);
       currentComment.score = {
         ...(currentComment.score || {}),
-        relevance: currentRelevance.toNumber(),
+        relevance: new Decimal(currentRelevance).toNumber(),
         reward: currentReward.mul(currentRelevance).toNumber(),
       };
     }
 
     return commentsWithScore;
   }
 
-  async _evaluateComments(specification: string, comments: string[]): Promise<Decimal[]> {
+  async _evaluateComments(
+    specification: string,
+    comments: { id: number; comment: string }[]
+  ): Promise<RelevancesByOpenAi> {
     const prompt = this._generatePrompt(specification, comments);
 
+    const response: OpenAI.Chat.ChatCompletion = await this._openAi.chat.completions.create({
+      model: "gpt-4o",
+      response_format: { type: "json_object" },
+      messages: [
+        {
+          role: "system",
+          content: prompt,
+        },
+      ],
+      temperature: 1,
+      max_tokens: 128,
+      top_p: 1,
+      frequency_penalty: 0,
+      presence_penalty: 0,
+    });
+
+    const rawResponse = String(response.choices[0].message.content);
+    logger.info(`OpenAI raw response: ${rawResponse}`);
+
+    const jsonResponse = JSON.parse(rawResponse);
+
     try {
-      const response: OpenAI.Chat.ChatCompletion = await this._openAi.chat.completions.create({
-        model: "gpt-4o",
-        messages: [
-          {
-            role: "system",
-            content: prompt,
-          },
-        ],
-        temperature: 1,
-        max_tokens: 128,
-        top_p: 1,
-        frequency_penalty: 0,
-        presence_penalty: 0,
-      });
-
-      const rawResponse = String(response.choices[0].message.content);
-      const parsedResponse = JSON.parse(rawResponse) as number[];
-      return parsedResponse.map((o) => new Decimal(o));
-    } catch (error) {
-      console.error(`Failed to evaluate comment`, error);
-      return [];
+      const relevances = Value.Decode(openAiRelevanceResponseSchema, jsonResponse);
+      logger.info(`Relevances by OpenAI: ${JSON.stringify(relevances)}`);
+      return relevances;
+    } catch (e) {
+      logger.error(`Invalid response type received from openai while evaluating: ${jsonResponse} \n\nError: ${e}`);
+      throw new Error("Error in evaluation by OpenAI.");
     }
   }
 
-  _generatePrompt(issue: string, comments: string[]) {
+  _generatePrompt(issue: string, comments: { id: number; comment: string }[]) {
     if (!issue?.length) {
       throw new Error("Issue specification comment is missing or empty");
     }
-    return `I need to evaluate the relevance of GitHub contributors' comments to a specific issue specification. Specifically, I'm interested in how much each comment helps to further define the issue specification or contributes new information or research relevant to the issue. Please provide a float between 0 and 1 to represent the degree of relevance. A score of 1 indicates that the comment is entirely relevant and adds significant value to the issue, whereas a score of 0 indicates no relevance or added value. Each contributor's comment is on a new line.\n\nIssue Specification:\n\`\`\`\n${issue}\n\`\`\`\n\nConversation:\n\`\`\`\n${comments
-      .map((comment) => comment)
-      .join(
-        "\n"
-      )}\n\`\`\`\n\n\nTo what degree are each of the comments in the conversation relevant and valuable to further defining the issue specification? Please reply with ONLY an array of float numbers between 0 and 1, corresponding to each comment in the order they appear. Each float should represent the degree of relevance and added value of the comment to the issue. The total length of the array in your response should equal exactly ${
+    return `I need to evaluate the relevance of GitHub contributors' comments to a specific issue specification. Specifically, I'm interested in how much each comment helps to further define the issue specification or contributes new information or research relevant to the issue. Please provide a float between 0 and 1 to represent the degree of relevance. A score of 1 indicates that the comment is entirely relevant and adds significant value to the issue, whereas a score of 0 indicates no relevance or added value. A stringified JSON is given below that contains the specification and contributors' comments. Each comment in the JSON has a unique ID and comment content. \n\n\`\`\`\n${JSON.stringify(
+      { specification: issue, comments: comments }
+    )}\n\`\`\`\n\n\nTo what degree are each of the comments in the conversation relevant and valuable to further defining the issue specification? Please reply with ONLY a JSON where each key is the comment ID given in JSON above, and the value is a float number between 0 and 1 corresponding to the comment. The float number should represent the degree of relevance and added value of the comment to the issue. The total number of properties in your JSON response should equal exactly ${
       comments.length
-    } elements.`;
+    }.`;
   }
 }