PR: correct discrepancies with the old bot #55

EresDev · 2024-07-11T18:24:51Z

Resolves #26

QA

What can be improved?

Lots of file changes just for prettier formatting. Probably need some stricter check on push or commit so that all team members send formatted code.

src/parser/content-evaluator-module.ts

old method was also detecting PR first comment as spec

otherwise tests fail as some files are created in runtime

EresDev · 2024-07-15T20:04:49Z

It wasn't just h5 missing from permit comment, there were other html tags missing too. It was because of removal of all new lines from comments. New line is needed to identify some html tags for example ##### h5. If it is not on a new line, it is not a heading. Here is a before/after effect on the same comment by the fix above.

Thing also shows a new concerns though. The bot scores 1 to tags not html listed in config. Is this expected?

0x4007 · 2024-07-15T23:58:08Z

The bot scores 1 to tags not html listed in config. Is this expected?

Defaults should be set to 0 value.

…on-rewards into development

EresDev · 2024-07-23T12:02:01Z

Question:
Review comments need relevance: 1 as given in #45
Are we scoring all review comments with the code and separate comments in PR with relevance:1? Or are they only the comments with code that need relevanace:1

@gentlementlegen @0x4007

gentlementlegen · 2024-07-23T12:08:13Z

From my understanding, any comment within the pull request should have its relevance set to 1 because comments in the PR don't related to the specification itself, so doesn't make sense to evaluate them against the spec. Probably this should be configurable though the configuration file.

0x4007 · 2024-07-24T04:25:53Z

Question: Review comments need relevance: 1 as given in #45 Are we scoring all review comments with the code and separate comments in PR with relevance:1? Or are they only the comments with code that need relevanace:1

@gentlementlegen @0x4007

Good catch on this task. Why don't you address that in a new pull? Also provide a time estimate. Consider that the main work will probably be to adjust the prompt.

EresDev · 2024-07-24T12:42:32Z

@gentlementlegen I see tests failing here, and on also the development branch https://github.com/ubiquibot/conversation-rewards/actions/runs/10076181691 . Probably because of your config changes yesterday?

In the failing test, it started to change your github role between contributor and collaborator.

gentlementlegen · 2024-07-31T10:40:10Z

@EresDev I'll run lots of tests and come back to you.

gentlementlegen · 2024-08-03T14:00:51Z

@EresDev I just ran it against ubiquity/pay.ubq.fi#195 and it seems some comments are not picked up still. And the specification comment doesn't seem to be displayed properly as well, can you confirm?
On #29 it doesn't seem to properly pick up all the review comments.

Since this is urgently needed I opened a PR against your repo here: EresDevOrg#1
It will contain fixes for what I mentioned above.

gentlementlegen · 2024-08-04T18:31:21Z

Latest QA (tests conducted with my fixes as well):

Custom query params for test deployments ubiquity/pay.ubq.fi#195 (comment) (contains missing comments from the previous version)
Post a waiting message to show that the conversation is being processed #29 (comment) (counts 0x4007 comments properly, Sadaf-A is ignored since only commented with commands)
Move Filters to Bottom Bar on Portrait ubiquity/work.ubq.fi#69 (comment) (issue reward is properly split, relevance is 1 and all the comments are picked up)

fix: fixing display and collection errors

EresDev · 2024-08-05T10:06:46Z

Since this is urgently needed I opened a PR against your repo here: EresDevOrg#1
It will contain fixes for what I mentioned above.

I have merged your PR. I believe everything is fixed for this PR? @gentlementlegen

gentlementlegen · 2024-08-05T10:09:25Z

@EresDev on the issues I mentioned it should be yes. Actually right now ubiquibot-test is running will all these changes. Needing code review now!

src/parser/content-evaluator-module.ts

0x4007 · 2024-08-13T17:35:29Z

README.md

+      multipliers:
+        - select: [ISSUE_SPECIFICATION]
+          relevance: 1
+        - select: [PULL_AUTHOR]
+          relevance: 1
+        - select: [PULL_ASSIGNEE]
+          relevance: 1
+        - select: [PULL_COLLABORATOR]
+          relevance: 1
+        - select: [PULL_CONTRIBUTOR]
+          relevance: 1


Sets these to relevance 1? This is not clear to me.

The requirement was to set the relevance of issue specifications, and PR comments to 1. To implement that I added something called "Fixed Relevance" and you can specify it in config. You can add fixed relevance to any comment type, and if you do so, fixed relevance will take precedence and that comment type will not be sent to OpenAI for evaluation, and the fixed relevance will be applied to that comment type. What you see in the readme file above are fixed relevance of 1 being applied to issue specifications and PR comments.

0x4007 · 2024-08-13T17:36:15Z

README.md

+    userExtractor:
+      redeemTask: true


What is this? We should just make it a single property without nesting probably.

The nesting is die to userExtractor being its own module and made it clearer that this module is the only one using these variables.
redeemTask means the tasks are redeemable e.g. the reward can be collected.

0x4007 · 2024-08-13T17:39:58Z

src/parser/user-extractor-module.ts

+   * Checks if the comment is made by a human user, not empty, and not a command.
   */
  _checkEntryValidity(comment: (typeof IssueActivity.prototype.allComments)[0]) {
-    return comment.body && comment.user?.type === "User";
+    return comment.body && comment.user?.type === "User" && comment.body.trim()[0] !== "/";


I'm surprised that empty comments have been found in testing. I didn't know that is possible.

It should not be but according to GitHub types body can eventually be undefined.

Abuchtela

Fix set specs as 1

Abuchtela

/start

fix: set specs relevance as 1

101d176

0x4007 reviewed Jul 11, 2024

View reviewed changes

src/parser/content-evaluator-module.ts Outdated Show resolved Hide resolved

EresDev added 9 commits July 12, 2024 23:27

fix: improve specification comment identification

264869f

old method was also detecting PR first comment as spec

test: update relevance for evalulator output

395d154

test: update relevance for github comment output

63b9201

test: update relevance for permit generator output

2e5c541

chore: ignore test outputs formatting by prettier

725e060

otherwise tests fail as some files are created in runtime

test: update relevance for reward split output

b90e8c6

chore: spell fix

77ce558

fix: include missing h5 and other html tags

a22defd

test: update expected output for missing html tags

1ca68d0

EresDev added 6 commits July 18, 2024 23:58

fix: score as 0 for unlisted html tags

8a5a3ef

test: update output for unlisted html tags

4e6ae91

chore: format with prettier

c6050d4

feat: prompt openai for json response

0c063a8

Merge branch 'development' of https://github.com/ubiquibot/conversati…

cd5b4d0

…on-rewards into development

fix: set relevance as 1 on chatgpt failure

708db4e

EresDev added 5 commits July 23, 2024 21:56

refactor: identify specification of issue

4689208

fix: set review comments relevance as 1

4d574b8

chore: fixed spells

9f8815e

feat: add relevance config for content evaluator

16d6c61

docs: add relevance config to readme

6f014f4

0x4007 mentioned this pull request Jul 24, 2024

Relevance Adjustment #45

Closed

refactor: rename variables & functions

5f9cf62

gentlementlegen added 5 commits August 4, 2024 17:54

fix: linked PRs are properly collected

8ea2713

fix: the metadata is properly escaped to avoid html rendering

6b512c5

fix: content preview is properly stripped down to 64 characters

7206411

chore: removed logs

92cbfb1

chore: removed logs

de942ee

gentlementlegen added 3 commits August 5, 2024 03:44

fix: comments are ignored for the final result

ee14b90

chore: fix test split reward

238f4d0

chore: fix test github comment

0917a07

gentlementlegen mentioned this pull request Aug 5, 2024

feat!: symbol coefficients and configurable scores per group #68

Merged

Merge pull request #1 from gentlementlegen/fork/test

50a4380

fix: fixing display and collection errors

gentlementlegen approved these changes Aug 8, 2024

View reviewed changes

EresDev mentioned this pull request Aug 8, 2024

PR: evaluate code review comments with OpenAI #79

Merged

whilefoo suggested changes Aug 10, 2024

View reviewed changes

src/parser/content-evaluator-module.ts Outdated Show resolved Hide resolved

src/parser/content-evaluator-module.ts Outdated Show resolved Hide resolved

EresDev added 2 commits August 12, 2024 13:54

docs: give correct reason for default relevance

c0c5ce7

refactor: improve typebox types usage

ac4b164

whilefoo approved these changes Aug 12, 2024

View reviewed changes

gentlementlegen merged commit 369d614 into ubiquity-os-marketplace:development Aug 13, 2024
3 checks passed

ubiquibot bot mentioned this pull request Aug 13, 2024

Analyze and correct eventual discrepancies with the old bot #26

Closed

0x4007 reviewed Aug 13, 2024

View reviewed changes

gentlementlegen linked an issue Aug 22, 2024 that may be closed by this pull request

Formatting issue #76

Closed

Abuchtela reviewed Sep 25, 2024

View reviewed changes

ubiquity-os-beta bot mentioned this pull request Oct 31, 2024

Consolidate comments ubiquity-os/ubiquity-os-kernel#184

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PR: correct discrepancies with the old bot #55

PR: correct discrepancies with the old bot #55

EresDev commented Jul 11, 2024 •

edited

Loading

EresDev commented Jul 15, 2024

0x4007 commented Jul 15, 2024

EresDev commented Jul 23, 2024

gentlementlegen commented Jul 23, 2024

0x4007 commented Jul 24, 2024

EresDev commented Jul 24, 2024 •

edited

Loading

gentlementlegen commented Jul 31, 2024

gentlementlegen commented Aug 3, 2024 •

edited

Loading

gentlementlegen commented Aug 4, 2024 •

edited

Loading

EresDev commented Aug 5, 2024

gentlementlegen commented Aug 5, 2024

0x4007 Aug 13, 2024 •

edited

Loading

EresDev Aug 13, 2024 •

edited

Loading

0x4007 Aug 13, 2024

gentlementlegen Aug 14, 2024

0x4007 Aug 13, 2024

gentlementlegen Aug 14, 2024

Abuchtela left a comment

Abuchtela left a comment

PR: correct discrepancies with the old bot #55

PR: correct discrepancies with the old bot #55

Conversation

EresDev commented Jul 11, 2024 • edited Loading

QA

What can be improved?

EresDev commented Jul 15, 2024

0x4007 commented Jul 15, 2024

EresDev commented Jul 23, 2024

gentlementlegen commented Jul 23, 2024

0x4007 commented Jul 24, 2024

EresDev commented Jul 24, 2024 • edited Loading

gentlementlegen commented Jul 31, 2024

gentlementlegen commented Aug 3, 2024 • edited Loading

gentlementlegen commented Aug 4, 2024 • edited Loading

EresDev commented Aug 5, 2024

gentlementlegen commented Aug 5, 2024

0x4007 Aug 13, 2024 • edited Loading

Choose a reason for hiding this comment

EresDev Aug 13, 2024 • edited Loading

Choose a reason for hiding this comment

0x4007 Aug 13, 2024

Choose a reason for hiding this comment

gentlementlegen Aug 14, 2024

Choose a reason for hiding this comment

0x4007 Aug 13, 2024

Choose a reason for hiding this comment

gentlementlegen Aug 14, 2024

Choose a reason for hiding this comment

Abuchtela left a comment

Choose a reason for hiding this comment

Abuchtela left a comment

Choose a reason for hiding this comment

EresDev commented Jul 11, 2024 •

edited

Loading

EresDev commented Jul 24, 2024 •

edited

Loading

gentlementlegen commented Aug 3, 2024 •

edited

Loading

gentlementlegen commented Aug 4, 2024 •

edited

Loading

0x4007 Aug 13, 2024 •

edited

Loading

EresDev Aug 13, 2024 •

edited

Loading