Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature enhancements and bug fixes for generative text task #2527

Open
9 tasks
kartik727 opened this issue Feb 2, 2024 · 0 comments
Open
9 tasks

Feature enhancements and bug fixes for generative text task #2527

kartik727 opened this issue Feb 2, 2024 · 0 comments
Labels
bug Something isn't working enhancement New feature or request ResponsibleAIDashboard

Comments

@kartik727
Copy link
Contributor

kartik727 commented Feb 2, 2024

A list of issues and potential enhancements for the newly added generative text task type

Bug Fixes

  • Cache not being used in the compute_genai_metrics method of the RAITextInsights class.
  • Cache does not appear to be updated from the front-end for the compute_question_answering_metrics and compute_genai_metrics methods of the RAITextInsights class.
  • Error tree currently uses the model being evaluated to compute the AI assisted metrics instead of the evaluation metric.
  • Additional text columns (like reference answers) in the input dataset not working as intended. As a result, metrics requiring reference answers (e.g., equivalence, exact match score, BLEU, etc.) are also currently not working.

Feature enhancements

  • The templates with metric definitions for AI assisted metrics is currently hard-coded into the metric computation scripts. These can be made configurable in the future.
  • Allowing the users to create new metrics by supplying their own templates with metrics definitions.
  • Error tree - show name of the metric used for training the error tree (currently show mean squared error).
  • Error tree - allow the use of other metrics for creating the error tree.
  • Implement the front-end UI for model explainer.
@kartik727 kartik727 added bug Something isn't working enhancement New feature or request ResponsibleAIDashboard labels Feb 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request ResponsibleAIDashboard
Projects
None yet
Development

No branches or pull requests

1 participant