Skip to content

Issues: MadcowD/ell

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

149 Open 119 Closed
149 Open 119 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Dataset construction needs to be easy and there should be libraries around this, also matching parity with OpenAI evals
#382 opened Nov 13, 2024 by MadcowD
Easily be able to isolate datapoints below. a certain score etx
#381 opened Nov 13, 2024 by MadcowD
# Be able to see the outputs of the criterion function as it used in the eval.
#380 opened Nov 13, 2024 by MadcowD
Working verbose mode for @function
#379 opened Nov 13, 2024 by MadcowD
Easy comparison across many models
#378 opened Nov 13, 2024 by MadcowD
Run summaries should work with annotations and no metrics or no metrics at all
#377 opened Nov 13, 2024 by MadcowD
Build a UX which is compatible with no metrics & prompt engineering by vibes.
#376 opened Nov 13, 2024 by MadcowD
Summaries for other type of data like annotations or criterion
#374 opened Nov 13, 2024 by MadcowD
Implement a bunch of standard criteria
#373 opened Nov 13, 2024 by MadcowD
UX for showing the model is different
#372 opened Nov 13, 2024 by MadcowD
UX for api params
#371 opened Nov 13, 2024 by MadcowD
clicking the graph or tooltip opens the run
#370 opened Nov 13, 2024 by MadcowD
about multimodal
#367 opened Nov 7, 2024 by a136214808
1
default_client is ignored if model is in default registry
#366 opened Nov 6, 2024 by ppabis
3
Fix the store interface.
#360 opened Oct 31, 2024 by MadcowD
1
need to sovle a really good UX for prpompt engineering with no criterion comparion outputs etc
#359 opened Oct 31, 2024 by MadcowD
Clarity into why a currently running invocation is working or not
#358 opened Oct 31, 2024 by MadcowD
Full UX for comparing different evals across any arbitrary axis
#357 opened Oct 31, 2024 by MadcowD
Group runs more cleanly so that they are a part of an eval in the invocation view
#356 opened Oct 31, 2024 by MadcowD
Implement parsers/structured outputs to make this cleaner
#355 opened Oct 31, 2024 by MadcowD
Support failure modes in metric computation
#354 opened Oct 31, 2024 by MadcowD
Fix ell.function in general
#353 opened Oct 31, 2024 by MadcowD
Distributional view if you hover over one individual datapoint.
#352 opened Oct 31, 2024 by MadcowD
Resumption of partilaly executed evals..
#351 opened Oct 31, 2024 by MadcowD
Commit messages for evals.
#350 opened Oct 31, 2024 by MadcowD
ProTip! Type g p on any issue or pull request to go back to the pull request listing page.