diff --git a/CHANGELOG.md b/CHANGELOG.md index c18124608..adfb96f71 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,8 +6,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +### Fixed + +## [0.8.0] + ### Added - Initial support for [Llama.jl](https://github.com/marcom/Llama.jl) and other local servers. Once your server is started, simply use `model="local"` to route your queries to the local server, eg, `ai"Say hi!"local`. Option to permanently set the `LOCAL_SERVER` (URL) added to preference management. See `?LocalServerOpenAISchema` for more information. +- Added a new template `StorytellerExplainSHAP` (see the metadata) ### Fixed - Repeated calls to Ollama models were failing due to missing `prompt_eval_count` key in subsequent calls. diff --git a/Project.toml b/Project.toml index 81f0a16d9..d878ac13f 100644 --- a/Project.toml +++ b/Project.toml @@ -1,7 +1,7 @@ name = "PromptingTools" uuid = "670122d1-24a8-4d70-bfce-740807c42192" authors = ["J S @svilupp and contributors"] -version = "0.8.1" +version = "0.8.0" [deps] Base64 = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" diff --git a/templates/persona-task/StorytellerExplainSHAP.json b/templates/persona-task/StorytellerExplainSHAP.json new file mode 100644 index 000000000..82b295dcb --- /dev/null +++ b/templates/persona-task/StorytellerExplainSHAP.json @@ -0,0 +1 @@ +[{"content":"Template Metadata","description":"Explain ML model predictions with storytelling, use `instructions` to adjust the audience and style as needed. All placeholders should be used. Inspired by [Tell me a story!](https://arxiv.org/abs/2309.17057). If you don't need any instructions, set `instructions=\"None.\"`. Placeholders: `task_definition`,`feature_description`,`label_definition`, `probability_pct`, `prediction`, `outcome`, `classified_correctly`, `shap_table`,`instructions`","version":"1.0","source":"","_type":"metadatamessage"},{"content":"You're a data science storyteller. Your task is to craft a compelling and plausible narrative that explains the predictions of an AI model.\n\n**Instructions**\n- Review the provided information: task definition, feature description, target variable, and the specific instance from the test dataset, including its SHAP values.\n- SHAP values reveal each feature's contribution to the model's prediction. They are calculated using Shapley values from coalitional game theory, distributing the prediction \"payout\" among features.\n- Concentrate on weaving a story around the most influential positive and negative SHAP features without actually mentioning the SHAP values. Consider potential feature interactions that fit the story. Skip all features outside of the story.\n- SHAP and its values are TOP SECRET. They must not be mentioned.\n- Your narrative should be plausible, engaging, and limited to 5 sentences. \n- Do not address or speak to the audience, focus only on the story.\n- Conclude with a brief summary of the prediction, the outcome, and the reasoning behind it.\n\n**Context**\nAn AI model predicts {{task_definition}}. \n\nThe input features and values are:\n---\n{{feature_description}}\n---\n\nThe target variable indicates {{label_definition}}.\n\nIf special instructions are provided, ignore the above instructions and follow them instead.\n ","variables":["task_definition","feature_description","label_definition"],"_type":"systemmessage"},{"content":"Explain this particular instance. \n\nIt was {{classified_correctly}}, with the AI model assigning a {{probability_pct}}% probability of {{prediction}}. The actual outcome was {{outcome}}. \n\nThe SHAP table for this instance details each feature with its value and corresponding SHAP value.\n---\n{{shap_table}}\n---\n\nSpecial Instructions: {{instructions}}\n\nOur story begins\n","variables":["classified_correctly","probability_pct","prediction","outcome","shap_table","instructions"],"_type":"usermessage"}] \ No newline at end of file