The Impact of Demonstrations on Multilingual In-Context Learning

This repository contains code for our paper: The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis. It allows for performing in-context learning with different types of demonstrations and templates on diverse models and tasks.

Models:

• XGLM
• LLama2
• Llama2-chat
• mT0
• BLOOMZ
• OpenAI APIs (via Microsoft Azure)

Multilingual tasks:

• XNLI
• IndicXNLI
• PAWS-X
• XCOPA
• XStoryCloze
• AfriSenti
• XQuAD
• TyDiQA-GoldP
• MAFAND

Setup

Create your docker image or virtual environment based on the provided docker file (Dockerfile). Or you can use this docker image: docker pull mrzhang11/micl:default. Specifically,

python: 3.8
torch: 1.14.0
CUDA: 11.8.0

Code

• configs/: configuration files for each task
• experiments/: reproduce results in the paper (See experiments/README.md for details.)
• openicl/: modified version of OpenICL
• scripts/: bash scripts that used by experiments
• arguments.py: all input arguments
• eval_openai.py: evaluation pipeline for OpenAI models
• eval_chat.py: evaluation pipeline for Llama2-chat
• eval_llm.py: evaluation pipeline for other models, e.g., XGLM
• utils.py: utility functions, e.g., data processing

Acknowledgement

• Thanks to the OpenICL codebase and Microsoft Azure.

Contact

Miaoran Zhang ([email protected])