models t5 large

t5-large

Overview

The developers of the Text-To-Text Transfer Transformer (T5) write:

With T5, we propose reframing all NLP tasks into a unified text-to-text-format where the input and output are always text strings, in contrast to BERT-style models that can only output either a class label or a span of the input. Our text-to-text framework allows us to use the same model, loss function, and hyperparameters on any NLP task.

T5-Large is the checkpoint with 770 million parameters.

Training Details

Training Data

The model is pre-trained on the Colossal Clean Crawled Corpus (C4), which was developed and released in the context of the same research paper as T5.

The model was pre-trained on a on a multi-task mixture of unsupervised (1.) and supervised tasks (2.). Thereby, the following datasets were being used for (1.) and (2.):

1. Datasets used for Unsupervised denoising objective:

• C4
• Wiki-DPR

2. Datasets used for Supervised text-to-text language modeling objective

• Sentence acceptability judgment
  • CoLA Warstadt et al., 2018
• Sentiment analysis
  • SST-2 Socher et al., 2013
• Paraphrasing/sentence similarity
  • MRPC Dolan and Brockett, 2005
  • STS-B Ceret al., 2017
  • QQP Iyer et al., 2017
• Natural language inference
  • MNLI Williams et al., 2017
  • QNLI Rajpurkar et al.,2016
  • RTE Dagan et al., 2005
  • CB De Marneff et al., 2019
• Sentence completion
  • COPA Roemmele et al., 2011
• Word sense disambiguation
  • WIC Pilehvar and Camacho-Collados, 2018
• Question answering
  • MultiRC Khashabi et al., 2018
  • ReCoRD Zhang et al., 2018
  • BoolQ Clark et al., 2019

Training Procedure

In their abstract, the model developers write:

In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks.

The framework introduced, the T5 framework, involves a training procedure that brings together the approaches studied in the paper. See the research paper for further details.

Evaluation Results

For full results for T5-Large, see the research paper, Table 14.

Testing Data, Factors & Metrics

The developers evaluated the model on 24 tasks, see the research paper for full details.

Sample inputs and outputs

Sample input

{
    "input_data": [
        "translate English to French: Life is so beautiful, once you learn how to live with it",
        "translate English to German: Berlin is the capital of Germany"
    ]
}

Sample output

[
  "La vie est si belle, une fois qu'on apprend à vivre avec elle",
  "Berlin ist die Hauptstadt Deutschlands"
]

Version: 13

Tags

Preview license : apache-2.0 task : text-translation huggingface_model_id : t5-large inference_compute_allow_list : ['Standard_DS4_v2', 'Standard_D8a_v4', 'Standard_D8as_v4', 'Standard_DS5_v2', 'Standard_D16a_v4', 'Standard_D16as_v4', 'Standard_D32a_v4', 'Standard_D32as_v4', 'Standard_D48a_v4', 'Standard_D48as_v4', 'Standard_D64a_v4', 'Standard_D64as_v4', 'Standard_D96a_v4', 'Standard_D96as_v4', 'Standard_FX4mds', 'Standard_F8s_v2', 'Standard_FX12mds', 'Standard_F16s_v2', 'Standard_F32s_v2', 'Standard_F48s_v2', 'Standard_F64s_v2', 'Standard_F72s_v2', 'Standard_FX24mds', 'Standard_FX36mds', 'Standard_FX48mds', 'Standard_E4s_v3', 'Standard_E8s_v3', 'Standard_E16s_v3', 'Standard_E32s_v3', 'Standard_E48s_v3', 'Standard_E64s_v3', 'Standard_NC4as_T4_v3', 'Standard_NC6s_v3', 'Standard_NC8as_T4_v3', 'Standard_NC12s_v3', 'Standard_NC16as_T4_v3', 'Standard_NC24s_v3', 'Standard_NC64as_T4_v3', 'Standard_NC24ads_A100_v4', 'Standard_NC48ads_A100_v4', 'Standard_NC96ads_A100_v4', 'Standard_ND96asr_v4', 'Standard_ND96amsr_A100_v4', 'Standard_ND40rs_v2']

View in Studio: https://ml.azure.com/registries/azureml/models/t5-large/version/13

License: apache-2.0

Properties

SHA: 150ebc2c4b72291e770f58e6057481c8d2ed331a

inference-min-sku-spec: 8|0|28|56

inference-recommended-sku: Standard_DS4_v2, Standard_D8a_v4, Standard_D8as_v4, Standard_DS5_v2, Standard_D16a_v4, Standard_D16as_v4, Standard_D32a_v4, Standard_D32as_v4, Standard_D48a_v4, Standard_D48as_v4, Standard_D64a_v4, Standard_D64as_v4, Standard_D96a_v4, Standard_D96as_v4, Standard_FX4mds, Standard_F8s_v2, Standard_FX12mds, Standard_F16s_v2, Standard_F32s_v2, Standard_F48s_v2, Standard_F64s_v2, Standard_F72s_v2, Standard_FX24mds, Standard_FX36mds, Standard_FX48mds, Standard_E4s_v3, Standard_E8s_v3, Standard_E16s_v3, Standard_E32s_v3, Standard_E48s_v3, Standard_E64s_v3, Standard_NC4as_T4_v3, Standard_NC6s_v3, Standard_NC8as_T4_v3, Standard_NC12s_v3, Standard_NC16as_T4_v3, Standard_NC24s_v3, Standard_NC64as_T4_v3, Standard_NC24ads_A100_v4, Standard_NC48ads_A100_v4, Standard_NC96ads_A100_v4, Standard_ND96asr_v4, Standard_ND96amsr_A100_v4, Standard_ND40rs_v2

languages: en, fr, ro, de