text_generation_pipeline_example.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyMPTKW2dgboQJpXpHYIBCHp",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/text_generation_pipeline_example.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Text generation example\n",
        "\n",
        "This is a brief example of how to run text generation with a causal language model and `pipeline`.\n",
        "\n",
        "Install [transformers](https://huggingface.co/docs/transformers/index) python package. This will be used to load the model and tokenizer and to run generation."
      ],
      "metadata": {
        "id": "tIQ1s96UCcJW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install --quiet transformers"
      ],
      "metadata": {
        "id": "4fUBJmXHCHw-"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Import the `AutoTokenizer`, `AutoModelForCausalLM`, and `pipeline` classes. The first two support loading tokenizers and generative models from the [Hugging Face repository](https://huggingface.co/models), and the last wraps a tokenizer and a model for convenience."
      ],
      "metadata": {
        "id": "5ZRNZgRJCt6Q"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "jwyK005xCFSF"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Load a generative model and its tokenizer. You can substitute any other generative model name here (e.g. [other TurkuNLP GPT-3 models](https://huggingface.co/models?sort=downloads&search=turkunlp%2Fgpt3)), but note that Colab may have issues running larger models. "
      ],
      "metadata": {
        "id": "6QJPDe3ZC_sL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MODEL_NAME = 'TurkuNLP/gpt3-finnish-large'\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)\n",
        "model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)"
      ],
      "metadata": {
        "id": "wqTxn_QaCNjZ"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Instantiate a text generation pipeline using the tokenizer and model."
      ],
      "metadata": {
        "id": "9ADWWb77e1sY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pipe = pipeline(\n",
        "    'text-generation',\n",
        "    model=model,\n",
        "    tokenizer=tokenizer,\n",
        "    device=model.device\n",
        ")"
      ],
      "metadata": {
        "id": "0IIJzNrEe5qx"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can now call the pipeline with a text prompt; it will take care of tokenizing, encoding, generation, and decoding:"
      ],
      "metadata": {
        "id": "eAohNr1ciwaU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "output = pipe('Terve, miten menee?', max_new_tokens=25)\n",
        "\n",
        "print(output)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jWcOJkiKi5vr",
        "outputId": "11cb09ab-310d-438e-e372-ef8d7f8f66a2"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[{'generated_text': 'Terve, miten menee?”\\n”Hyvin, kiitos.”\\n”Kiva kuulla.”\\n”Kuule, minulla on sinulle asiaa.”\\n'}]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Just print the text"
      ],
      "metadata": {
        "id": "SNRMsxXOjSo0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(output[0]['generated_text'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9Op7MJ6XjahG",
        "outputId": "5d9e26dd-3e80-4663-a712-d85093fad073"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Terve, miten menee?”\n",
            "”Hyvin, kiitos.”\n",
            "”Kiva kuulla.”\n",
            "”Kuule, minulla on sinulle asiaa.”\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can also call the pipeline with any arguments that the model `generate` function supports. For details on text generation using `transformers`, see e.g. [this tutorial](https://huggingface.co/blog/how-to-generate).\n",
        "\n",
        "Example with sampling and a high `temperature` parameter to generate more chaotic output:"
      ],
      "metadata": {
        "id": "YROp3hyikXPO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "output = pipe(\n",
        "    'Terve, miten menee?',\n",
        "    do_sample=True,\n",
        "    temperature=10.0,\n",
        "    max_new_tokens=25\n",
        ")\n",
        "\n",
        "print(output[0]['generated_text'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "22QjXE88jkim",
        "outputId": "372a4fc2-e305-4034-cea3-a52b6103c24f"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Terve, miten menee? kysyi Heikki yhtäkkiä astuessaan taloon sisään kantaen kahvipöytä-tarvikkeita käsissään sisään tultuaan.\n",
            "(Ryökäle luuli varmasti hänen tulevan meille istumaan)\n",
            "\n",
            "\n"
          ]
        }
      ]
    }
  ]
}