Notebook_5_2024_answers.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/ATP_kurssi/blob/master/Notebook_5_2024_answers.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Hands-on work\n",
        "\n",
        "### Exercise 1\n",
        "\n",
        "* I (Anna) work with Finnish parliamentary speeches. Let's try to wrangle a file containing data from year 2019. NB! This file has been pre-cleaned a little to make it work here!\n",
        "\n",
        "* We will be examining the file which contains all speeches given in year 2019: https://raw.githubusercontent.com/aristila/totally-random-stuff/refs/heads/main/finparlspeeches_2019.csv\n",
        "\n",
        "\n",
        "Your tasks:\n",
        "\n",
        "1. Get the file.\n",
        "2. Check what the contents look like.\n",
        "3. One row has one speech and related metadata. How many speeches have been given this year?\n",
        "4. Some interesting columns are \"date\" (the day the speech was given), \"party\" (parliamentary faction), \"lang\" (language the speech is given in; either Finnish, Swedish, or both), and \"content\" which contains the actual speech. These columns are numbers **4, 6, 7, and 9.** Cut these four columns, (Hint: If you don't know how to cut multiple columns at once, Google it!), then save the truncated dataset as a new file called \"**speeches_2019_4c.csv**\". Use this file from now on.\n",
        "5. How many unique parties (parliamentary factions) appear in the data?\n",
        "6. Which party has given the most speeches?\n",
        "7. How many of the speeches are given only in Swedish? How many percentages is that?\n",
        "8. How many times does the word \"virus\" appear in the speeches? What about \"korona\" or \"covid\"?\n",
        "9. Get all the speeches that mention \"ilmastonmuutos\" (climate change). Which party has the most of those?\n",
        "10. On which date were there most speeches about \"ilmastonmuutos\"?\n",
        "\n",
        "EXTRA: Formulate a question of your own! What would you like to find out?"
      ],
      "metadata": {
        "id": "1APs5gVvh14Y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 1. Get the file.\n",
        "! wget 'https://raw.githubusercontent.com/aristila/totally-random-stuff/refs/heads/main/finparlspeeches_2019.csv'"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oCwCo--Th1eO",
        "outputId": "9a918932-474b-4dda-83a7-bb2b42353572"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-11-13 18:21:51--  https://raw.githubusercontent.com/aristila/totally-random-stuff/refs/heads/main/finparlspeeches_2019.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 23630196 (23M) [text/plain]\n",
            "Saving to: ‘finparlspeeches_2019.csv’\n",
            "\n",
            "finparlspeeches_201 100%[===================>]  22.54M  --.-KB/s    in 0.09s   \n",
            "\n",
            "2024-11-13 18:21:53 (239 MB/s) - ‘finparlspeeches_2019.csv’ saved [23630196/23630196]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 2. Check what the contents look like.\n",
        "! cat finparlspeeches_2019.csv | head"
      ],
      "metadata": {
        "id": "QLHnMlgbcGLJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d21545eb-5d34-49da-cf24-0050c96d943c"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "id|speaker|birth_year|party__prefLabel|speechType__prefLabel|language|date_|item__prefLabel|content\r\n",
            "s2018_1_137_001|Risikko, Paula|1960|Kansallinen Kokoomus|Puhemiehen puheenvuoro|fin|2019-01-08|Hallituksen esitys eduskunnalle laiksi työeläkevakuutusyhtiöistä annetun lain muuttamisesta ja eräiksi siihen liittyviksi laeiksi|      Lähetekeskustelua varten esitellään päiväjärjestyksen 2. asia. Puhemiesneuvosto ehdottaa, että asia lähetetään talousvaliokuntaan. Lähetekeskusteluun varataan enintään 30 minuuttia. Asian käsittelyssä noudatetaan aikataulutettujen asioiden osalta sovittuja menettelytapoja.\r\n",
            "s2018_1_137_002|Mattila, Pirkko|1964|Perussuomalaiset|Esittelypuheenvuoro|fin|2019-01-08|Hallituksen esitys eduskunnalle laiksi työeläkevakuutusyhtiöistä annetun lain muuttamisesta ja eräiksi siihen liittyviksi laeiksi|      Arvoisa rouva puhemies! Käsittelyssä on hallituksen esitys laiksi työeläkevakuutusyhtiöistä annetun lain muuttamisesta ja eräiksi siihen liittyviksi laeiksi. [Hälinää — Puhemies koputtaa] Esityksen tavoitteena on selventää, mitä institutionaalisten sijoittajien on julkisesti kerrottava siitä, miten ne käyttävät osakkeisiin perustuvia oikeuksia omistamissaan yrityksissä. Hallitus esittää, että myös lakisääteistä työeläkevakuuttamista harjoittavien eläkelaitosten omistajaohjauksen periaatteita koskevia säännöksiä selkiytetään ja täydennetään nykyisestä. Työeläkelaitosten omistajaohjauksen periaatteista ei ole säädetty kovinkaan kattavasti voimassa olevassa lainsäädännössä, vaan omistajaohjauksen periaatteiden laatiminen on jätetty pitkälti työeläkelaitoksille itselleen. Hallituksen esityksessä ehdotetaan työeläkelaitoksille riittävän kattavia ja yhtenäisiä säännöksiä. Kaikilla työeläkelaitoksilla olisi esityksen mukaan oltava vähintään yleisluontoiset omistajaohjauksen periaatteet. Lisäksi jos työeläkelaitos sijoittaa sellaisten kohdeyhteisöjen osakkeisiin, jotka ovat kaupankäynnin kohteena säännellyllä markkinalla Eta-valtiossa, työeläkelaitoksen olisi omistajaohjauksen periaatteissa kerrottava tarkempia tietoja. [Hälinää — Puhemies koputtaa] Omistajaohjauksen periaatteet olisi julkistettava maksutta työeläkelaitoksen internetsivuilla. Eläkesäätiöille ja -kassoille sallittaisiin lisäksi muu vastaava julkistamistapa. Tällä hallituksen esityksellä on asiallinen yhteys toiseen, samanaikaisesti annettuun lakiehdotukseen HE 305/2018 vp, joka liittyy Euroopan unionissa hyväksytyn osakkeenomistajien oikeuksista annetun muutosdirektiivin täytäntöönpanoon. Siksi työeläkevakuutusyhtiöistä annettuun lakiin ehdotetaan myös eräitä lähipiiritoimia ja johdon palkitsemista sekä niistä päättämistä koskevia lähinnä teknisluontoisia tarkennuksia, jotka ovat tarpeellisia muihin lakeihin, vakuutusyhtiölakiin ja osakeyhtiölakiin, ehdotettujen muutosten johdosta.\r\n",
            "s2018_1_137_003|Järvinen, Heli|1963|Vihreä liitto|Varsinainen puheenvuoro|fin|2019-01-08|Hallituksen esitys eduskunnalle laiksi työeläkevakuutusyhtiöistä annetun lain muuttamisesta ja eräiksi siihen liittyviksi laeiksi|      Arvoisa puhemies! On hyvä, että työeläkelaitosten pelisääntöjä selkeytetään. Puhumme kuitenkin lähes 200 miljardin euron... [Hälinää — Puhemies koputtaa] Puhumme lähes 200 miljardin euron kansallisomaisuudesta, joka on erinomaisen tärkeä osa meidän eläkejärjestelmämme kestävyyttä ja kantavuutta tulevaisuuteen. Tästä samasta syystä tässä salissa pitäisi käydä keskustelua lähiaikoina myös siitä, miten aiomme turvata etenkin julkisen puolen eläkkeet maailman muuttuessa.  Meillä on julkisella puolella paljon pitkäikäisiä naisia, joista aiheutuvat eläkemenot ovat suuria. Samoin siellä on paljon erityisalojen, kuten sotilaiden ja palomiesten, alemmista eläkei’istä johtuvia kuluja. Meidän on ratkaistava niiden kohtalo jatkossa, etenkin tässä tilanteessa, jossa jo nyt terveyspuolen henkilökuntaa on isossa määrin siirtynyt yksityiselle puolelle, ja riippumatta siitä, saammeko sote-uudistuksen aikaan vai emme, sama suun-taus tulee jatkumaan.  Tätä kiireellisyyttä kuvaa hyvin sekin, että mitä pitemmälle tätä päätöstä rukkaamme ja pitkitämme, sitä kalliimmaksi se yhteiskunnalle tulee. Eli meidän on viimeistään hallitusneuvotteluissa tehtävä esityksiä ja ratkaisuja tulevaisuuden suuntaviivoista, miten saamme turvattua myös julkisen alan eläkkeet tulevaisuudessa, mikä lienee vahva yhteinen tahtotilamme koko tässä salissa.\r\n",
            "s2018_1_137_004|Sarkomaa, Sari|1965|Kansallinen Kokoomus|Varsinainen puheenvuoro|fin|2019-01-08|Hallituksen esitys eduskunnalle laiksi työeläkevakuutusyhtiöistä annetun lain muuttamisesta ja eräiksi siihen liittyviksi laeiksi|      Arvoisa rouva puhemies! Tämä lakiesitys on tervetullut ja osoittaa sen, että hallitus tekee loppuun asti työtä, niin kuin eduskuntakin. Työeläkejärjestelmän toimivuuden tärkein turva on se, että Suomeen syntyy uusia työpaikkoja ja jokainen työkykyinen myöskin pääsee työhön.  Tämän hallituksen esityksen tavoitteena on todellakin lisätä lakisääteisen työeläketurvan toimeenpanosta huolehtivien työeläkelaitosten toiminnan läpinäkyvyyttä ja avoimuutta, raportoinnin oikea-aikaisuutta ja julkista luotettavuutta. Ja jotta nämä tavoitteet, todella tärkeät tavoitteet, toteutuvat — kiitos ministeri Mattilalle, että tämä esitys todellakin eduskuntaan tulee — niin meidän on todellakin säädettävä uusia tarkentavia säädöksiä omistajaohjauksen periaatteista ja ulottaa myöskin säätely työeläkevakuutusyhtiöiden ja Kevan lisäksi myös muihin työeläkelaitoksiin sekä valtion eläkerahastoihin.  Arvoisa puhemies! Haluan tässä lähetekeskustelussa vielä todeta, että todellakin nykyistä avoimempi omistajaohjaus vahvistaa osakkeenomistajan ja sijoituskohteena olevan yhtiön suhdetta ja parantaa myös yleistä sijoitussuojaa. Ja pidän erittäin hyvänä sitä, että omistajaohjauksen periaatteet ulotetaan myös niille työeläkelaitoksille, joita velvoitteet eivät tällä hetkellä todellakaan koske, ja tämä tuo todellakin sijoitustoimintaan ja markkinoille tasapuolisemmat toiminta- ja kilpailuedellytykset. Ja lopuksi vielä totean, että tuo tasapuolinen toimintaympäristö edistää rahoitusmarkkinoiden vakautta ja ennakoitavuutta ja lisää myöskin sijoittajan suojaa.  Eli pidän tätä esitystä hyvänä, mutta on tärkeää, että talousvaliokunta tämän tarkasti käy läpi.\r\n",
            "s2018_1_137_005|Juvonen, Arja|1967|Perussuomalaiset|Varsinainen puheenvuoro|fin|2019-01-08|Hallituksen esitys eduskunnalle laiksi työeläkevakuutusyhtiöistä annetun lain muuttamisesta ja eräiksi siihen liittyviksi laeiksi|      Arvoisa rouva puhemies! Selailin tätä täältä asian valmistelu-osiosta. Se on yleensä hyvä tapa ottaa kantaa myös siihen, millä tavalla asiaan perehtyneet tahot ovat asiaa kommentoineet. Huomasin, että täällä mainitaan, että esitysluonnos on lausuntokierroksen yhteydessä lähetetty valtioneuvoston kanslian yhteydessä toimivalle lainsäädännön arviointineuvostolle, mutta se ei ole kuitenkaan antanut esityksestä lausuntoaan. Mielestäni täällä eduskunnassa on ollut hyvin usein lakiesityksiä, joista on keskusteltu, onko lainvalmistelu ollut asiantuntevaa ja oikeaa ja hyvää.  Kysyisinkin teiltä, ministeri, onko tässä lakiesityksessä mielestänne vielä mitään sellaista korjattavaa ja minkä takia esimerkiksi lainsäädännön arviointineuvosto ei ole nyt tästä antanut lausuntoaan. Vaikutusten arviointihan on hyvin tärkeää ja kuuluu muun muassa arviointoneuvoston toimenkuvaan. Onko tässä jotakin sellaista? Muun muassa täältä Vastaan tai varauksellinen -osiosta löytyy huomioita tähän käsitteillä olevaan lakiin.\r\n",
            "s2018_1_137_006|Laukkanen, Antero|1958|Suomen Kristillisdemokraatit|Varsinainen puheenvuoro|fin|2019-01-08|Hallituksen esitys eduskunnalle laiksi työeläkevakuutusyhtiöistä annetun lain muuttamisesta ja eräiksi siihen liittyviksi laeiksi|      Arvoisa rouva puhemies! Täällä edustaja Järvinen oli huolissaan eläkkeiden kohdentumisesta. Kyllä nostaisin tähän myös tämän tulevien sukupolvien eläkkeiden tason, sen, miten turvataan nuorten eläkkeiden taso, nykyisten nuorten, jotka pohtivat, mahtavatko he saada eläkettä ollenkaan.  Itse tähän asiaan: Tämä säännösten tarkistaminen on mitä tervetullein asia, ja se yhtenäistää. Niin kuin täällä edustaja Sarkomaa totesi, tämä on erittäin hyvä esitys ja tuo sen tarvittavan uudistuksen ja jatkaa tätä hallituksen sujuvoittamis- ja selkeyttämislinjaa ja on siinä mielessä tervetullut.\r\n",
            "s2018_1_137_007|Essayah, Sari|1967|Suomen Kristillisdemokraatit|Varsinainen puheenvuoro|fin|2019-01-08|Hallituksen esitys eduskunnalle laiksi työeläkevakuutusyhtiöistä annetun lain muuttamisesta ja eräiksi siihen liittyviksi laeiksi|      Arvoisa rouva puhemies! Niin kuin täällä kollegat edellä ovat todenneet, työeläkevarojen huolellinen ja pitkäjänteinen hoitaminen on koko kansakunnan etu. Ja totta kai on tärkeää, että sijoitusstrategiat ja omistajaohjauksen periaatteet ovat avoimet ja läpinäkyvät.  Tässä samassa yhteydessä keskustelu on paikallaan toki myöskin siitä, millä tavalla tänä päivänä voidaan huolehtia siitä, että tulevaisuuden eläkevarat ovat kestävällä pohjalla. Ja silloin puhutaan siitä, miten me voimme varmistaa, että meillä syntyvyys on kasvussa, ja siitä, millä tavalla me voimme varmistaa sen, että ne nuoret, jotka ovat työelämässä tai työikäisiä, sinne työelämään todellakin sitten pääsevät siirtymään ja ettei tapahdu sitä, mitä tänä päivänä valitettavan usein on, että meillä suuri joukko nuoria syrjäytyy kokonaan, ja meillä näitten nuorien työkyvyttömyyseläkkeiden määrä, joissa usein on taustalla mielenterveysongelmia, on valitettavasti ollut kasvussa. Ja nämä ovat niitä asioita, joihin tuommoisen työeläkejärjestelmän sisällä on syytä kiinnittää huomiota.  On ollut paljon keskustelua myös siitä, että ihan viime aikoina myös työkyvyttömyys-eläkehakemusten määrä on kasvanut, ja on esitetty jopa arvioita, että esimerkiksi aktiivimallilla olisi saattanut olla jotakin taustaa tämän työkyvyttömyyseläkkeiden hakemusten määrän kasvussa. En tiedä, pystyykö ministeri tähän asiaan kommentoimaan tässä yhteydessä, mutta tämä on asia, mikä olisi varmaan syytä selvittää. Työeläkeyhtiöiden johto on esittänyt muun muassa huolestuneisuutta siitä, miten nyt näiden työkyvyttömyyseläkkeiden määrä ja niiden hakemusten määrä on yhtäkkiä pompsahtanut ylöspäin.  Kaiken kaikkiaan tämä lakiesitys on varmasti sellainen, joka on paikallaan ja on entisestään vahvistamassa omistajaohjauksen periaatteitten, voisiko sanoa, johdonmukaisuutta ja pitkäjänteisyyttä.\r\n",
            "s2018_1_137_008|Kankaanniemi, Toimi|1950|Perussuomalaiset|Varsinainen puheenvuoro|fin|2019-01-08|Hallituksen esitys eduskunnalle laiksi työeläkevakuutusyhtiöistä annetun lain muuttamisesta ja eräiksi siihen liittyviksi laeiksi|      Arvoisa rouva puhemies! Työeläkerahastoissa oli viime vuoden kesäkuun lopussa 202 miljardia pääomia, ja se on tietysti erittäin merkittävä summa kansantaloudessa. Eläkkeitä maksetaan noin 15 miljardia euroa ja saman verran kerätään eläkemaksuilla näihin rahastoihin vuosittain tuloja. Tämä tulovirta on siis tavallaan läpijuoksu näillä luvuilla — se, mitä tulee, menee — ja silloin avainkysymys on, miten tämä pääoma, joka on kerätty menneinä aikoina työssä olleilta ja nykyisin työssä olevilta, saadaan tuottamaan edelleen jatkossakin niin hyvin kuin se on nyt tuottanut.  Tarkistin, että vuoden 2010 lopussa eläkerahastoissa oli yhteensä 136 miljardia euroa, ja nyt siis 202 miljardia, kahdeksan vuotta myöhemmin. Se tekee noin 8 miljardia per vuosi kasvua, sijoitustuottoja, rahastoihin — se on valtava summa — ja tämän varassa meidän tuleva eläkepolitiikkamme tietysti pitkälti on. Emme varmaan voi lähteä siitä, että eläkkeitä leikattaisiin — ainakaan perussuomalaiset eivät siihen ole sinänsä valmiita, vaikka eläkekatto sinänsä voisi olla paikallaan meilläkin — mutta ei myöskään voida eläkemaksuja korottaa hallitsemattomasti, vaan täytyy toimia pitkäjänteisesti, jotta myös ne turvaavat osaltaan työllisyyttä ja elinkeinoelämän toimintamahdollisuuksia.  Tämä, että eläkevarat sijoitetaan edelleen tuottavasti ja turvaavasti, on tärkeä tehtävä sekä eläkeyhtiöille että valvovalle viranomaiselle, ja toivon, että tämä lainsäädäntö vahvistaa ja edistää tätä. Toisaalta vielä nostan esille sen, että noin puolet eläkevaroista on sijoitettu ulkomaille ja toinen puoli, suurin piirtein, kotimaahan. Olen itse pohtinut sitä, voitaisiinko päästä siihen, että eläkevaroja hyödynnettäisiin koko kansantalouden kannalta esimerkiksi niin, että sopivalla tavalla niitä voitaisiin käyttää perusinfran parantamiseen eli esimerkiksi suuriin liikennehankkeisiin. Niistä ei saada ihan sellaista suoraa tuottoa kuin saadaan sijoituksista ulkomailta, mutta niistä saadaan kansantaloudelle hyötyä, ja työllisyyden paranemisen kautta voisi tulla jopa parempi tuotto.\r\n",
            "s2018_1_137_009|Tanus, Sari|1964|Suomen Kristillisdemokraatit|Varsinainen puheenvuoro|fin|2019-01-08|Hallituksen esitys eduskunnalle laiksi työeläkevakuutusyhtiöistä annetun lain muuttamisesta ja eräiksi siihen liittyviksi laeiksi|      Arvoisa rouva puhemies! Tämä hallituksen esitys on varmasti oikein hyvä, ja on hyvä tarkemmin katsoa näitä työeläkevakuutusyhtiöihin liittyviä asioita niin omistajaohjauksen kuin julkisuuden kuin sijoittamisenkin näkökulmasta.  Meillähän ovat työeläkevakuutusrahastot todella kasvaneet merkittävästi, ja paitsi että meidän täytyy todellakin huolehtia tulevien eläkeläisten eläkkeistä, niin meidän tulisi käsittääkseni huolehtia paljon paremmin myös nykyisten eläkeläisten eläkkeistä, varsinkin siitä suuresta eläkeläisten joukosta, joka tämän päivän hyvinvointivaltiossa elää eläkeläisköyhyydessä. Meillä on todella merkittävä määrä eläkeläisiä, joiden päivittäinen toimeentulo on hyvin veitsenterällä. Ja aivan niin kuin edellinen puhuja nosti esiin, itsekin olisin halunnut nostattaa esiin myös sen, mihinkä näitä valtavia eläkerahastoja sijoitetaan. Meillähän suurin osa eläkevaroista on tällä hetkellä sijoitettu ulkomaille ja hyödyntää siellä, ja olen tässä salissa jo aiemmin lausumissani nostanut esiin sitä, voisiko aloittaa neuvotteluja siitä, että sijoitettaisiin kotimaahan isompi osa näistä rahastoista, jotka hyödyttäisivät kotimaan markkinoita ja kotimaan eläkeläisiä — esimerkiksi joihinkin sellaisiin sijoituskohteisiin, jotka todella helpottaisivat myös tämän päivän eläkeläisten jokapäiväistä toimeentuloa. Meillä on monia eri vaihtoehtoja. Ajattelisin, että kun tätä esitystä tarkastellaan ja sitä, kuinka tulevaisuudessa käytetään työeläkevakuutusyhtiöiden varoja, niin meillä olisi huomattavasti laajempi skaala käytettävissämme sijoitusmahdollisuuksia ja nimenomaan mahdollisuuksia sijoittaa myös kotimaahan sillä tavalla, että se hyödyttäisi meitä kaikkia ja tietysti ennen muuta nykyisiä ja myös tulevia eläkeläisiä.\r\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 3. One row has one speech and related metadata. How many speeches have been given this year?\n",
        "! cat finparlspeeches_2019.csv | wc -l"
      ],
      "metadata": {
        "id": "jNqRgs4IcLf7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d3eaae44-8391-42c8-b2bf-ebc5a94e1641"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "13018\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 4. Some interesting columns are \"date_\" (the day the speech was given), \"party__prefLabel\" (parliamentary faction),\n",
        "# \"language\" (language the speech is given in; either Finnish, Swedish, or both), and \"content\" which contains\n",
        "# the actual speech. These columns are numbers 4, 6, 7, and 9. Cut these four columns, (Hint: If you don't\n",
        "# know how to cut multiple columns at once, Google it!), then save the truncated dataset as a new file called\n",
        "# \"speeches_2019_4c.csv\". Use this file from now on.\n",
        "\n",
        "# NB! it says 'csv' but the separator is a pipe character (|), define it for cut with -d '|'\n",
        "# (you can Google an exampmle of this as well)\n",
        "! cat finparlspeeches_2019.csv | cut -d '|' -f 4,6,7,9 | head  # check that we got what we wanted\n",
        "! cat finparlspeeches_2019.csv | cut -d '|' -f 4,6,7,9 > speeches_2019_4c.csv\n",
        "! echo '------------------------'\n",
        "! ls  # check that we have it"
      ],
      "metadata": {
        "id": "RFsvuoYpcLc4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3833442a-4f28-4dd8-d9f8-4d28edf99e5a"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "party__prefLabel|language|date_|content\r\n",
            "Kansallinen Kokoomus|fin|2019-01-08|      Lähetekeskustelua varten esitellään päiväjärjestyksen 2. asia. Puhemiesneuvosto ehdottaa, että asia lähetetään talousvaliokuntaan. Lähetekeskusteluun varataan enintään 30 minuuttia. Asian käsittelyssä noudatetaan aikataulutettujen asioiden osalta sovittuja menettelytapoja.\r\n",
            "Perussuomalaiset|fin|2019-01-08|      Arvoisa rouva puhemies! Käsittelyssä on hallituksen esitys laiksi työeläkevakuutusyhtiöistä annetun lain muuttamisesta ja eräiksi siihen liittyviksi laeiksi. [Hälinää — Puhemies koputtaa] Esityksen tavoitteena on selventää, mitä institutionaalisten sijoittajien on julkisesti kerrottava siitä, miten ne käyttävät osakkeisiin perustuvia oikeuksia omistamissaan yrityksissä. Hallitus esittää, että myös lakisääteistä työeläkevakuuttamista harjoittavien eläkelaitosten omistajaohjauksen periaatteita koskevia säännöksiä selkiytetään ja täydennetään nykyisestä. Työeläkelaitosten omistajaohjauksen periaatteista ei ole säädetty kovinkaan kattavasti voimassa olevassa lainsäädännössä, vaan omistajaohjauksen periaatteiden laatiminen on jätetty pitkälti työeläkelaitoksille itselleen. Hallituksen esityksessä ehdotetaan työeläkelaitoksille riittävän kattavia ja yhtenäisiä säännöksiä. Kaikilla työeläkelaitoksilla olisi esityksen mukaan oltava vähintään yleisluontoiset omistajaohjauksen periaatteet. Lisäksi jos työeläkelaitos sijoittaa sellaisten kohdeyhteisöjen osakkeisiin, jotka ovat kaupankäynnin kohteena säännellyllä markkinalla Eta-valtiossa, työeläkelaitoksen olisi omistajaohjauksen periaatteissa kerrottava tarkempia tietoja. [Hälinää — Puhemies koputtaa] Omistajaohjauksen periaatteet olisi julkistettava maksutta työeläkelaitoksen internetsivuilla. Eläkesäätiöille ja -kassoille sallittaisiin lisäksi muu vastaava julkistamistapa. Tällä hallituksen esityksellä on asiallinen yhteys toiseen, samanaikaisesti annettuun lakiehdotukseen HE 305/2018 vp, joka liittyy Euroopan unionissa hyväksytyn osakkeenomistajien oikeuksista annetun muutosdirektiivin täytäntöönpanoon. Siksi työeläkevakuutusyhtiöistä annettuun lakiin ehdotetaan myös eräitä lähipiiritoimia ja johdon palkitsemista sekä niistä päättämistä koskevia lähinnä teknisluontoisia tarkennuksia, jotka ovat tarpeellisia muihin lakeihin, vakuutusyhtiölakiin ja osakeyhtiölakiin, ehdotettujen muutosten johdosta.\r\n",
            "Vihreä liitto|fin|2019-01-08|      Arvoisa puhemies! On hyvä, että työeläkelaitosten pelisääntöjä selkeytetään. Puhumme kuitenkin lähes 200 miljardin euron... [Hälinää — Puhemies koputtaa] Puhumme lähes 200 miljardin euron kansallisomaisuudesta, joka on erinomaisen tärkeä osa meidän eläkejärjestelmämme kestävyyttä ja kantavuutta tulevaisuuteen. Tästä samasta syystä tässä salissa pitäisi käydä keskustelua lähiaikoina myös siitä, miten aiomme turvata etenkin julkisen puolen eläkkeet maailman muuttuessa.  Meillä on julkisella puolella paljon pitkäikäisiä naisia, joista aiheutuvat eläkemenot ovat suuria. Samoin siellä on paljon erityisalojen, kuten sotilaiden ja palomiesten, alemmista eläkei’istä johtuvia kuluja. Meidän on ratkaistava niiden kohtalo jatkossa, etenkin tässä tilanteessa, jossa jo nyt terveyspuolen henkilökuntaa on isossa määrin siirtynyt yksityiselle puolelle, ja riippumatta siitä, saammeko sote-uudistuksen aikaan vai emme, sama suun-taus tulee jatkumaan.  Tätä kiireellisyyttä kuvaa hyvin sekin, että mitä pitemmälle tätä päätöstä rukkaamme ja pitkitämme, sitä kalliimmaksi se yhteiskunnalle tulee. Eli meidän on viimeistään hallitusneuvotteluissa tehtävä esityksiä ja ratkaisuja tulevaisuuden suuntaviivoista, miten saamme turvattua myös julkisen alan eläkkeet tulevaisuudessa, mikä lienee vahva yhteinen tahtotilamme koko tässä salissa.\r\n",
            "Kansallinen Kokoomus|fin|2019-01-08|      Arvoisa rouva puhemies! Tämä lakiesitys on tervetullut ja osoittaa sen, että hallitus tekee loppuun asti työtä, niin kuin eduskuntakin. Työeläkejärjestelmän toimivuuden tärkein turva on se, että Suomeen syntyy uusia työpaikkoja ja jokainen työkykyinen myöskin pääsee työhön.  Tämän hallituksen esityksen tavoitteena on todellakin lisätä lakisääteisen työeläketurvan toimeenpanosta huolehtivien työeläkelaitosten toiminnan läpinäkyvyyttä ja avoimuutta, raportoinnin oikea-aikaisuutta ja julkista luotettavuutta. Ja jotta nämä tavoitteet, todella tärkeät tavoitteet, toteutuvat — kiitos ministeri Mattilalle, että tämä esitys todellakin eduskuntaan tulee — niin meidän on todellakin säädettävä uusia tarkentavia säädöksiä omistajaohjauksen periaatteista ja ulottaa myöskin säätely työeläkevakuutusyhtiöiden ja Kevan lisäksi myös muihin työeläkelaitoksiin sekä valtion eläkerahastoihin.  Arvoisa puhemies! Haluan tässä lähetekeskustelussa vielä todeta, että todellakin nykyistä avoimempi omistajaohjaus vahvistaa osakkeenomistajan ja sijoituskohteena olevan yhtiön suhdetta ja parantaa myös yleistä sijoitussuojaa. Ja pidän erittäin hyvänä sitä, että omistajaohjauksen periaatteet ulotetaan myös niille työeläkelaitoksille, joita velvoitteet eivät tällä hetkellä todellakaan koske, ja tämä tuo todellakin sijoitustoimintaan ja markkinoille tasapuolisemmat toiminta- ja kilpailuedellytykset. Ja lopuksi vielä totean, että tuo tasapuolinen toimintaympäristö edistää rahoitusmarkkinoiden vakautta ja ennakoitavuutta ja lisää myöskin sijoittajan suojaa.  Eli pidän tätä esitystä hyvänä, mutta on tärkeää, että talousvaliokunta tämän tarkasti käy läpi.\r\n",
            "Perussuomalaiset|fin|2019-01-08|      Arvoisa rouva puhemies! Selailin tätä täältä asian valmistelu-osiosta. Se on yleensä hyvä tapa ottaa kantaa myös siihen, millä tavalla asiaan perehtyneet tahot ovat asiaa kommentoineet. Huomasin, että täällä mainitaan, että esitysluonnos on lausuntokierroksen yhteydessä lähetetty valtioneuvoston kanslian yhteydessä toimivalle lainsäädännön arviointineuvostolle, mutta se ei ole kuitenkaan antanut esityksestä lausuntoaan. Mielestäni täällä eduskunnassa on ollut hyvin usein lakiesityksiä, joista on keskusteltu, onko lainvalmistelu ollut asiantuntevaa ja oikeaa ja hyvää.  Kysyisinkin teiltä, ministeri, onko tässä lakiesityksessä mielestänne vielä mitään sellaista korjattavaa ja minkä takia esimerkiksi lainsäädännön arviointineuvosto ei ole nyt tästä antanut lausuntoaan. Vaikutusten arviointihan on hyvin tärkeää ja kuuluu muun muassa arviointoneuvoston toimenkuvaan. Onko tässä jotakin sellaista? Muun muassa täältä Vastaan tai varauksellinen -osiosta löytyy huomioita tähän käsitteillä olevaan lakiin.\r\n",
            "Suomen Kristillisdemokraatit|fin|2019-01-08|      Arvoisa rouva puhemies! Täällä edustaja Järvinen oli huolissaan eläkkeiden kohdentumisesta. Kyllä nostaisin tähän myös tämän tulevien sukupolvien eläkkeiden tason, sen, miten turvataan nuorten eläkkeiden taso, nykyisten nuorten, jotka pohtivat, mahtavatko he saada eläkettä ollenkaan.  Itse tähän asiaan: Tämä säännösten tarkistaminen on mitä tervetullein asia, ja se yhtenäistää. Niin kuin täällä edustaja Sarkomaa totesi, tämä on erittäin hyvä esitys ja tuo sen tarvittavan uudistuksen ja jatkaa tätä hallituksen sujuvoittamis- ja selkeyttämislinjaa ja on siinä mielessä tervetullut.\r\n",
            "Suomen Kristillisdemokraatit|fin|2019-01-08|      Arvoisa rouva puhemies! Niin kuin täällä kollegat edellä ovat todenneet, työeläkevarojen huolellinen ja pitkäjänteinen hoitaminen on koko kansakunnan etu. Ja totta kai on tärkeää, että sijoitusstrategiat ja omistajaohjauksen periaatteet ovat avoimet ja läpinäkyvät.  Tässä samassa yhteydessä keskustelu on paikallaan toki myöskin siitä, millä tavalla tänä päivänä voidaan huolehtia siitä, että tulevaisuuden eläkevarat ovat kestävällä pohjalla. Ja silloin puhutaan siitä, miten me voimme varmistaa, että meillä syntyvyys on kasvussa, ja siitä, millä tavalla me voimme varmistaa sen, että ne nuoret, jotka ovat työelämässä tai työikäisiä, sinne työelämään todellakin sitten pääsevät siirtymään ja ettei tapahdu sitä, mitä tänä päivänä valitettavan usein on, että meillä suuri joukko nuoria syrjäytyy kokonaan, ja meillä näitten nuorien työkyvyttömyyseläkkeiden määrä, joissa usein on taustalla mielenterveysongelmia, on valitettavasti ollut kasvussa. Ja nämä ovat niitä asioita, joihin tuommoisen työeläkejärjestelmän sisällä on syytä kiinnittää huomiota.  On ollut paljon keskustelua myös siitä, että ihan viime aikoina myös työkyvyttömyys-eläkehakemusten määrä on kasvanut, ja on esitetty jopa arvioita, että esimerkiksi aktiivimallilla olisi saattanut olla jotakin taustaa tämän työkyvyttömyyseläkkeiden hakemusten määrän kasvussa. En tiedä, pystyykö ministeri tähän asiaan kommentoimaan tässä yhteydessä, mutta tämä on asia, mikä olisi varmaan syytä selvittää. Työeläkeyhtiöiden johto on esittänyt muun muassa huolestuneisuutta siitä, miten nyt näiden työkyvyttömyyseläkkeiden määrä ja niiden hakemusten määrä on yhtäkkiä pompsahtanut ylöspäin.  Kaiken kaikkiaan tämä lakiesitys on varmasti sellainen, joka on paikallaan ja on entisestään vahvistamassa omistajaohjauksen periaatteitten, voisiko sanoa, johdonmukaisuutta ja pitkäjänteisyyttä.\r\n",
            "Perussuomalaiset|fin|2019-01-08|      Arvoisa rouva puhemies! Työeläkerahastoissa oli viime vuoden kesäkuun lopussa 202 miljardia pääomia, ja se on tietysti erittäin merkittävä summa kansantaloudessa. Eläkkeitä maksetaan noin 15 miljardia euroa ja saman verran kerätään eläkemaksuilla näihin rahastoihin vuosittain tuloja. Tämä tulovirta on siis tavallaan läpijuoksu näillä luvuilla — se, mitä tulee, menee — ja silloin avainkysymys on, miten tämä pääoma, joka on kerätty menneinä aikoina työssä olleilta ja nykyisin työssä olevilta, saadaan tuottamaan edelleen jatkossakin niin hyvin kuin se on nyt tuottanut.  Tarkistin, että vuoden 2010 lopussa eläkerahastoissa oli yhteensä 136 miljardia euroa, ja nyt siis 202 miljardia, kahdeksan vuotta myöhemmin. Se tekee noin 8 miljardia per vuosi kasvua, sijoitustuottoja, rahastoihin — se on valtava summa — ja tämän varassa meidän tuleva eläkepolitiikkamme tietysti pitkälti on. Emme varmaan voi lähteä siitä, että eläkkeitä leikattaisiin — ainakaan perussuomalaiset eivät siihen ole sinänsä valmiita, vaikka eläkekatto sinänsä voisi olla paikallaan meilläkin — mutta ei myöskään voida eläkemaksuja korottaa hallitsemattomasti, vaan täytyy toimia pitkäjänteisesti, jotta myös ne turvaavat osaltaan työllisyyttä ja elinkeinoelämän toimintamahdollisuuksia.  Tämä, että eläkevarat sijoitetaan edelleen tuottavasti ja turvaavasti, on tärkeä tehtävä sekä eläkeyhtiöille että valvovalle viranomaiselle, ja toivon, että tämä lainsäädäntö vahvistaa ja edistää tätä. Toisaalta vielä nostan esille sen, että noin puolet eläkevaroista on sijoitettu ulkomaille ja toinen puoli, suurin piirtein, kotimaahan. Olen itse pohtinut sitä, voitaisiinko päästä siihen, että eläkevaroja hyödynnettäisiin koko kansantalouden kannalta esimerkiksi niin, että sopivalla tavalla niitä voitaisiin käyttää perusinfran parantamiseen eli esimerkiksi suuriin liikennehankkeisiin. Niistä ei saada ihan sellaista suoraa tuottoa kuin saadaan sijoituksista ulkomailta, mutta niistä saadaan kansantaloudelle hyötyä, ja työllisyyden paranemisen kautta voisi tulla jopa parempi tuotto.\r\n",
            "Suomen Kristillisdemokraatit|fin|2019-01-08|      Arvoisa rouva puhemies! Tämä hallituksen esitys on varmasti oikein hyvä, ja on hyvä tarkemmin katsoa näitä työeläkevakuutusyhtiöihin liittyviä asioita niin omistajaohjauksen kuin julkisuuden kuin sijoittamisenkin näkökulmasta.  Meillähän ovat työeläkevakuutusrahastot todella kasvaneet merkittävästi, ja paitsi että meidän täytyy todellakin huolehtia tulevien eläkeläisten eläkkeistä, niin meidän tulisi käsittääkseni huolehtia paljon paremmin myös nykyisten eläkeläisten eläkkeistä, varsinkin siitä suuresta eläkeläisten joukosta, joka tämän päivän hyvinvointivaltiossa elää eläkeläisköyhyydessä. Meillä on todella merkittävä määrä eläkeläisiä, joiden päivittäinen toimeentulo on hyvin veitsenterällä. Ja aivan niin kuin edellinen puhuja nosti esiin, itsekin olisin halunnut nostattaa esiin myös sen, mihinkä näitä valtavia eläkerahastoja sijoitetaan. Meillähän suurin osa eläkevaroista on tällä hetkellä sijoitettu ulkomaille ja hyödyntää siellä, ja olen tässä salissa jo aiemmin lausumissani nostanut esiin sitä, voisiko aloittaa neuvotteluja siitä, että sijoitettaisiin kotimaahan isompi osa näistä rahastoista, jotka hyödyttäisivät kotimaan markkinoita ja kotimaan eläkeläisiä — esimerkiksi joihinkin sellaisiin sijoituskohteisiin, jotka todella helpottaisivat myös tämän päivän eläkeläisten jokapäiväistä toimeentuloa. Meillä on monia eri vaihtoehtoja. Ajattelisin, että kun tätä esitystä tarkastellaan ja sitä, kuinka tulevaisuudessa käytetään työeläkevakuutusyhtiöiden varoja, niin meillä olisi huomattavasti laajempi skaala käytettävissämme sijoitusmahdollisuuksia ja nimenomaan mahdollisuuksia sijoittaa myös kotimaahan sillä tavalla, että se hyödyttäisi meitä kaikkia ja tietysti ennen muuta nykyisiä ja myös tulevia eläkeläisiä.\r\n",
            "------------------------\n",
            "finparlspeeches_2019.csv  sample_data  speeches_2019_4c.csv\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 5. How many unique parties (parliamentary factions) appear in the data?\n",
        "! cat speeches_2019_4c.csv | cut -d '|' -f 1 | sort | uniq\n",
        "\n",
        "# NB! There are speeches with empty party labels. Also, the column name is taken as one party here...\n",
        "! cat speeches_2019_4c.csv | cut -d '|' -f 1 | sort | uniq | wc -l"
      ],
      "metadata": {
        "id": "q9TMqy3gcLak",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "82df4bec-f6d5-4a0b-b7f7-d6f1431c20ab"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Kansallinen Kokoomus\n",
            "Korjausliike\n",
            "Liike Nyt\n",
            "party__prefLabel\n",
            "Perussuomalaiset\n",
            "Seitsemän tähden liike\n",
            "Suomen Keskusta\n",
            "Suomen Kristillisdemokraatit\n",
            "Suomen ruotsalainen kansanpuolue\n",
            "Suomen Sosialidemokraattinen Puolue\n",
            "Vasemmistoliitto\n",
            "Vihreä liitto\n",
            "13\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 6. Which party has given the most speeches?\n",
        "! cat speeches_2019_4c.csv | cut -d '|' -f 1 | sort | uniq -c | sort -rn"
      ],
      "metadata": {
        "id": "JQcpUA6Jca_Q",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0e17508a-0523-4b64-829a-303ddc0d0ef5"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "   2957 Suomen Sosialidemokraattinen Puolue\n",
            "   2572 Perussuomalaiset\n",
            "   2472 Suomen Keskusta\n",
            "   2436 Kansallinen Kokoomus\n",
            "    856 Vihreä liitto\n",
            "    742 Vasemmistoliitto\n",
            "    451 Suomen Kristillisdemokraatit\n",
            "    312 Suomen ruotsalainen kansanpuolue\n",
            "    166 Korjausliike\n",
            "     38 Liike Nyt\n",
            "      9 \n",
            "      6 Seitsemän tähden liike\n",
            "      1 party__prefLabel\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 7. How many of the speeches are given only in Swedish?\n",
        "! cat speeches_2019_4c.csv | cut -d '|' -f 2 | sort | uniq -c | sort -rn # let's see what values we have here\n",
        "# only 99 swe out of 13018\n",
        "\n",
        "# How many percentages is that? (Google if there is a way to make Bash do calculations with decimals ;) )\n",
        "# Bash doesn’t support floating-point arithmetic, so if we want to know how many percentages that is, use a calculator ;)"
      ],
      "metadata": {
        "id": "pZYhkwDEca46",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "040fe092-4d33-4fbd-f014-1b9ab683780a"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  12875 fin\n",
            "     99 swe\n",
            "     25 finswe\n",
            "     18 \n",
            "      1 language\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 8. How many times does the word \"virus\" appear in the speeches? What about \"korona\" or \"covid\"?\n",
        "\n",
        "# NB! cutting the column with the speeches is optional here,\n",
        "# since the other columns would not interfere with the  search in any way\n",
        "# but I've cut it anyway just to practice\n",
        "! cat speeches_2019_4c.csv | cut -d '|' -f 4 | egrep -i 'virus' | wc -l\n",
        "! cat speeches_2019_4c.csv | cut -d '|' -f 4 | egrep -i 'korona' | wc -l\n",
        "! cat speeches_2019_4c.csv | cut -d '|' -f 4 | egrep -i 'covid' | wc -l\n"
      ],
      "metadata": {
        "id": "xmZUmGMTca1R",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "752bb863-d5c6-459d-cb59-eed694b36528"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2\n",
            "0\n",
            "0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 9. Get all the speeches that mention \"ilmastonmuutos\" (climate change). Which party has the most of those?\n",
        "\n",
        "# Here we should NOT cut -f 4 before making the search, so that we still have the party column\n",
        "! cat speeches_2019_4c.csv | egrep -i 'ilmastonmuutos' | wc -l\n",
        "! cat speeches_2019_4c.csv | egrep -i 'ilmastonmuutos' | cut -d '|' -f 1 | sort | uniq -c | sort -rn\n"
      ],
      "metadata": {
        "id": "QBOV5GmMcag9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "86f5ba25-37e1-49f7-89bf-9263b3ec4edf"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "196\n",
            "     70 Vihreä liitto\n",
            "     32 Suomen Sosialidemokraattinen Puolue\n",
            "     30 Suomen Keskusta\n",
            "     22 Vasemmistoliitto\n",
            "     18 Kansallinen Kokoomus\n",
            "     17 Perussuomalaiset\n",
            "      3 Suomen Kristillisdemokraatit\n",
            "      2 Liike Nyt\n",
            "      1 Suomen ruotsalainen kansanpuolue\n",
            "      1 Korjausliike\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 10. On which date were there the most speeches about \"ilmastonmuutos\"?\n",
        "\n",
        "# Again, not cutting -f 4 keeps the date columns for use\n",
        "! cat speeches_2019_4c.csv | egrep -i 'ilmastonmuutos' | cut -d '|' -f 3 | sort | uniq -c | sort -rn"
      ],
      "metadata": {
        "id": "boPhp0xMcaeU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b9c91d39-9422-455f-d4dd-4c8464bc8dcc"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "     20 2019-01-16\n",
            "     18 2019-06-11\n",
            "     14 2019-09-12\n",
            "     12 2019-12-17\n",
            "      8 2019-12-16\n",
            "      8 2019-06-26\n",
            "      7 2019-11-26\n",
            "      7 2019-10-09\n",
            "      7 2019-10-08\n",
            "      6 2019-12-18\n",
            "      6 2019-11-07\n",
            "      6 2019-10-10\n",
            "      6 2019-10-01\n",
            "      6 2019-06-12\n",
            "      6 2019-02-20\n",
            "      5 2019-11-14\n",
            "      5 2019-03-04\n",
            "      5 2019-01-15\n",
            "      4 2019-02-12\n",
            "      4 2019-01-31\n",
            "      3 2019-11-20\n",
            "      3 2019-10-16\n",
            "      3 2019-02-07\n",
            "      2 2019-12-19\n",
            "      2 2019-12-04\n",
            "      2 2019-10-24\n",
            "      2 2019-06-25\n",
            "      2 2019-01-30\n",
            "      2 2019-01-29\n",
            "      1 2019-12-10\n",
            "      1 2019-11-05\n",
            "      1 2019-10-23\n",
            "      1 2019-10-03\n",
            "      1 2019-09-26\n",
            "      1 2019-09-19\n",
            "      1 2019-09-04\n",
            "      1 2019-06-27\n",
            "      1 2019-06-19\n",
            "      1 2019-02-26\n",
            "      1 2019-02-21\n",
            "      1 2019-02-14\n",
            "      1 2019-01-22\n",
            "      1 2019-01-10\n",
            "      1 2019-01-08\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# (EXTRA) Formulate a question of your own! What would you like to find out?\n",
        "\n",
        "# What about how many speeches were given on January?\n",
        "\n",
        "# egrep -o\n",
        "# -o, --only-matching\n",
        "#     Print  only  the matched (non-empty) parts of a matching line,\n",
        "#     with each such part on a separate output line.\n",
        "\n",
        "! cat speeches_2019_4c.csv | cut -d '|' -f 3 | sort | egrep -o '2019-01' | uniq -c"
      ],
      "metadata": {
        "id": "9TH6dv70cp8r",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5a3ac96c-0153-4908-fad0-6a08585ff3ac"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "   1409 2019-01\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "### Exercise 2\n",
        "\n",
        "11. Let's use the CORE-corpus availablet at https://github.com/TurkuNLP/CORE-corpus. Clone it to your notebook. Focus on the development set of the dataset. How many documents does it include?\n",
        "\n",
        "12. Column 1 indicates the register abbreviations of the documents. They are explained in the txt-file found in the repo. What are the 10 most frequent register combinations found in the dataset? What actual classes do these abbreviations correspond to?\n",
        "\n",
        "13. Select two registers you want to focus on. How to make a regular expression that matches the register label in the first column but not the actual text?  Make files that consist of frequency lists of the words of these two registers. Normalize the words first to lower case and delete numbers and punctuation. Can you think of how to combine the delete commands to one using regexes?\n",
        "\n",
        "14. How many unique words do the two registers have?\n",
        "\n",
        "15. How would you search for words that occur at least 100 times using regex? Note that the frequency list has sometimes whitespaces in the beginning before the numbers. See how fast the frequency of the words drops!"
      ],
      "metadata": {
        "id": "_hV8Po1VAmCx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 11. Clone the repo. (Hint: Go to the website and get the repo address from the green button)\n",
        "! git clone https://github.com/TurkuNLP/CORE-corpus.git"
      ],
      "metadata": {
        "id": "-k-zafibBng3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "53a24e5c-8751-40d2-e97b-cbf2cd267071"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'CORE-corpus'...\n",
            "remote: Enumerating objects: 31, done.\u001b[K\n",
            "remote: Counting objects: 100% (23/23), done.\u001b[K\n",
            "remote: Compressing objects: 100% (21/21), done.\u001b[K\n",
            "remote: Total 31 (delta 12), reused 5 (delta 2), pack-reused 8 (from 1)\u001b[K\n",
            "Receiving objects: 100% (31/31), 124.72 MiB | 25.16 MiB/s, done.\n",
            "Resolving deltas: 100% (13/13), done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# (11. continued) Change to the right directory\n",
        "%cd CORE-corpus"
      ],
      "metadata": {
        "id": "8bYvrLFEBu5i",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c9cb1d55-42f9-4fb3-fecd-a868e20c1e42"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/CORE-corpus\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Focus on the development set of the dataset (dev.tsv.gz).\n",
        "! zcat dev.tsv.gz | head  # how does it look like?"
      ],
      "metadata": {
        "id": "iLgBD2GOBvrB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5e8dd657-22ca-40cb-8ce8-d354452847ff"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "NA SR\t0601771\tAlready an accomplished adventure and 'ultramarathon' cyclist, the 31-year-old Sudbury native will push himself to new heights -- or lows, depending on geography -- in the Race Across America, a 4,800-kilometre bicycle race from Oceanside, Calif., to Annapolis, Md., across 16 states, through mountains, deserts, hills and plains. But less than a month before hitting the highway on June 13, Lane said he wasn't nervous about facing one of the greatest challenges of his athletic career. \"It's more respect -- a lot of respect for what it is,\" Lane said during a press conference at Laurentian University on Thursday. \"I went to California and did the first 500 miles and I did the last 500 through West Virginia, to get more of an idea what it is, and I have more respect for what other people have done in the past and just how difficult it is.\" 'Difficult' may seem an understatement to some. Not only does the race require Lane to be in top physical condition, but also extremely mentally tough. Race Across America competitors spend 22 or 23 hours each day on the bike, sleeping for as little as an hour at a time. \"I have been adventure racing for 10 years and those races last about five to 10 days and you only sleep about one or two hours a day,\" said Lane, a graduate of Laurentian's adventure leadership program who works as a paramedic. \"And while you never really get used to it, I have kind of gotten used to how it feels. \"But yeah, that is one of the biggest challenges, the sleep deprivation, not just for myself, but for but support group, too. They'll sleep much more, of course, about six hours a night, but they'll be pretty stressed, I'm sure.\" That support group will include a few people who constantly follow Lane in a minivan, ready to provide food and water and meet his medical, mechanical or moral needs. \"They'll have this amazing stereo system and a big PA speaker on top of the vehicle, so they'll have a mic and they can play music, shout encouragement or obscenities, whatever,\" Lane said with a chuckle. \"And I'll have a two-way radio on me at all times.\" The rest will reside in an RV that will leapfrog along the course, stopping for Lane's sleep sessions. But he'll eat, drink and even answer nature's call from atop his bike. If he finishes the race, he'll be only the sixth Canadian to do so. \"What I'm going for is rookie of the year, which is about the best thing you can get next to winning,\" Lane said. \"And to be top three. I'm aiming for the podium.\" Training for Race Across America is difficult enough -- \"like another full-time job,\" Lane says -- but he has done it with a pair of surgically-altered knees. He went under the knife twice in 2010, once in March and again in October. \"It wasn't an injury as much as a genetic predisposition,\" Lane said. \"My kneecaps were too high. So I found a surgeon at Sunnybrook and he basically broke my legs apart, lowered my kneecaps and put them back together. \"I have recovered well and I'm better than ever.\" The doctor told him to avoid running or lifting weights for a couple of years, hence the shift to a straight cycling event. A pair of independent filmmakers plans to follow Lane as well, making a documentary about his trip. The trek will benefit science, as well, as members of Laurentian's school of human kinetics will test the effects of the race on the cyclist's body. \"We will be collecting some data,\" said Sebastien Nault, a sessional lecturer at the university. \"We'll be doing some physical research to see what are the effects on his condition throughout the race.\" He's also using the race as a platform to raise money for the SickKids Foundation, the not-for- profit charity which raises funds on behalf of the Hospital for Sick Children in Toronto. \"We had hoped to raise $50,000,\" Lane said. \"I don't know if we'll quite get there, but SickKids is fantastic to work with and they're happy with whatever we can raise, not just money, but awareness, too. \"They're a fantastic charity. I work as a paramedic, with the City of Greater Sudbury and with Ornge, so I'm in direct contact with SickKids and they do incredible work.\" The Laurentian Alumni Association presented Lane with a cheque for $10,000 at Thursday's presser. More information on Lane's upcoming race can be found at www.teamhammerfest.comor www.raceacrossamerica.org. \n",
            "OP OB\t3160437\tBeing a Successful Band Has Nothing to Do With Talent [Editor's Note : Are you a musician? Is your group having issues? Ask Fan Landers ! Critic Jessica Hopper has played in and managed bands, toured internationally, booked shows, produced records, worked as a publicist, and is the author of The Girls' Guide to Rocking , a how-to for teen ladies. She is here to help you stop doing it wrong. Send your problems to her -- confidentiality is assured, unless you want to use your drama as a ticket to Internet microfame.] Dear Fan, We're having a really hard time finding an audience. We have a solid album, and no one has heard it. We play an absolutely killer live show and we can't get enough people to come and see us play. The people that do make it to our shows always have a great time and we get a lot of good feedback. We've started to solve the problem by doing showcases, where eight bands play, and this tends to draw a larger crowd, but we are READY to move on and do great things. I admit I am clueless at marketing, at generating interest, even the basic aspects of talking to people about my music. I know there's no magic formula, but I'd really appreciate any tips or advice you could give about what I could work on. We are genuinely talented and we keep making progress, I just feel we are capable of so much more. Sincerely, Us Dear Us, Your letter gets at the crux of why it sucks to be in a band; it's a little heartbreaking. Reading over your letter and listening to the link to your band's album on Bandcamp, I see that your problem is actually something other than what you think it is. Getting people to your shows, having a record people like doesn't really have anything to do with talent. This is the great myth of the music industry. Sure, beautiful songs and punchy hooks help, but you need to purge your mind of the idea that if you just make great songs and play them well that everything else will fall into place, because you deserve it. Mourn the death of that dream of music as a meritocracy and that the world is going to discover your genius and come scampering to your doorstep to pad your path with dollars. Thinking that way will make the reality of being in a band really discouraging and painful. You live in a city that is a great place to be a musician... if you are a country session player. You are a macho bar rock kind of band and admittedly horrible at networking or self-promoting. Does another member of your band have any palpable social skill or gladhanding ability that you can call upon? Could you recruit a second guitar player from another band who has connections to bookers and other bands? You guys need to work on ingratiating yourself into your city's rock scene and getting on the radar of the people that make things happen within it -- bookers, club staff, other bands. Think about the marginally talented local band that gets all the plum opening slots on the big national-band shows and get annoyed. Let that competitive fire fuel your pro-active networking. Covet thy neighbor's opening slot. You know what I mean. Be a super nice dude, accommodating, show gratitude to the people that help you, don't act like you deserve things and people will want to work with you again just on the basis of that. Most dudes in bands act like the world owes them and they never say thanks and they want everyone else to make it happen for them. No one wants to work with a big baby. Nepotism is a real thing. Anyone who tells you otherwise is not well-connected. \n",
            "NA IN NE DP\t1445020\t'I always like to think of myself as a cartoon, and now I'm a Sim!' Katy Perry is immortalised as a computer character Her real life might be in turmoil, but Katy Perry couldn't be happier in the virtual world of The Sims. The singer is to join the popular video game that simulates real life and families. Just weeks after the end of her marriage to Russell Brand the 27 year old star has been named as the star of a new game. SCROLL DOWN FOR VIDEO... Game on!: Katy Perry has been immortalised as a computer game character in The Sims 3 Showtime Her tie in with the makers of The Sims will feature Perry-themed content and virtual goods, including clothing, hairstyles, furniture and props. Perry, a fan of the game, said: 'I love how you're able to play out different stories through your Sims characters - giving them different careers and watching them succeed. It's cool to see the Sims' stage performances in The Sims 3 Showtime decked out just like my California Dreams Tour - even my cotton candy video screens are in there! 'I always like to think of myself as a cartoon, and now I'm a Sim!' Sim-ple: Katy is thrilled to become a part of the virtual reality game Electronic Arts, makers of The Sims, said Perry will play a key role in developing the games featuring her likeness. The deal includes creating Perry-themed in-game items and virtual goods and begins with a Perry-themed collector's edition of The Sims 3 Showtime, which will go on sale in March. Steve Schnur, executive of music for EA, said that a Sims-powered music video of Perry's Hot and Cold made in The Sims 2 became one of the most popular \"Simlish\" music videos to date. Sweet touch: Sets from Katy Perry's tour feature in the game Dance break: The computer game pulls out all the stops to recreate Katy's world Lifelike: Katy Perry made sure she had her say when it came to creating her character 'Today marks the start of the most integrated and exciting collaboration with any artist in the history of The Sims,' said Schnur. 'Katy Perry has been a fan of The Sims for several years, and her Simlish recording of her hit Hot and Cold featured in The Sims 2 series quickly became the most popular Simlish music video to date...No performer could better represent the humor and fun of The Sims brand.' The partnership will kick off with a special Collector's Edition for The Sims 3 Showtime, launching in March. Perry will also be featured in advertising and marketing campaigns for the brand. Key role: Katy will have lots of involvement with the brand in her three in-game deal Feeling fruity: Katy is known for her elaborate concerts, recreated in the game Limited edition: A special collector's version of the game has been released in Katy's honour Will she have a tall lanky ex drug addict ex husband to play with? And a virtual whirlwind romance and 5 minute marriage? Let's hope The Sims have more suitable virtual boys for her to go out with - maybe not tick the 'ex sex addict' box again Katy - never a good idea..... \n",
            "NA SR\t3060117\tQPR vs Southampton - Match preview and team news Friday 16 November 2012 The bottom two of the Premier League meet on Saturday, with Queens Park Rangers looking for their first win of the season at the 12th time of asking. Mark Hughes' side have picked up four draws so far, but have to go back 6 months for their last league win, at home to Stoke at the beginning of May. All is not lost just yet for R's fans - one win would put them two points behind 17th place Aston Villa. Southampton are sticking by the manager that led them to successive promotions, regardless of Premier League performances, yet Nigel Adkins will be desperately disappointed to have picked up just one solitary victory - now seven games back. Defeats to West Brom, Spurs and West Ham in recent weeks have kept the Saints at the foot of the table. This weekend QPR will have their squad depth tested - Stephane M'Bia is out suspended, while Fabio (hamstring), will miss the tie. The club await fitness news on Jamie Mackie, Kieron Dyer, Ryan Nelson and Park Ji-Sung. Djibril Cisse will start up-front, preferred to Bobby Zamora. Southampton have most of their first team members available, with just Frazer Richardson out with a thigh injury. \n",
            "NA NE\t0119728\tSearch age: Search in: US says it's ready to talk to Taliban chief Dean Nelson Washington US Secretary of State Hillary Clinton at the House foreign affairs committee. She told the committee the US would continue to 'fight, talk and build' in Afghanistan and Pakistan. Photo: Alex Wong WASHINGTON is ready to negotiate with Mullah Omar, the Taliban leader who sheltered Osama bin Laden as he plotted the September 11 attacks. Secretary of State Hillary Clinton has told a Congress committee that the US regards his involvement as crucial to the prospects for peace in Afghanistan. Her comments have been taken as a significant shift in US policy, from moves to divide the Taliban-led insurgency and isolate Mullah Omar to an acknowledgement of his leadership. Earlier this month it emerged that US officials had met leaders of the Haqqani network, the Taliban faction blamed for some of the most devastating attacks on US and NATO forces in Afghanistan. Advertisement Last week, the faction's commander, Sirajuddin Haqqani, warned Washington that only the Quetta Shura, the Taliban-led militant organisation, could negotiate a peace deal and that his fighters would not be divided. In an appearance before the House foreign affairs committee, Mrs Clinton said the US would continue to ''fight, talk and build'' in Afghanistan and Pakistan to test any willingness to negotiate. She said there was ''evidence going both ways'' on its intentions. Doubts were raised over the Taliban leadership's intentions after the assassination last month by a suicide bomber of Burhanuddin Rabbani, the former Afghan president and the head of the High Peace Council. Mrs Clinton said last month's Haqqani meeting was not a ''negotiation'' and that no meetings had followed it, but stressed that any future negotiations with the Taliban would need the Quetta Shura's blessing. A US Defence Department report to Congress released on Friday said violence in Afghanistan went down in the past year, reversing five straight years of sharp increases. Still, the insurgency remained ''capable'' and ''resilient'', and its havens across the border in Pakistan remained the biggest risks ''to the process of turning security gains into a durable, stable Afghanistan'', according to the report. The violence dropped particularly in the country's south-west, west and north, the Pentagon reported. Security in the country's eastern region remains tenuous because of the sanctuary and support from Pakistan. The report sets the stage for President Barack Obama to proceed with his plan to give the Afghan government full command of its security by the end of 2014. The US has begun to pull out troops, with the first 10,000 out of almost 100,000 expected to leave by the end of this year. ''The most significant development during this reporting period is the reduction in year-over-year violence,'' the Pentagon said in the congressionally mandated report. \n",
            "IN OP DT\t3119905\tWelcome to India Here at The Great Recovery HQ we have been glued to our TV screens for the last few weeks watching BBC 2's three part documentary series: Welcome to India . Over 1 in 6 of the world's population live in India, and during the series we see the film crew follow a handful of 'backstreet entrepreneurs' who are all \"learning to survive in a crowded world\". In episode 1 we meet 23 year old Kaale who has moved from the countryside to the city of Kolkata in search of gold. Gold is a precious metal with huge cultural significance in India -- it acts as insurance, and Indian housewives own an incredible 11% of the world's gold stocks. That's more than the USA, Germany and Switzerland put together. Kaale lives with 20 other men in a small room in Kolkata's Jewellery district, and wakes at 3am every morning to begin work. The huge mass of goldsmiths who live and work in the area leave particles of gold on the street which falls off their clothes, hands and hair when they wash, and gathers with the dirt and dust. Kaale and his friends spend hours sweeping the streets and using traditional American gold panning techniques to extract the metal. \"When people look at the street, all they see is garbage... But we know what's hiding here.\" During the hour long programme, we see Kaale taking his gold panning venture even further by lowering himself down the tiny drains of Kolkata, extracting sack loads of 'sludge' and selling it to Javed, whose family own a backstreet gold panning workshop on a disused bank of the River Ganges. Here, Javed's 20 employees work tirelessly to first dry the mud, then sieve and crush it into powder (the only mechanised part of the process) before using 19 th century Californian techniques with a wooden board and water to massage the mud to trap gold particles in tiny grooves on the board. Mercury is added to the remaining mud to 'stick' to the gold, and this is moulded into 'ranga' balls which are heated to first drive off any impurities, then to melt off the mercury. All that remains is one final smelting with nitric acid and they are left with 24 carat gold! These tiny gold particles that could so easily be dismissed as lost are being meticulously sourced and implemented back into circulation. In episode 2 we meet Kanye, who also works recycling metals, but on a much bigger scale. Mumbai is home to a cargo ship graveyard, and it is down to 16,000 workers to 'butcher' these 12,000 tonne ships from across the world. It is an extremely dangerous job -- every part of the ginormous vessels is taken apart by hand. Not a bit of the boats go to waste. The reusable parts go to the market and supply the Indian construction industry with cheap recycled steel, and the bits that cannot be reused are turned into something else and sold on. Even the tiny specs of iron that gather on the floor are collected and sold on. Moving away from metals, to plastic recycling: Johora lives with her 12 children and husband alongside the railway tracks in Kolkata. She has worked her way up from a rag-picker to business-woman, with 3 warehouses collecting, sorting and selling on empty plastic bottles. Her young son, who has a network of 'suppliers' (who scour rubbish tips, restaurant and shop refuse) and a rickshaw collects the bottles, and brings them back to the warehouses where 4 employees remove the labels and sort the bottles by colour. They are then bundled into huge balls weighing up to 100 kilos and sold by weight for around 50 each to the next link in the recycling chain. (Johora cheats a little here by filling the odd bottle with water to increase the weight!). At the next stage of the recycling process, some of the bottles are shredded, and the 'chips' are sold to manufacturers of poly-synthetic fibres who can use them to make everything from polar fleece to the stuffing for your sofa cushions. All of these examples encapsulate so well the ethos behind a circular economy, however the tragic part of this story is the fact that these people aren't dredging mud out of drains or collecting plastic bottles because they care about the environment, for them it a necessity for survival: \"Where you see trash, we see a livelihood\". It's time that here in the west we began to realise that 'trash' shouldn't really exist as a concept any more. Money can be made from the recycling and recovery process. What if we started collecting and selling all of our plastic bottles? What if recycling became more of a business than a chore? This illegal 'backstreet entrepreneurship' can also be seen in San Francisco, where due to the California Bottle Bill of 1987 (which means a deposit is paid on all plastic and glass bottles sold in the state) recycling has become an extremely profitable business for gangs who raid recycling bins. In February's edition of BBC 4's 'Costing the World' , Tom Heap meets a 78-year-old Vietnamese woman who has spent all night rummaging through bins to collect just 17 worth of recycling. While this illegal activity means that 78% of all waste in the city is diverted from landfill, imagine if we could make this a legitimate business model, instead of relying on those in desperate need to do the work for us, illegally. One of the main drivers for the Technology Strategy board putting up 1.25million for designers to come up with new approaches to the closed loop system are the rising cost of metals and other resources. China now own 97% of the Rare Earth market. We don't use the words 'sustainable' or 'eco' as the drivers are now financial as well as moral. One of the most interesting things to note about all of these stories is the networks that these people have. When Kaale decided to take extend his business to selling drain sludge, he knew exactly who to call who could process mud into gold. When Johora brought her own van so that she could sell her bottles on directly, she knew exactly where to take them to get the best price. It's developing these networks that we need to build on. We need to make sure that we know where to take our goods to recycle them; we need to make sure we know a materials expert to consult when designing new products and we need to make sure we are building and using these networks at every stage of a products life span. We need to become a connected society . Join the Great Recovery community by signing up online, and find out more about material flow and closed loop behaviour by coming to one of our workshops , taking place throughout November. RSA Links RSA Mobile Connect to the RSA The RSA The RSA: an enlightenment organisation committed to finding innovative practical solutions to today's social challenges. Through its ideas, research and 27,000-strong Fellowship it seeks to understand and enhance human capability so we can close the gap between today's reality and people's hopes for a better world. \n",
            "NA NE\t0002697\tToronto cycling activists were gnashing their spokes and rending their spandex after city council voted to kill the Jarvis Street bike lanes on Wednesday. Put in only last summer, the curbside lanes are to be pulled right back out again by the end of next year. It looks like a huge step backward. In fact, it could turn into a big win for cycling in the city. Almost lost in the hubbub over Jarvis was the fact that council also voted to push ahead with Toronto's first network of separated bike lanes. That means cyclists will be able to travel on lanes that are not just painted lines on the asphalt, like those on Jarvis, but fully separated from car traffic. Council voted to build separated lanes across the Bloor Viaduct, to start design work on separated lanes for Sherbourne, Wellesley, Harbord and Beverley and to look into separated lanes on Richmond and Peter or Simcoe. The result would be a system that would take cyclists smoothly and safely from, say, the Danforth to the financial district or from the University of Toronto to the waterfront without having to fight their way through mixed traffic. It is the biggest step yet toward creating a true bike network for Toronto -- not just the dog's breakfast of disconnected, unprotected lanes we have now, but a real network like they have in Montreal. Thousands of commuters who now fear to ride in the turmoil of the open road would start cycling instead of driving to work, with all the benefits that brings for health, air quality and traffic congestion Denzil Minnan-Wong (Ward 34-Don Valley East), the moderate conservative councillor behind the project, says the city is planning to spend twice as much on its bike plans over four years as former mayor David Miller did in his final four. On top of 14 kilometres of separated lanes, the city would create 100 kilometres of off-street bike trails in ravines and rail or hydro corridors. After years of slow, stuttering progress on cycling infrastructure, that would be real progress. Great, say cycling advocates -- if it ever happens (they have understandable doubts given all the talk about the city's money troubles). But if you're going to build a bike network, they say, why start by killing bike lanes on Jarvis? The answer is simple: politics. The Jarvis lanes were a red flag to motorists from the start. Jarvis is one of the few broad streets taking car commuters in and out of downtown. Removing the roadway's reversing fifth lane to make room for bikes added minutes to that painful commute. Suburban councillors with car-commuting residents denounced the bike lanes. They were doomed from the moment Mayor Rob Ford took office on a pledge to end the \"war on the car.\" Jarvis had to be sacrificed if the mayor and hostile councillors were ever going to back bike lanes elsewhere. It was an unspoken tradeoff: You can have your lost traffic lane on Jarvis back if we can take away space on other, less vital roads for bike lanes. That will strike cycling zealots as the worst kind of appeasement. In their world, cycling is so virtuous and car commuting so ruinous that making any kind of concession amounts to surrender. They are vowing to fight on to save the Jarvis lanes during the 18-month reprieve they won for the lanes at Wednesday's council meeting. But in the real world, most people still get to work by car. Telling them they are bad people won't get us anywhere. Killing Jarvis should soothe their war-on-the-car fears and ease the way for bike lanes on other roads. Jarvis had to die so bigger bike plans could be born. Now let's make sure they come to something. \n",
            "IN IB\t0427612\tIn a series of terrifying events, otherworldly beings intervene to save innocent lives. The world community reacts with relief as they realize that guardian angels may in fact exist, and they are diligently protecting us. But there are those who would seek to stop what they feel is a threat against their livelihoods and aspirations. How far will some go to battle the Guardians? Is the fairy tale over before it even begins? Guardian Angels is a powerful and compelling story about the catalyst that has the power to unite society in the hope for a better future. The spark of hope is fragile -- can it last? NeoCell is the number one collagen brand in the world and a leader in the fields of nutritional science and methods development. And while they have grown considerably over the years, one thing has never changed -- their commitment to finding the most innovative and effective nutraceutical supplements. The Don Sabrosa Company makes incredible salsas in a variety of flavors. It is made with all natural healthy ingredients, such as tomatoes, peaches, honey, and blueberries. They have a heat level to suit every taste. \n",
            "IN IB\t1480827\tLooking at the Islamic Bond (Sukuk): A Case of Legal Uncertainty? The financial markets of the Middle East (in line with much of the Western world) are currently in an economic turmoil and this has caused a number of investors in the financial sector to revisit some of the concepts of Islamic (Sharia) Law. The most important principles of Sharia Law in relation to the financial sector include- ? the prohibition of interest; ? profit and loss sharing (ie, parties must share risks and profits); ? asset-backing principle -- in each financial transaction, there must be an underlying asset. Following the recent press relating to Dubai and the Nakheel Sukuk, it is the 'legal status' of an Islamic Bonds ('Sukuk') that are under intense scrutiny. The questions that remain open are: From the jurisdictional point of view, is Sukuk to be governed by Sharia Law? If Sukuk is governed by Sharia Law, are the current Islamic Bonds that have been issued Sharia compliant? Do the bond holders have any rights to the assets? Sukuk is, in principle, a Sharia compliant bond. Conventional bonds are not Sharia compliant because they are centered around the provision of interest and do not necessarily uphold the asset-backing principle. In contrast, Sukuk upholds the prohibition of interest and asset-backing principles of Sharia Law. The transaction forming Sukuk does not involve interest and the claim embodied in Sukuk is an ownership claim, and as such, it is similar to the conventional concept of securitization. It is important to note that there is not a standard interpretation of Sharia Law with regards to Sukuk and some Islamic scholars have argued that some of the Islamic Bonds that are in circulation in the financial markets are not Sharia compliant. Therefore, this could potentially result in the bonds being unlawful and unenforceable. The issue is that some of the Islamic Bonds issued may not actually involve the transfer of assets to investors, therefore, the Bonds may well be 'asset-based', but not 'asset-backed.' Hence, such bonds will not uphold the asset-backing principle. This is a crucial issue for investors, for if the legal title of the asset has not been transferred to the investor, in the case of bankruptcy, the assets will be 'swallowed up' by 'returning' to the originators. The jurisdictional question and consequent issue of who has rights to the assets can be further complicated if the financial agreement contains a clause along the lines of 'subject to the principles of the glorious Sharia the agreement would be governed by and constructed in accordance with the laws of England.' This is the clause that was contained in agreements which were subject to court proceedings in the case of Shamil Bank of Bahrain EC v Beximco Pharmaceuticals Ltd (2004) EWCA Civ 19. The Court of Appeal held that the general reference in the agreements did not identify any specific aspects of Sharia Law intended to be incorporated into the contracts and so therefore, Sharia Law did not govern the agreement. In such cases, it is likely that, after looking at the intention of parties and identifying the 'commercial purpose' of the agreement, the court will deem Sharia Law inapplicable. Significant transactions involving Islamic Bonds are now in need of legal interpretation. Presently, a court case is still proceeding over the issue of whether investors in Sukuk issued by East Cameron (a US energy firm) own the assets. Furthermore, Nakheel (the property unit of Dubai World) was supposed to pay the last rental coupon and buy back its assets by 14 December 2009, but has declared that it cannot do so. We await to see to any further legal proceedings. \n",
            "NA MA\t0028689\tChelsea Flower Show through the years The Chelsea Flower Show is the jewel in the Royal Horticultural Society's crown. Held each year in May, the event attracts hundreds of thousands of visitors over its five days. A staggering 800 people spend just over three weeks building the show putting together the show gardens, nursery exhibits, floristry displays, garden product stands and educational displays. The Great Pavilion is 12000m 2 -- the same size as two football pitches with enough room to park 500 London buses. The magic began in 1804 when Sir Joseph Banks and John Wedgewood founded the Horticultural Society of London. Prince Albert, its then President, renamed the charity in 1861 as the Royal Horticultural Society and the following year the Great Spring Show was born. In 1912 the show was forced to relocate from Temple Garden to the Royal Hospital at Chelsea. The venue proved such a success that in 1913, the Chelsea Flower Show was premiered. The picture shown left shows RHS staff from the show that year. In the early years of World War I the show was still held, but was cancelled in 1917 and 1918. By the 1920s the show was back in full force with the Chelsea Flower Show as THE social setting for tea parties and royal visits. Photographs from 1930-1939 (see above) reveal that the show was a roaring success and in 1937, when King George VI and Queen Elizabeth celebrated their Coronation Year, a magnificent Empire Exhibition was staged. Unfortunately the show had to be cancelled once again during World War II, as the War Office required the land for an anti-aircraft site. When the show resumed in 1947 there was still some doubt as to whether it should take place at all. Exhibitors wanted a postponement as stocks of plants were low, staff much depleted and fuel for greenhouses was obtainable only with special permits. However, Lord Aberconway (then RHS President) and the RHS Council felt strongly that the show should resume as soon as possible. The show was a great success. In 1953 the coronation of Queen Elizabeth II caused for a flower show that reflected the celebratory mood of the country. While the majority of the royal family were able to attend, other engagements meant that unfortunately the Queen herself wasn't. The show continues to grow in strength and popularity with the 2010 Chelsea Flower Show the 88th show to be held at the Royal Hospital grounds having seen numerous changes in fashion and design over the years, and the launch of some 600 new plant species. Here's hoping that 2010 will be another winning year. Photographs copyright RHS wih thanks to RHS, Lindley Library Key dates from the RHS Chelsea Flower Show's blooming history 1862 First show held at the RHS garden in Kensington called the Royal Horticultural Society's Great Spring Show. 1888 The RHS Garden was closed and the show was moved to Temple Gardens near Embankment. 1911 The final show at Temple Gardens. 1913 The Great Spring Show is moved to the Royal Hospital grounds permanently, there after known as the Chelsea Flower Show. Japanese dwarf trees, now known as bonsai, are first exhibited. 1914-1916 Despite World War I, the show is still held. Rock garden displays prove to be particularly popular. 1917-1918 The show is cancelled. 1920s The show is back in full swing. 1926 The General Strike causes a delay of one week. 1928 Torrential flooding of the marquees due to heavy rain and hailstones the night before, but the show opens the next morning without any signs of damage. 1937 King George VI and Queen Elizabeth celebrate their Coronation Year and the Empire Exhibition is staged in their honour. 1940s The War Office needs the grounds during World War II and subsequently the show is cancelled. 1947 The show resumes and is a success despite concerns from exhibitors. 1953 The coronation of Queen Elizabeth II causes a celebratory show. Unfortunately, the Queen herself wasn't able to attend. 1959 The Times 'Garden of To-morrow', complete with radio-controlled lawn mower is debuted. 1968 Wisley exhibit hostas, which gives a great boost to their popularity. 1980s Paved back gardens and cottage gardens are in high fashion. 1988 Visitor numbers are capped at 157,000 due to huge popular demand at the show in 1987. 1997 Christopher Bradley-Hole's Latin Garden is the first garden at Chelsea to exhibit the new fashion for sparse planting. 2000 Gardens Illustrated present their own garden and win Best In Show. 2009 Top Gear presenter James May breaks the mould as he constructs a garden entirely made out of plasticine. \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# (11. continued) How many documents does it include?\n",
        "! zcat dev.tsv.gz | wc -l # one document per line, so by counting lines we get the number of documents"
      ],
      "metadata": {
        "id": "NtXU1EePB95g",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0c2a2862-f335-4449-ea2b-1008727f6229"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "4845\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 12. Column 1 indicates the register abbreviations of the documents.\n",
        "# They are explained in the txt-file found in the repo (register_label_abbreviations.txt).\n",
        "# What are the 10 most frequent register combinations found in the dataset?\n",
        "! zcat dev.tsv.gz | cut -f 1 | sort | uniq -ci | sort -rn | head -10 # focus on the column 1, make a frequency list"
      ],
      "metadata": {
        "id": "QBjSYgd3CF8V",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ca303736-5951-45f8-ae62-e4a3a289c5ae"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "    987 NA NE\n",
            "    373 OP OB\n",
            "    329 IN DT\n",
            "    279 NA SR\n",
            "    255 NA PB\n",
            "    194 ID DF\n",
            "    185 IN\n",
            "    169 OP RV\n",
            "    143 IN IB\n",
            "    117 HI HT\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# What actual classes do these abbreviations correspond to?\n",
        "! cat register_label_abbreviations.txt | egrep \"NA|NE|OP|OB|IN|DT|SR|ID|DF|IN|RV|IB|HI|HT\" # this will give us explanations for the labels"
      ],
      "metadata": {
        "id": "HcgLKeEvDU3R",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b168d608-6430-4279-cba9-ecc4a9736e59"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "DF\tdiscussion forum\n",
            "DT\tdescription of a thing\n",
            "HI\thow-to instructional\n",
            "HT\thow-to\n",
            "IB\tinformation blog\n",
            "ID\tinteractive discussion\n",
            "IN\tinformational description / explanation\n",
            "NA\tnarrative\n",
            "NE\tnews report / blog\n",
            "OB\topinion blog \n",
            "OP\topinion\n",
            "RV\treview\n",
            "SR\tsports report\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "--> So it would seem that Narrative + news article texts are the most frequent in the CORE corpus, followed by Opinion + opinion blogs\n",
        "\n",
        "I will compare personal blogs (pb) and travel blogs (tb), I want to understand what's the difference between these two!"
      ],
      "metadata": {
        "id": "zTr-If1GDnnL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 13. Select two registers you want to focus on.\n",
        "# How to make a regular expression that matches the register label in the first column but not the actual text?\n",
        "# Make new files that consist of frequency lists of the words of these two registers.\n",
        "# Normalize the words first to lower case and delete numbers and punctuation.\n",
        "! zcat dev.tsv.gz | egrep \"^[A-Z]+ PB\" | head  # beginning of line (^), any uppercase letter [A-Z] one or more times (+), whitespace,  PB\n"
      ],
      "metadata": {
        "id": "2Wyj8VczDkuD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "52d03641-5e94-48ee-94fb-7ea1cf8a3a17"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "NA PB\t1717484\tWednesday, August 08, 2007 Before I write anything else tonight (and believe me have some stuff to say - screams of joy are much better than silences of despair), I want to talk about Jacqueline Howat. She said that a police recruit (who has name suppression) raped her. He has just been found not guilty , after the judge told the jury to consider the credibility of the witnesses. The subtext of the defence was, she did too much drugs, she had too much sex, she couldn't tell you who attacked and raped her. The man is applying to re-enter police college - I'm not feeling any safer. 2 comments: I just stumbled onto the news about this case tonight - addressed in post, Gee, why is it women and youth in the sex trade won't trust law enforcement*? - and would be interested to know if you've heard anything new about the case. (I'm in Richmond, Virginia in the U.S., so you're obviously a lot closer to the relevant news sources here, though I've dug up some stuff online.) I understand the police recruit was found not guilty, but is still seeking to restart his police career - a notion so infuriating to me I don't know where to begin. Most maddening of all to me, though, is the fact that he gets to remain anonymous. In any event, I applaud the bravery of Jacqueline Howat for at least pursuing the charges. That had to be a rough situation. I hope she is able to recover from this travesty. Not really. This case got swamped with a more profile case (which had a better result). I imagine we won't hear anything more about it. The coverage made me so angry. Although the radio journalist coverage did a passably ok job. \"The feminist agenda is not about equal rights for women. It is about a socialist, anti-family political movement that encourages women to leave their husbands, kill their children, practice witchcraft, destroy capitalism, and become lesbians.\" \n",
            "NA PB\t0302026\tThe BBC will support social innovation by encouraging users' efforts to build sites and projects that meet their needs and those of their communities...This is exemplified through plans for Backstage, a public site for the BBC's in-house development teams to share development plans with their peers and audiences. In a similar way to Google's Labs test-site, this will be a place to demonstrate work in progress, share expertise and invite contributions and collaboration with expert users. The BBC will also be committed to using open standards that will enable users to find and repurpose BBC content in more flexible ways It was a laudable aim, and the site launched with some prototypes built by internal people who had been given early access to some of the feeds. I contributed two efforts. \"Where is the BBC News?\" crudely parsed news headlines for country names, and made the countries currently being mentioned light up on a map, and the BBC Complaints RSS feed, which as I mentioned yesterday , was an effort at screen-scraping to produce content in a format that the BBC should have been publishing anyway. The public launch was at Open Tech 2005 , where it understandably over-shadowed my simultaneous talk on \" Putting a F__k Off Dalek on the BBC Homepage isn't big or clever \" \"It's one of the quirkier products of the BBC's new Backstage project, where people with enormous brains, mad haircuts and pale complexions do extraordinary, innovative and often frightening things with our content.\" As I said back in 2005 though, I was pleased that the yay-sayers overcame the nay-sayers to launch Backstage, which really was quite a shock to some of the closed 'old media' thinking within the BBC at the time. I'm glad that I got a chance to work with people like Ben Metcalfe , Matt Cashmore , Ian Forrester and Rain Ashford , and I wish Ian and Rain all the best with what they go on to do next. By Martin Belam on 6 January 2011 I'm the founder of Emblem , a digital consultancy offering user experience design, information architecture and training services. I've spent over a decade building products for brands like the Guardian, Sony, Vodafone and the BBC, and now work with clients in the publishing, media, arts, heritage and culture sectors. To find out about working with us, please drop us an email . 1 Comment Ah, I have fond memories of BBC Backstage. A friend and I remember when it was launched (well, it wasn't too long ago, after all, which is why I can still remember it!) and enjoyed more than one chuckle at the hacks (viva Daleks). BBC Backstage is dead; Long live BBC Backstage. \n",
            "NA PB\t0427884\tWelcome I am a process artist, a teacher and a facilitator of social technologies for face to face conversation in the service of emergence. My business is supporting invitation: the invitation to collaborate, to organize, to find one another and make a difference in our communities, organizations and lives. In a changing and complex world, there are no answers, there is no certainty. Leadership and questions are everywhere. We are called to be in active engagement with the world around us, to make sense of things we are seeing and to act on our visions and ideas for the good and benefit of all. The fundamental human capacities of this era are among the most ancient: invitation and conversation. When we invite people to work with us, whoever shows up are the right people. When we live a life of invitation, our work becomes about making connections to make things happen. And once we have a group of people acting on an invitation, deep and meaningful conversation becomes the way we collaborate sustainably together to co-create the world we want. I work with organizations seeking to improve their work, communities seeking to improve their future, people looking to improve their lives. I hold and care for process -- the ways in which we work together -- to encourage people to make their best possible contributions. I have an unflagging belief that the answers and leadership we need arise out of collaboration and conversation. By facilitating skillful dialogue, I do my best to hold space for futures to emerge. \n",
            "NA PB\t1762533\tFeeling Good. Feeling Bad. And learning to spot the difference. It's an incredibly subtle and delicate business. Trying to detect and quantify the daily changes in my mental and physical condition. As I've observed before, the first big lesson to learn is that cancer is, by and large, a slow, drawn out affair. Nothing happens over-night. At least if it does, it's such an incremental change that it's easy to overlook. I also have to filter the data, search through the background static of everyday common-or-garden fifty year old pathology and neurosis -- the aching bones, headaches and oral decay, to try and spot any meaningful trends. It's like panning for gold, only less rewarding. So the only useful way to answer that question, is to compare how I am feeling today, not with yesterday, but with a month ago. Or three months ago. Are there any patterns to be detected, or trends to discover? Well let's see. Looking at photos from last spring, I don't look much different. My weight is about the same, if anything I've gained a few pounds over Christmas and the new year. Also on the up side, many of the accumulated years of worry lines and wrinkles seem to have smoothed themselves out, I can only put this down to my premature release from the advertising mad-house. Against all the predictions, after three rounds of chemo I seem to have held onto most of my hair, It's a bit thinner, and a little greyer, but still hanging on to my skull tenaciously. My head, arms and legs are tanned brown from the sun and wind, the rest of my carcass is still official United Kingdom white, and untidily punctuated with random moles and freckles. In the middle of my mostly hairless chest however, is a patchy brown rectangle about the size of a paperback book. There is a corresponding blemish in the middle of my back just under my shoulder-blades. They are souvenirs from the radiation treatment I enjoyed back in November. I have no idea if they are permanent but I consider them something of a badge of honor A couple of inches bellow my right collar bone is a pronounced bump about the size of a two dollar coin. (that's 50p to you limeys), which is the site of my porta-cath. This is the filler-cap where I take delivery of my chemo drugs. As the surgeon who installed it for me promised, this has been largely forgotten about now, I'm really only reminded of its presence when I'm wearing my shoulder bag, or a car seatbelt, although Jo tells me I play with it absent-mindedly when I'm watching TV. So, externally at least. No real changes to report. Jo reckons I turn yellow during the middle week of each cycle, but she's the only one to have mentioned it -- probably everyone else is to polite. Internally who know's what is going on. Apart from the intermittent nausea and light-headedness brought on by my the various drugs, I feel mostly normal. I can't 'feel' my cancer. I thought I could in the early days, but that's gone now. What does a tumor or a lymph node feel like anyway? Oh! Tinitus. I almost forgot. This is a new one. A high pitched ringing in my ears. Not all the time, just intermittently. How a drug that is supposed to kill cancer cells in your stomach can make your ears ring is completely beyond me. But there you go. Apparently it's not a big deal. Unless it becomes constant and permanent, in which case it will be a very big deal. The thing I'm watching most closely is my energy levels. This seems as good a marker as any for my overall condition. I'm not sure if these are improving, or if I'm just getting used to operating on a lower power setting. For the first one or two cycles, I did find myself retiring to the bedroom for a little mid-afternnon siesta most days, particularly in the middle and last week of the three week cycle. This time 'round I've done this less, and coped pretty well, although after a couple of active days, I often feel the need to have a sedentary day to recover. Across the street from our house is a reserve, and along the side of it a leafy, overgrown footpath used by the locals as a short-cut to the village. We probably walk up and down this path at least once a day, and often more. It's a steep hill down to the village, and a fair pull back up to the cottage. This is my personal fitness gauge. not very scientific granted, but a good indicator of how I'm doing. Some days I can breeze up the grassy incline without even getting out of breath (with forty-a-day Jo bringing up the rear). on other days, my legs are burning, and I'm sweating and gasping for air by the time I get to the top. Weird. Since before Christmas I've had a niggling low level infection. Ear ache. Tooth ache. Sore throat. Dry cough. My voice is broken and croaky, and sometimes gives out altogether. It comes and goes, but never clears up entirely. It's annoying rather than debilitating, and is almost certainly down to my low immunity levels, brought on by a low white blood cell count. I had my fourth chemo cycle, due to start last Wednesday deferred for a week to allow my 'bloods' to recover, but I have a feeling this is something I'm just going to have to put up with for the duration of my treatment. Compared with some of the horrors many of my fellow travellers have to endure, this is just a minor irritation and I'm really not complaining. So how am I feeling? Well, down but not out. Mentally I'm feeling strong and positive. Physically, a bit depleted but nothing like as bad as I was expecting, and I get the feeling that even the doctors are quietly impressed, although they don't give much away. Most of them would make excellent poker players. The chemo clearly has a a slow but insidious cumulative effect, and my job is to try and stay on top of things for another three cycles -- possibly more. Chemo kills cancer. The more of it I can stand, the better my chances are. Anyone for tennis? * * * An big appology and a small celebration For anyone who has left comments recently, and was wondering where they have disappeared to, or why they have gone unanswered, I finally figured it out. For some reasons beyond my ken all your comments were being redirected directly to the spam folder on my blog site. I was beginning to think you had all given up on me. I think I've sorted this out now, so hopefully normal services have been resumed. Please don't stop leaving comments, I love hearing from you, and it keeps me motivated to write. You might be interested to learn that we are coming up to 12,000 visits on the blog since last October, which just goes to prove what a creapy, morbid bunch of sickos you are. My kind of people. 9 views shared on this article. Join in... Thank goodness you sorted out the glitch. My messages to you were just not getting through, all of them encouraging, some even a bit witty! Glad to hear you are faring well and Richard, Sam and I raised a glass to your news at Christmas. We might be coming back for a visit end of Feb so it would be good to catch up. Lots of love. Txxxx Hey Linds. Tricia pointed me this way but it's taken a while to search the blog out. Plus I allowed myself the luxury of unplugging from the Matrix while on holiday. Glad to see you're still in the ring with your gloves up. Actually I think your mother is right; you should turn this into a book. I've never met you or heard of you until now. I don't work in advertising and i'm not a cancer survivor but I stumbled upon your blog via a link on a friend's facebook page and now i'm hooked. Your story appeals to me not because it's a story about dying but because it's a story about living; what it means to be alive. It has made me question what's important, and what's not, what to reflect on in life and what to let go of. As someone who has suffered with depression my whole life, I find your posts and views on life uplifting and inspiring. You have made me laugh and cry. \n",
            "NA PB\t3144416\t7.00 Up for breakfast and then break camp 8.30 leave camp site for another wreath laying this time at Ospel memorial site, this is another memorial erected by a local family who were moved to do something in thanks for the 7th Armoreds sacrifices in the area. 9.30 short lecture by Niek Hendrix about the fighting around the area of Ospel . It was Niek and his father Harry Hendrix who had the memorial erected and dedicated in May 2007. Harry was one of only a few civilians who stayed in the area during the fighting. 10.30 We moved of again this time to Weert where we were we had been asked by the Dutch army to attend their military academy passing out ceremony. Some of the vehicles parked in the town square where the ceremony took place, but because the two Sherman's were too heavy we were parked on the main road near by. At 15.00 after the passing out ceremony had taken place there was a parade through the town by the Dutch Army and again we were asked to join them with all of our vehicles. After the parade we moved off to some park land in the town and set up camp. 18.30 evening meal 19.30 March into town to liberate the bars and clubs. Saturday 6th October The camp area was open to the public to walk around and chat to us. The weather had turned wet and so not many turned up, but those who did were very interested and the older folk had some fascinating tales to tell. 11.00 Wreath laying ceremony at the bridge of Weert. 18.30 Evening meal then wash and change into class \"A\" uniform 21.00 Into town for a \"Special\" evening and more liberation of bars and clubs. Sunday 7th October. 7,30 breakfast and then it was break camp for the last time. At about 14.00 hours we lined the vehicles up for loading. The Dutch army were so impressed by the whole group that they actually supplied 8 low loaders for the return trip to the port. So all of the vehicles were loaded and the crews went in a coach also provided by the army. It was good to meet up with old friends on this trip from all over the continent, like Olaf, Tim and Thomas Plus others whose names slip my memory from Holland .Martin Dusek, His brother and their team from Czech Republic.. 21.30 Sail from Hook of Holland to Harwich Monday 8th October Loaded My Sherman on the lowloader then rode back the Copemans farm where we said our good byes. \n",
            "NA PB\t3193856\tI have followed the recent controversy at my alma mater, Sacred Heart Academy, with keen interest. I have passionate feelings about the decision not to let Dominic Sheahan speak at the 2012 graduation ceremonies, but for many, those opinions would distract from the point of this commentary. I wouldn't want to do that. While I have empathy for everyone caught in this crossfire, my prayers and emotional connection have largely been with a man who I am quite certain has been saddened and hurt by events of the last month, Dominic's grandpa, Jerry Sheahan. Jerry Sheahan is one of the most pivotal figures in my blessed life. I have never publicly told him that. In this very difficult month for Jerry, and before either of us pass; it strikes me as important to tell him and the world. I also think the tale carries larger lessons for all of us. I was born in Mt. Pleasant in 1949 with Arthrogriposis Multicongenita, a congenital birth defect that deformed my limbs. I required braces and almost annual surgeries. One spring day in 1956 Jerry Sheahan called my dad, Jim McGuire, and asked if Tim could be batboy for Sheahan's highly successful Little League baseball team. Sheahan and his co-coach (the late) Bob Wohlscheid were both parishioners at Sacred Heart. They had known my parents, Jim and Anita, for years. Without much notice, a uniform appeared at the house and that evening I was in the Roosevelt Oil team's dugout. I arrived at the West Side stadium that night more than a little confused. Jerry briefed me on my duties and I tentatively followed his instructions. I vaguely knew only a couple of the players. The game had barely begun when a photographer from the local newspaper, the Mt Pleasant Daily Times-News, showed up at the game. The photographer took one picture of my back looking onto the field. My omnipresent braces are prominent in the photo. The second picture the paper published showed me handing a bat to a player well more than a head taller than me (Joe Feldman) and the caption said I told him to hit a homerun. Both captions took considerable liberties describing me as a \"gallant sparkplug' and decreeing I \"handled the bats like diamonds.\" At the time I was no gallant sparkplug and I don't think I had the good sense to tell anyone to hit a home run. There would come a day, however, when I did handle those bats like diamonds and all those characterizations would become more or less true. Roosevelt Oil (later Leonard Refinery) and Little League became an integral part of my springs and summers for the next five years. Usually the season was close to complete by the time I headed for my annual summer surgeries and casts. I still remember minute details of games, seasons and players. If the players regarded me as an oddity I never felt it for a moment. My sense of team and self-esteem was as strong as the fastball of the star pitcher, John Schade. I fondly remember team picnics at amusement parks and other team outings. I was always front and center in team photos and celebrations. To this day, my good friend Mick Natzel, talks about when we \"played\" Little League together. Sheahan and Wohlscheid accomplished their probable goal. The sports-crazed little kid who would never play baseball felt like he belonged to a team. I never felt like the lonely, handicapped boy. Sheahan, Wohlscheid and the players made me feel as if I had a real role with a championship team. These days I read a lot about the dramatic increase of shameful bullying in schools. It angers me and saddens me that some youngsters are harassed to the point of suicide. News stories like that make me reflect on two things. First, I reflect on the two young girls I remember in school that were unmercifully teased. Every time I read one of those stories I have a major pang of regret that I never helped them. Yet, I know why I didn't. I was just so bloomin' grateful that I wasn't the one the other kids were teasing. That's when I marvel at how little I was picked on as a young kid even though I \"walked funny.\" I am convinced much of that was due to my young friends in the community. Practically every kid under 12 knew the batboy for Roosevelt Oil and they knew his team liked him. With the frequent help of my young friends from my Little League team, Mike Hackett, Bill McDonald and my brother Marty I survived those prepubescent years in fine shape. I knew there were always people there to help me. Jerry Sheahan and Bob Wohlscheid with one big act of kindness offered me a hand up. It changed my young life. I salute Jerry and Bob and I salute all my friends who saved me from bullying. At the same time I wonder if there is a youngster each of us could save from bullying by caring as much as Jerry and Bob did. And, I wonder if we're coaching our kids enough to stand up for those youngsters around them who are getting the short end of the stick. Tim McGuire is a Mt Pleasant native who retired as Editor of The Minneapolis Star Tribune in 2002. He is now The Frank Russell Chair for the Business of Journalism at The Walter Cronkite School of Journalism at Arizona State University . \n",
            "NA PB\t3357569\tInto The Pacific & Off The Richter Scale 7.7 magnitude earthquake reported off the coast of British Columbia last night. There were some reports of slight tremors felt in the Vancouver downtown area and a tsunami warning was in effect. This did not stop us from exploring today and trying to spot whales. We took off and the skies were cloudy and we experienced a heavy fog for a while. It cleared up fairly well once we navigated a bit further so that was a plus. Curious cat over here sneaked up into the Captain's cabin after spending some time on the outside deck and had a chat with him. Laid back & friendly dude I must say. He showed me how the sonars worked, how to check water levels and temperature and a whole ton of other stuff such as measuring speed in nautical miles. Where are the whales? 525 feet deep and 8 degrees Celsius. You do not want to be in the water. I have very vivid lucid dreams from time to time and they are simply beautiful. I have decided from this point on to share every single one that I encounter and give you an insight on some of the imagery I have very vivid lucid dreams from time to time and they are simply beautiful. I have decided from this point on to share every single one that I encounter and give you an insight on some of the imagery Totem poles are truly a work of art. Standing underneath one can give one a sense of power they exude. I stood there for a long time, zoned out, absorbing every minute detail that was carved out. Typical of me Totem poles are truly a work of art. Standing underneath one can give one a sense of power they exude. I stood there for a long time, zoned out, absorbing every minute detail that was carved out. Typical of me Many conversations in my life involving a total stranger are opened with a simple hello. Toss in a charming smile in the mix and you can walk right through that now open door. Noticed this on the steps of Emily Many conversations in my life involving a total stranger are opened with a simple hello. Toss in a charming smile in the mix and you can walk right through that now open door. Noticed this on the steps of Emily Decided to hop around the Vancouver Aquarium today. Caught a 4D documentary, saw some really cool stuff, watched a Beluga whale show (those things were very talented) and saw some people get slapped with water from top to bottom as Decided to hop around the Vancouver Aquarium today. Caught a 4D documentary, saw some really cool stuff, watched a Beluga whale show (those things were very talented) and saw some people get slapped with water from top to bottom as Stumbled upon these well crafted water houses as we were walking around Granville Island. Imagine just getting away from it all for a while and living in one of these. Blissful right? Planning long term and visualizing the future on Stumbled upon these well crafted water houses as we were walking around Granville Island. Imagine just getting away from it all for a while and living in one of these. Blissful right? Planning long term and visualizing the future on \n",
            "NA PB\t3033110\tSunday, 18 November 2012 Wiki challenge BBC style Further to my recent discovery of the BBC's take on our own wiki challenge, I've just been over to the official site to have a look at next week's subjects, and the links that the official website gives. Now, I'm a bit shamefaced to admit this, being an English teacher, but the one I'm least likely to score any unwikied points on is Graham Greene, despite the fact that I did once meet him while I was at university. He struck me as a bit of a grumpy old devil, if truth be known, but then I doubt he was all that impressed with me. I've never read any of his stuff, but then I'm sure that he never read any of mine either. Enough of such pointless chaff. These are the links that the Beeb gives for the subjects : - Interesting to see that they do give Wikipedia as one of the Graham Greene links. I have had just a cursory look at these pages, and in some cases I do wonder just how much help they would be if you were going to do a wiki-style challenge on these rounds. I doubt that I'm going to have enough free time this week to take the challenge myself. Still, if anyone does take the challenge with any of these web pages I'd be really interested to hear how useful you actually found them. \n",
            "NA PB\t3197434\tHip Replacement for the Orthopedic Surgeon -- On the Other Side of the Scalpel -- pt.11&12 24 hours post-op Dr. Alan Beyer has had all of his drains and catheters removed, has had two physical therapy session, walked a few laps and is feeling good. He still hasn't needed anything more than Tylenol for the pain, but warns that that isn't for everyone. It's going home day for Dr. Beyer. Day two after surgery and he has had his dressings changed a few times and it's looking good. Still only using Tylenol for the pain. He reminds us of the importance of taking it easy once getting home -- just because you're home, doesn't mean you're healed! Take it slow, do your exercises and let everything heal! It takes time, and doesn't heal faster just because you want it to. Hoag Orthopedic Institute specializes in the most advanced orthopedic surgery, total joint replacement surgery and spine surgery. From the newest minimally invasive techniques to total joint replacement to complex spine procedures, their expertise in orthopedic medicine translates to better results and fewer complications. \n",
            "NA PB SS\t3027643\tI heard distant booming. Was it the sound of bombs or bitterns, warfare or ornithology? I decided I had to find out, for there was a great deal of difference between the two, as I am sure you realise. If it was the boom of exploding bombs, and war had been declared, then it was no comfort to me that the booms were distant. Today they might be, but I was in no doubt that the marauding and rapacious foe, having bombed land far away, would advance, and bomb ever closer to the agreeable faubourg where I lived, until they bombed my own neighbourhood, and me in it, unless I fled. Paradoxically, perhaps, before making the decision to flee I would have to do the opposite, to approach the distant booms, to discover if they were indeed the booms of bombs. If, on the other hand, the booms were the booms of bitterns, I would have no need to flee. I could return happily to my faubourg without a care in the world. If anything, I would be happier than before, having learned of the presence of bitterns, albeit distant bitterns. If there were bitterns, I reasoned, there would be a lake, or a similar body of water. There are few things I like better than to go plashing about up to my knees in a lake, plunging my hands into the water now and again with scarcely credible rapidity, to pluck out a fish for my dinner. It is a skill I learned at my mother's knee. She had studied the sudden darting movements of certain insects, and learned through long hard practice to imitate them. All I had to do, with equally long hard practice, was to imitate her. Surely, you might ask, given my predilections I would know of the existence of a lake, albeit a distant one? It is a reasonable question. All I can say by way of explanation is that I had never before heard booms, booms that might be those of bitterns, from that direction, north-west, by the way. To which I would add that there has been a flurry of artificial lake and pond construction all across the land of late. The king is keen on bodies of water, and dreams of a land riddled with them. Some have joshed that if he gets his way there will be more water than land, just little ribbons and eyots dotted higgledy-piggledy in a great network of ponds and lakes connected by canals. But the king is also a bellicose fellow with a seething hatred of . . . oh, you name it! His list of hatreds is long, and it includes the neighbouring lands, their kings and queens and armies and civilian populations. So it would be no surprise if the booms I heard were the booms of bombs. The king might well have declared war, almost as an afterthought or aside, or one of his shouty declamations from his balcony could have led a neighbour to declare war on him, which, by extension, means us, and me. Given the king's twin obsessions, I judged that the probability of the booms being bombs or bitterns was about fifty-fifty. That was in the absence of any other information. My faubourg is quite an isolated one, and we are often the last to hear any news of national import. Last time there was a war, for example, we only found out about it when there were pitched battles in the streets between our side and the marauding and rapacious foe. That foe had not made use of bombs. They were somewhat primitive barbarians armed with cutlasses and catapults. Nor do we tend to get news of the king's latest lake-creation schemes. Generally what happens is that the plans are made in his chamber, by him alone, and then the bulldozers move in and villages are destroyed and a huge pit dug into which unimaginable gallons of water are poured. Fish and birds follow, drawn by nature. Over the next week or so, I made preparations for my journey. I put my faubourg affairs in order as best I could, and prepared an enormous number of sandwiches. These I wrapped in greaseproof paper. I bought, at a discount, several multipacks of Squelcho!. I patched the odd hole in my knitwear, and had my hiking boots renovated by the best hiking boot renovator in the faubourg. I mention these things to demonstrate that I was serious in my endeavour, not a mere flibbertigibbet disconcerted by distant booms. All the while, I could not help noticing that these distant booms were growing louder and thus, I reasoned, closer. This would tend to tip the balance in favour of the booming of bombs, but there was no knowing the level of fanaticism to which the king might take his body of water building activities, so there was still a fair chance that the booming was the booming of bitterns. There came a day when I had nearly finished putting my affairs in order and had packed most of my supplies. By now the booms were so loud, and so close, that I began to doubt I would need to go investigating at all. So unlikely, now, was the prospect of my journey that I tucked in to several of the sandwiches and drank two cans of Squelcho!. No sooner had I shoved the crumpled greaseproof paper and the crushed cans down my waste chute, than a bomb crashed through the ceiling. It landed on my rug, and made a ticking noise. I reasoned that I had mere seconds to dash as far away as possible before it exploded with a boom. At least, I thought, I now know the provenance of the booms. But as I made for the front door, ready to scarper, my ears were assailed by a deafening boom. Dizzy with panic, I pirouetted round and round like a man who has lost his senses. I was utterly befuddled that my chalet still stood, solid and safe and agreeable apart from the gaping hole in the ceiling. And then, as I peered out into the front garden, I saw a flock of bitterns had come to land next to my garden puddle, and they were booming, booming, booming. \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! zcat dev.tsv.gz | egrep \"^[A-Z]+ PB\" | cut -f 3 > pb.txt # direct these to a file for simplicity"
      ],
      "metadata": {
        "id": "LCPlrBLdBAHz"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "! cat pb.txt | tr '[:upper:]' '[:lower:]' | tr '[0-9]|[:punct:]' ' ' | tr ' ' '\\n' | head -100 # one per line and normalized"
      ],
      "metadata": {
        "id": "KNVKTubv_Vtn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aed5c8e9-ffd0-418b-e66e-e8c336d50b9d"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "wednesday\n",
            "\n",
            "august\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "before\n",
            "i\n",
            "write\n",
            "anything\n",
            "else\n",
            "tonight\n",
            "\n",
            "and\n",
            "believe\n",
            "me\n",
            "have\n",
            "some\n",
            "stuff\n",
            "to\n",
            "say\n",
            "\n",
            "\n",
            "screams\n",
            "of\n",
            "joy\n",
            "are\n",
            "much\n",
            "better\n",
            "than\n",
            "silences\n",
            "of\n",
            "despair\n",
            "\n",
            "\n",
            "i\n",
            "want\n",
            "to\n",
            "talk\n",
            "about\n",
            "jacqueline\n",
            "howat\n",
            "\n",
            "she\n",
            "said\n",
            "that\n",
            "a\n",
            "police\n",
            "recruit\n",
            "\n",
            "who\n",
            "has\n",
            "name\n",
            "suppression\n",
            "\n",
            "raped\n",
            "her\n",
            "\n",
            "he\n",
            "has\n",
            "just\n",
            "been\n",
            "found\n",
            "not\n",
            "guilty\n",
            "\n",
            "\n",
            "after\n",
            "the\n",
            "judge\n",
            "told\n",
            "the\n",
            "jury\n",
            "to\n",
            "consider\n",
            "the\n",
            "credibility\n",
            "of\n",
            "the\n",
            "witnesses\n",
            "\n",
            "the\n",
            "subtext\n",
            "of\n",
            "the\n",
            "defence\n",
            "was\n",
            "\n",
            "she\n",
            "did\n",
            "too\n",
            "much\n",
            "drugs\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! cat pb.txt | tr '[:upper:]' '[:lower:]' | tr '[0-9]|[:punct:]' ' ' | tr ' ' '\\n' | egrep -v \"^$\" | sort | uniq -ci | sort -rn > pb-freks.txt # also empty lines away!"
      ],
      "metadata": {
        "id": "SW_zguxrAd7q"
      },
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# let's run the same pipeline now for travel blogs\n",
        "! zcat dev.tsv.gz | egrep \"^[A-Z]+ TB\" | cut -f 3 > tb.txt\n",
        "! cat tb.txt | tr '[:upper:]' '[:lower:]' | tr '[0-9]|[:punct:]' ' ' | tr ' ' '\\n' | egrep -v \"^$\" | sort | uniq -ci | sort -rn > tb-freks.txt\n"
      ],
      "metadata": {
        "id": "zfif2KPRHx_z"
      },
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "! cat tb-freks.txt | head -5 # to make sure the file looks ok"
      ],
      "metadata": {
        "id": "2_XD2M-aIcE4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e9188bca-3a7f-4b09-db81-6c549187d8ba"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "   1829 the\n",
            "    879 and\n",
            "    788 to\n",
            "    751 of\n",
            "    707 a\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 14. How many unique words do the two registers have?\n",
        "\n",
        "! cat pb-freks.txt | wc -l # how many unique words\n",
        "! cat tb-freks.txt | wc -l\n",
        "# or even\n",
        "! wc -l *freks.txt"
      ],
      "metadata": {
        "id": "249ILl9PEWay",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9e8360bd-ab12-408a-c851-ef5cb7977205"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "18838\n",
            "5380\n",
            " 18838 pb-freks.txt\n",
            "  5380 tb-freks.txt\n",
            " 24218 total\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 15. How many occur at least 100 times?\n",
        "! cat pb-freks.txt | egrep \"^ *[0-9][0-9][0-9] \" | wc -l # another option would be \"^ *[0-9]{3} \" where {} denote how many numbers we want"
      ],
      "metadata": {
        "id": "fltaAFlwEhoi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "03c5b0c2-1a82-4054-bdcf-3031a968fcf8"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "241\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! cat tb-freks.txt | head -150 | tail -50 # the beginning is so uninformative I wanted to look at the ranks 100-150"
      ],
      "metadata": {
        "id": "gPxjIn95GGCb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "425a60f1-838c-4e09-dca7-5771fc768fa2"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "     33 little\n",
            "     33 how\n",
            "     33 here\n",
            "     33 good\n",
            "     32 its\n",
            "     32 could\n",
            "     31 trip\n",
            "     31 night\n",
            "     31 new\n",
            "     31 go\n",
            "     30 sea\n",
            "     30 got\n",
            "     30 bit\n",
            "     29 best\n",
            "     29 before\n",
            "     28 world\n",
            "     28 take\n",
            "     28 old\n",
            "     28 long\n",
            "     28 even\n",
            "     27 home\n",
            "     27 few\n",
            "     26 went\n",
            "     26 well\n",
            "     26 took\n",
            "     26 place\n",
            "     26 park\n",
            "     26 now\n",
            "     25 while\n",
            "     25 made\n",
            "     25 country\n",
            "     25 any\n",
            "     24 walk\n",
            "     24 those\n",
            "     24 right\n",
            "     24 much\n",
            "     24 food\n",
            "     24 along\n",
            "     23 ve\n",
            "     23 think\n",
            "     23 still\n",
            "     23 decided\n",
            "     23 d\n",
            "     22 three\n",
            "     22 make\n",
            "     22 bus\n",
            "     21 years\n",
            "     21 reef\n",
            "     21 re\n",
            "     21 her\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! cat pb-freks.txt | head -150 | tail -50 # the beginning is so uninformative I wanted to look at the ranks 100-150"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IP5FzQ7_4LcL",
        "outputId": "20f89f16-8ffd-48b3-c59e-8ee4a1d96ffe"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "    332 work\n",
            "    332 here\n",
            "    329 little\n",
            "    318 after\n",
            "    316 off\n",
            "    313 still\n",
            "    311 go\n",
            "    310 going\n",
            "    309 through\n",
            "    307 life\n",
            "    307 around\n",
            "    303 most\n",
            "    295 two\n",
            "    288 being\n",
            "    284 things\n",
            "    284 any\n",
            "    283 where\n",
            "    282 d\n",
            "    281 year\n",
            "    278 too\n",
            "    273 those\n",
            "    272 got\n",
            "    271 did\n",
            "    270 love\n",
            "    268 these\n",
            "    268 last\n",
            "    264 few\n",
            "    261 long\n",
            "    255 years\n",
            "    255 take\n",
            "    251 him\n",
            "    250 want\n",
            "    248 before\n",
            "    241 great\n",
            "    241 down\n",
            "    239 again\n",
            "    238 made\n",
            "    231 many\n",
            "    230 home\n",
            "    228 never\n",
            "    227 need\n",
            "    224 right\n",
            "    220 something\n",
            "    217 find\n",
            "    214 week\n",
            "    214 should\n",
            "    214 say\n",
            "    213 always\n",
            "    212 blog\n",
            "    210 lot\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Exercise 3 (if there is still time) (not checked by Anna ;) )\n",
        "\n",
        "Let's work on some Tweets. Unfortunately these are in Finnish, but use the file at http://dl.turkunlp.org/atp/all_tweets_one_line.txt.gz\n",
        "\n",
        "Basic stats\n",
        "* How many tweets?\n",
        "* Can you figure out what the tweets are about?\n",
        "* When have they been Tweeted?\n",
        "* Can you make a distribution of how many tweets per month the data has? The `cut` command can be useful for this. `-d ` can specify the delimiter, so e.g.,  `cut -f 1,2 -d ' '` will get the first and second columns separated by a whitespace.\n",
        "\n",
        "Contents\n",
        "* Let's count the most frequent words of the tweets\n",
        "* And also the most important hashtags. And user names.\n",
        "These should tell us what we have in the data!\n",
        "\n",
        "Keywords by time and / or person?\n",
        "\n",
        "* Let's count username or time-related keywords with the python above script. How do the keywords look like, can you trace anything interesting?\n"
      ],
      "metadata": {
        "id": "RUH-e4nF_1az"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! wget http://dl.turkunlp.org/atp/all_tweets_one_line.txt.gz"
      ],
      "metadata": {
        "id": "UvQmvCbsqW5b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bc23cd0e-7eef-489a-ff27-739b2b62e31a"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-11-13 18:55:08--  http://dl.turkunlp.org/atp/all_tweets_one_line.txt.gz\n",
            "Resolving dl.turkunlp.org (dl.turkunlp.org)... 195.148.30.23\n",
            "Connecting to dl.turkunlp.org (dl.turkunlp.org)|195.148.30.23|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 89304688 (85M) [application/octet-stream]\n",
            "Saving to: ‘all_tweets_one_line.txt.gz’\n",
            "\n",
            "all_tweets_one_line 100%[===================>]  85.17M  42.4MB/s    in 2.0s    \n",
            "\n",
            "2024-11-13 18:55:10 (42.4 MB/s) - ‘all_tweets_one_line.txt.gz’ saved [89304688/89304688]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! zcat all_tweets_one_line.txt.gz | head -5 #again a tweet per line!\n",
        "! zcat all_tweets_one_line.txt.gz | wc -l # this many tweets"
      ],
      "metadata": {
        "id": "pHFyOou02l9I",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "71a07075-d3aa-4166-f7e7-7d5ac2eaa9f3"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "###C:\t1244776401124851713\t2020-03-30 23:59:55\thexademus1\t1234607328009302019\t8\ttweet\t['valtioneuvosto', 'MarinSanna']\t['koronafi', 'koronasuomi', 'koronaepidemia']\t@valtioneuvosto @MarinSanna Yllättävän hyvin hallitus on hoitanut asian ja pysynyt Ruotsin linjoilla. Siellä on perinteisesti tehty yhteiskunnan kannalta hyviä päätöksiä ja maa on menestynyt. #koronafi #koronasuomi #koronaepidemia\n",
            "###C:\t1244776074380292104\t2020-03-30 23:58:37\tjukka235\t1225011411648237573\t153\ttweet\t[]\t['koronavirusfi', 'korona', 'covid19fi', 'covid19finland', 'covid2019']\tThe Lancetissa julkaistu tutkimus Kiinan aineistosta: potilaat olivat keskimäärin 18.5 päivää sairaana ennen kuolemaa.  7/x  #koronavirusfi  #korona  #covid19fi  #covid19finland  #covid2019  https://t.co/VKzxCtG8C0\n",
            "###C:\t1244774409937850369\t2020-03-30 23:52:00\tmarsusanoo\t1928168198\t1477\ttweet\t['miksuksi']\t[]\t@miksuksi kiitos! luulen, että itellä lähtee vasta nyt rullaamaan terapia kunnolla, kun ei ole ketään, kenestä kantaa huolta. lamaantuminen tulee varmasti hetkeksi, mutta se tulee kotona turvallisissa olosuhteissa ja se on jo ennakoitu.\n",
            "###C:\t1244774080538120192\t2020-03-30 23:50:42\tjukka235\t1225011411648237573\t153\ttweet\t[]\t['koronavirusfi', 'korona', 'covid19fi', 'covid19finland', 'covid2019']\tKuvan tulkinta:  -   ensin pitää tehdä oletus siitä, montako päivää kului tartunnan saamisesta kuolemaan -   esim. 30.3. yhteensä 13 kuollutta, 15 pv aikaisemmin (15.3.) yht 244 tartuntaa, 13/224=5.3% 5/x  #koronavirusfi  #korona  #covid19fi  #covid19finland  #covid2019\n",
            "###C:\t1244773402814095360\t2020-03-30 23:48:00\tjukka235\t1225011411648237573\t153\ttweet\t[]\t['koronavirusfi', 'korona', 'covid19fi', 'covid19finland', 'COVID2019']\tArvio kuolleiden määrästä (%). Avainkysymys: mitä tapahtui niille, jotka sairastuivat samaan aikaan? Esim. niistä, jotka olivat sairaana 15 päivää aikaisemmin, on kuollut noin 5.3%.  4/x  #koronavirusfi  #korona  #covid19fi  #covid19finland  #COVID2019 https://t.co/RWBXMHPNsJ\n",
            "691128\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# it would seem that the column 3 has the time stamps. Let's see how the head and tail look like\n",
        "! echo \"head\"\n",
        "! zcat all_tweets_one_line.txt.gz | cut -f 3 | head\n",
        "! echo \"tail\"\n",
        "! zcat all_tweets_one_line.txt.gz | cut -f 3 | tail\n",
        "\n",
        "#Based on these, we could guess that the time stamps range from March 2020 to August 2021. But better yet sort the time stamps and then check the head and tail!"
      ],
      "metadata": {
        "id": "wMeBXQzW37p-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "069eb54e-efd1-4d05-efd5-cbee975d0bf5"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "head\n",
            "2020-03-30 23:59:55\n",
            "2020-03-30 23:58:37\n",
            "2020-03-30 23:52:00\n",
            "2020-03-30 23:50:42\n",
            "2020-03-30 23:48:00\n",
            "2020-03-30 23:47:39\n",
            "2020-03-30 23:46:08\n",
            "2020-03-30 23:43:33\n",
            "2020-03-30 23:38:08\n",
            "2020-03-30 23:38:08\n",
            "tail\n",
            "2021-08-06 10:15:23\n",
            "2021-08-06 10:15:48\n",
            "2021-08-06 10:16:02\n",
            "2021-08-06 10:16:30\n",
            "2021-08-06 10:17:27\n",
            "2021-08-06 10:17:59\n",
            "2021-08-06 10:19:49\n",
            "2021-08-06 10:19:50\n",
            "2021-08-06 10:20:13\n",
            "2021-08-06 10:20:36\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! echo \"here's the head\"\n",
        "! zcat all_tweets_one_line.txt.gz | cut -f 3 | sort | head\n",
        "! echo \"here's the tail\"\n",
        "! zcat all_tweets_one_line.txt.gz | cut -f 3 | sort | tail\n",
        "# oups, in fact the tweets start from 2019 and continue until August 2021. We almost missed the beginning!"
      ],
      "metadata": {
        "id": "qhlhfoOW4wl7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3b6b07c9-3e37-4bd7-ad0f-1073f120f1c7"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "here's the head\n",
            "2019-09-26 15:55:58\n",
            "2019-09-26 15:58:16\n",
            "2019-09-26 16:10:09\n",
            "2019-09-26 16:15:51\n",
            "2019-09-26 16:35:04\n",
            "2019-09-26 16:36:24\n",
            "2019-09-26 16:37:23\n",
            "2019-09-26 18:14:16\n",
            "2019-09-26 18:22:39\n",
            "2019-09-26 18:23:19\n",
            "here's the tail\n",
            "2021-08-06 10:15:23\n",
            "2021-08-06 10:15:48\n",
            "2021-08-06 10:16:02\n",
            "2021-08-06 10:16:30\n",
            "2021-08-06 10:17:27\n",
            "2021-08-06 10:17:59\n",
            "2021-08-06 10:19:49\n",
            "2021-08-06 10:19:50\n",
            "2021-08-06 10:20:13\n",
            "2021-08-06 10:20:36\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! zcat all_tweets_one_line.txt.gz | cut -f 3 | head"
      ],
      "metadata": {
        "id": "1Hiyfkaz5QMA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "757bc8db-2b03-4757-dfb0-c43cd8fc75fb"
      },
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2020-03-30 23:59:55\n",
            "2020-03-30 23:58:37\n",
            "2020-03-30 23:52:00\n",
            "2020-03-30 23:50:42\n",
            "2020-03-30 23:48:00\n",
            "2020-03-30 23:47:39\n",
            "2020-03-30 23:46:08\n",
            "2020-03-30 23:43:33\n",
            "2020-03-30 23:38:08\n",
            "2020-03-30 23:38:08\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! zcat all_tweets_one_line.txt.gz | cut -f 3 | cut  -f 1 -d ' ' | head # this gives us the dates"
      ],
      "metadata": {
        "id": "7FXSrMLV5-6q",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c2df81e4-2692-421a-88c3-7a5f17957e8b"
      },
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2020-03-30\n",
            "2020-03-30\n",
            "2020-03-30\n",
            "2020-03-30\n",
            "2020-03-30\n",
            "2020-03-30\n",
            "2020-03-30\n",
            "2020-03-30\n",
            "2020-03-30\n",
            "2020-03-30\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! zcat all_tweets_one_line.txt.gz | cut -f 3  | cut  -f 1 -d ' ' | cut -f 1,2  -d '-' | head # and here we get just the years and months!"
      ],
      "metadata": {
        "id": "N7RK4ocZ8n98",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5aee6e3a-4776-4d9c-ecdc-0545e4b88133"
      },
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2020-03\n",
            "2020-03\n",
            "2020-03\n",
            "2020-03\n",
            "2020-03\n",
            "2020-03\n",
            "2020-03\n",
            "2020-03\n",
            "2020-03\n",
            "2020-03\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! zcat all_tweets_one_line.txt.gz | cut -f 3  | cut  -f 1 -d ' ' | cut -f 1,2  -d '-' | sort | uniq -c | sort -rn # and by making a freq. list of these, we get the distribution!\n",
        "\n",
        "# so it seems that April 2020 has clearly the most tweets, and 2019 the least"
      ],
      "metadata": {
        "id": "EsKhbzDt82UV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fde82bfa-0285-4e33-d8a8-74995e342020"
      },
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " 109111 2020-04\n",
            "  94675 2020-03\n",
            "  52515 2021-03\n",
            "  42116 2020-05\n",
            "  41764 2020-08\n",
            "  41407 2020-10\n",
            "  40854 2021-02\n",
            "  31498 2020-12\n",
            "  31215 2020-06\n",
            "  30787 2020-09\n",
            "  27917 2021-06\n",
            "  26714 2021-07\n",
            "  26357 2021-05\n",
            "  24240 2020-07\n",
            "  23467 2021-01\n",
            "  19193 2021-04\n",
            "   8562 2021-08\n",
            "   8521 2020-11\n",
            "   2358 2020-02\n",
            "   2056 2019-10\n",
            "   2020 2020-01\n",
            "   1788 2019-11\n",
            "   1752 2019-12\n",
            "    241 2019-09\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! zcat all_tweets_one_line.txt.gz | head # so the tweets were in column 10\n",
        "! echo \"just checking\"\n",
        "! zcat all_tweets_one_line.txt.gz | cut -f 10 | head # yeah, this one!"
      ],
      "metadata": {
        "id": "WZ01SK5p9bRq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "07b8286f-9c19-4d2a-ae10-9ffe53073c08"
      },
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "###C:\t1244776401124851713\t2020-03-30 23:59:55\thexademus1\t1234607328009302019\t8\ttweet\t['valtioneuvosto', 'MarinSanna']\t['koronafi', 'koronasuomi', 'koronaepidemia']\t@valtioneuvosto @MarinSanna Yllättävän hyvin hallitus on hoitanut asian ja pysynyt Ruotsin linjoilla. Siellä on perinteisesti tehty yhteiskunnan kannalta hyviä päätöksiä ja maa on menestynyt. #koronafi #koronasuomi #koronaepidemia\n",
            "###C:\t1244776074380292104\t2020-03-30 23:58:37\tjukka235\t1225011411648237573\t153\ttweet\t[]\t['koronavirusfi', 'korona', 'covid19fi', 'covid19finland', 'covid2019']\tThe Lancetissa julkaistu tutkimus Kiinan aineistosta: potilaat olivat keskimäärin 18.5 päivää sairaana ennen kuolemaa.  7/x  #koronavirusfi  #korona  #covid19fi  #covid19finland  #covid2019  https://t.co/VKzxCtG8C0\n",
            "###C:\t1244774409937850369\t2020-03-30 23:52:00\tmarsusanoo\t1928168198\t1477\ttweet\t['miksuksi']\t[]\t@miksuksi kiitos! luulen, että itellä lähtee vasta nyt rullaamaan terapia kunnolla, kun ei ole ketään, kenestä kantaa huolta. lamaantuminen tulee varmasti hetkeksi, mutta se tulee kotona turvallisissa olosuhteissa ja se on jo ennakoitu.\n",
            "###C:\t1244774080538120192\t2020-03-30 23:50:42\tjukka235\t1225011411648237573\t153\ttweet\t[]\t['koronavirusfi', 'korona', 'covid19fi', 'covid19finland', 'covid2019']\tKuvan tulkinta:  -   ensin pitää tehdä oletus siitä, montako päivää kului tartunnan saamisesta kuolemaan -   esim. 30.3. yhteensä 13 kuollutta, 15 pv aikaisemmin (15.3.) yht 244 tartuntaa, 13/224=5.3% 5/x  #koronavirusfi  #korona  #covid19fi  #covid19finland  #covid2019\n",
            "###C:\t1244773402814095360\t2020-03-30 23:48:00\tjukka235\t1225011411648237573\t153\ttweet\t[]\t['koronavirusfi', 'korona', 'covid19fi', 'covid19finland', 'COVID2019']\tArvio kuolleiden määrästä (%). Avainkysymys: mitä tapahtui niille, jotka sairastuivat samaan aikaan? Esim. niistä, jotka olivat sairaana 15 päivää aikaisemmin, on kuollut noin 5.3%.  4/x  #koronavirusfi  #korona  #covid19fi  #covid19finland  #COVID2019 https://t.co/RWBXMHPNsJ\n",
            "###C:\t1244773316034015233\t2020-03-30 23:47:39\tMr_Riionsaari\t387549103\t2903\ttweet\t[]\t['koronavirus']\tDeittaileva Tuomas, 42, ei pelkää #koronavirus ´ta. \"Kyllä minä pussailisin! Jos toinenkin osapuoli haluaa, niin se on silloin molempien päätös.\"  📞 Haloo?Järjen💡o?  https://t.co/wlfNx4jDC8\n",
            "###C:\t1244772934356545536\t2020-03-30 23:46:08\tJantunenKaarina\t1886444366\t2753\tquote\t['kirkko_kaupunki']\t['Korona']\tTästä olen @kirkko_kaupunki suorastaan loukkaantunut. Julkaisee näin halveksivan,vähättelevän ja vain ihmisen valeturvaksi esittämänsä,uskonsa menettäneen papin,Kari Kuulan kirjoitelman Ilmestyskirjasta,joka on Jeesuksen antama näky lopunajoista,ikäänkuin loppua ei olisi. #Korona https://t.co/QTygwdNTlC\n",
            "###C:\t1244772282716880899\t2020-03-30 23:43:33\thexademus1\t1234607328009302019\t8\ttweet\t[]\t['vankila', 'koronafi', 'koronasuomi', 'karanteeni', 'Suomi']\thttps://t.co/zggMdSTbct  Ruotsissa ja Suomessa osataan vielä kunnioittaa kansalaisten perusoikeuksia, kuten liikkumisvapautta, päätösvapautta ja sananvapautta! Tämä on yllättävää. Etelässä ihmiset ovat kotonaan vankilassa. #vankila #koronafi #koronasuomi #karanteeni #Suomi\n",
            "###C:\t1244770919056396292\t2020-03-30 23:38:08\tkahvihukka\t883418120072855553\t76\ttweet\t[]\t[]\tNyt ois paljon tarroja bujoiluun mut mitäs sitä kalenterilla ku kaikki on peruttu ja kykkii vaa kotona. 🤣😭 https://t.co/NRwkOOLHxR\n",
            "###C:\t1244770917269602307\t2020-03-30 23:38:08\tLeDiff7\t866073828379721733\t36\ttweet\t['YouTube']\t['stayhome', 'withme']\tAmerican Valhalla #stayhome #withme https://t.co/YzxjLt8PGy via @YouTube !\n",
            "just checking\n",
            "@valtioneuvosto @MarinSanna Yllättävän hyvin hallitus on hoitanut asian ja pysynyt Ruotsin linjoilla. Siellä on perinteisesti tehty yhteiskunnan kannalta hyviä päätöksiä ja maa on menestynyt. #koronafi #koronasuomi #koronaepidemia\n",
            "The Lancetissa julkaistu tutkimus Kiinan aineistosta: potilaat olivat keskimäärin 18.5 päivää sairaana ennen kuolemaa.  7/x  #koronavirusfi  #korona  #covid19fi  #covid19finland  #covid2019  https://t.co/VKzxCtG8C0\n",
            "@miksuksi kiitos! luulen, että itellä lähtee vasta nyt rullaamaan terapia kunnolla, kun ei ole ketään, kenestä kantaa huolta. lamaantuminen tulee varmasti hetkeksi, mutta se tulee kotona turvallisissa olosuhteissa ja se on jo ennakoitu.\n",
            "Kuvan tulkinta:  -   ensin pitää tehdä oletus siitä, montako päivää kului tartunnan saamisesta kuolemaan -   esim. 30.3. yhteensä 13 kuollutta, 15 pv aikaisemmin (15.3.) yht 244 tartuntaa, 13/224=5.3% 5/x  #koronavirusfi  #korona  #covid19fi  #covid19finland  #covid2019\n",
            "Arvio kuolleiden määrästä (%). Avainkysymys: mitä tapahtui niille, jotka sairastuivat samaan aikaan? Esim. niistä, jotka olivat sairaana 15 päivää aikaisemmin, on kuollut noin 5.3%.  4/x  #koronavirusfi  #korona  #covid19fi  #covid19finland  #COVID2019 https://t.co/RWBXMHPNsJ\n",
            "Deittaileva Tuomas, 42, ei pelkää #koronavirus ´ta. \"Kyllä minä pussailisin! Jos toinenkin osapuoli haluaa, niin se on silloin molempien päätös.\"  📞 Haloo?Järjen💡o?  https://t.co/wlfNx4jDC8\n",
            "Tästä olen @kirkko_kaupunki suorastaan loukkaantunut. Julkaisee näin halveksivan,vähättelevän ja vain ihmisen valeturvaksi esittämänsä,uskonsa menettäneen papin,Kari Kuulan kirjoitelman Ilmestyskirjasta,joka on Jeesuksen antama näky lopunajoista,ikäänkuin loppua ei olisi. #Korona https://t.co/QTygwdNTlC\n",
            "https://t.co/zggMdSTbct  Ruotsissa ja Suomessa osataan vielä kunnioittaa kansalaisten perusoikeuksia, kuten liikkumisvapautta, päätösvapautta ja sananvapautta! Tämä on yllättävää. Etelässä ihmiset ovat kotonaan vankilassa. #vankila #koronafi #koronasuomi #karanteeni #Suomi\n",
            "Nyt ois paljon tarroja bujoiluun mut mitäs sitä kalenterilla ku kaikki on peruttu ja kykkii vaa kotona. 🤣😭 https://t.co/NRwkOOLHxR\n",
            "American Valhalla #stayhome #withme https://t.co/YzxjLt8PGy via @YouTube !\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! zcat all_tweets_one_line.txt.gz | cut -f 10 | tr '[:punct:]|[0-9]' ' ' | tr ' ' '\\n' | sort | uniq -c | sort -rn | head -20 # so here's the most frequent words. quite a big data, so takes some time!"
      ],
      "metadata": {
        "id": "JDi9bobe9le4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c5bc0e49-2a61-460c-9c3b-8539fbbc91f9"
      },
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "6726979 \n",
            " 379370 ja\n",
            " 328512 on\n",
            " 287487 t\n",
            " 283097 co\n",
            " 281512 https\n",
            " 166185 ei\n",
            " 133506 että\n",
            " 107952 korona\n",
            "  92856 koronafi\n",
            "  89684 kotona\n",
            "  76819 kun\n",
            "  65873 ole\n",
            "  60479 apua\n",
            "  60069 se\n",
            "  59191 nyt\n",
            "  59010 niin\n",
            "  52558 mutta\n",
            "  47700 koronavirus\n",
            "  46014 voi\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# taking all punctuation was maybe not the brightest idea, it takes away the hashtags as well...\n",
        "# so let's take just the hashtags and see those\n",
        "! zcat all_tweets_one_line.txt.gz | cut -f 10 | tr ' ' '\\n' |egrep \"^#\" | sort | uniq -c | sort -rn | head -20"
      ],
      "metadata": {
        "id": "zvk_Qb5oHZex",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "501012c7-1604-4d46-9bc0-3797357d290d"
      },
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  90544 #koronafi\n",
            "  81620 #korona\n",
            "  43560 #koronavirus\n",
            "  25179 #koronakriisi\n",
            "  17964 #koronasuomi\n",
            "  16815 #COVID19\n",
            "  15985 #Korona\n",
            "  10598 #covid19\n",
            "  10445 #hallitus\n",
            "   6042 #pysykotona\n",
            "   5835 #politiikka\n",
            "   5787 #COVID19FI\n",
            "   5590 #talous\n",
            "   3952 #etätyö\n",
            "   3864 #koronavirusfi\n",
            "   3773 #Koronavirusfi\n",
            "   3539 #karanteeni\n",
            "   3415 #covid19fi\n",
            "   3374 #yrittäjät\n",
            "   3265 #koronarokote\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# and usernames\n",
        "! zcat all_tweets_one_line.txt.gz | cut -f 10 | tr ' ' '\\n' |egrep \"^@\" | sort | uniq -c | sort -rn | head -20"
      ],
      "metadata": {
        "id": "IV-3BWbkIKGr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "25b05b9c-8aec-4dc6-c6e7-2f6fb9c96a33"
      },
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  11104 @THLorg\n",
            "   9941 @MarinSanna\n",
            "   4837 @yleuutiset\n",
            "   4728 @hsfi\n",
            "   4577 @STM_Uutiset\n",
            "   3659 @valtioneuvosto\n",
            "   3419 @KristaKiuru\n",
            "   3362 @liandersson\n",
            "   3212 @lasleh\n",
            "   3152 @iltasanomat\n",
            "   2589 @MariaOhisalo\n",
            "   2284 @iltalehti_fi\n",
            "   2187 @mika_salminen\n",
            "   1733 @Vastuullisuus\n",
            "   1728 @pekkasauri\n",
            "   1711 @HenriAlen\n",
            "   1708 @Vapaavuori\n",
            "   1577 @MattiMuukkonen\n",
            "   1526 @MTVUutiset\n",
            "   1434 @AVItwiittaa\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "--> So these are clearly (at least to some extent) related to Covid.\n",
        "--> The most frequent username is THLorg, followed by news agencies and politicians"
      ],
      "metadata": {
        "id": "SzFhFFyaIn7k"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! zcat all_tweets_one_line.txt.gz | cut -f 3,10 | egrep \"^2019\" | cut -f 2 > 2019_tweets.txt\n",
        "\n",
        "! zcat all_tweets_one_line.txt.gz | cut -f 3,10 | egrep \"^2020-08\" | cut -f 2 > 2020_tweets.txt"
      ],
      "metadata": {
        "id": "daO_LmCQIamU"
      },
      "execution_count": 52,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "! head *tweets.txt # looks about right!"
      ],
      "metadata": {
        "id": "GW4O9NcEJTZj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "49cff945-c34a-4840-886f-c4b11696cddc"
      },
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "==> 2019_tweets.txt <==\n",
            "Mulla on pää kipeä. Perkelele.\n",
            "@jatsi Väsynyt mutta onnellinen 😊😁\n",
            "Vähitellen pauke vaimenee, jokohan pian loppuu? Se olis huomenna, tai siis tänään, vuoden ensimmäinen työpäivä... – väsynyt\n",
            "Viimeistä vuotta miettiessä tulee mieleen lähinnä vaikeita asioita, kuten uupumus, masennus ja lomautus. Oon ollu melkeen koko vuoden toipumassa kotona, ja tää kuntouttavakin toiminta tuntuu vaan huonontaneen oloo... Takapakit kuuluu kuitenkin toipumiseen.\n",
            "Tää on oikeasti eka kerta ikinä ainakin mun muistiin, kun oon kipeä ja en pysty viettämään uutta vuotta niin kuin haluaisin~ 😷😩💦 Harmittaa kyl, mutta aina on ensi vuosi!! 💕💕 Uuden Vuoden lupaus on vaikka extra-positiivisuus, kun elämä pettää. 😁✨✨ https://t.co/bGl1aUUNJt\n",
            "@jofistics @Ellahhelena @vanamolli Massiivinen keuhkoveritulppa ja keuhkokuume ilman pysyviä vaurioita tai muutoksia. I like my chances. 😎\n",
            "@Naitsooo @maenalanen Oli kuulemma ollut aamujäillä, mutta ei pelaa kuitenkaan. Joku flunssa yms ollut.\n",
            "@Ruiskukkahattu Mie voin kertoa vaan itsestäni jos minä juon vettä ( kuumana päivänä) ja en syö suolaa, tulee pää kipeäksi ja jos en juo kahvia yhteen päivään tuleee pää kipeäksi, toivottavasti pääkipu paranee\n",
            "Vannon että en oo nauttinut kuin marjakompottia, tuorejuustoa, kaurapellavaleipää ja Tehoa nyt toviin. En siis oo päissäni vaikka ulosanti siltä vaikuttaakin. Oon vaan hiton väsynyt nukuttuani viimeset kaks viikkoa 4 - 5 tuntia yössä.\n",
            "@Makia_Velli Joku flunssa vaa, tuskin mitään sen pahempaa! Joo, syön ja köllöttelen sohvalla, lomaa viel muutama päivä!\n",
            "\n",
            "==> 2020_tweets.txt <==\n",
            "@AnttiPKoivisto @MarinSanna @KristaKiuru @MariaOhisalo Ja ehdoton 14 vrk karanteeni kaikille poikkeuksetta. Oireetonkin voi tartuttaa.\n",
            "Se olis viikonloppu nyt ja kylmät kaljat odottaa kotona\n",
            "Islanti ei täytä enää koronan matkustuskriteeriä Siellä ilmaantuvuus on 9,9 tapausta / 100 000 asukasta / 14 päivää (Saisi olla max 8 tai max 10) (Suomen oma tilanne 2,4) Eri mittari kuvassa alla @KirsiVarhila @mika_salminen #koronafi https://t.co/zxzfTpIvAI https://t.co/lYPEOz2WnS\n",
            "...juttu -oikea \"monikulttuurisuus\" on vaikeaa, ja ihmisiä ahdistaa (puolin ja toisin), että jaettua tonttia miehitetään. Missä kohtaa ollaan siirrytty esim. ideologiseen/aatteellisen tuputtamiseen(?) Missä taas menee uskonnon&amp;kulttuurin raja(?) Vaan, se mikä on normi/ideaali...\n",
            "@VirtaWeeruska @ApeKanerva @jakivisto Parempi olis vaan olla hiljaa, kuin tollasta diibadaabaa kirjoitella. Mä ainakin tulin vaan huonolle tuulelle. Paras oli toi tuntemattomia... 🤦🏼‍♂️ Ja lopun auringot vielä. ☀️☀️☀️☀️☀️☀️☀️☀️☀️ Vittu täällä sataa vettä. Kukkuu. Onko ketään kotona? 🥱 Ei.\n",
            "@JAP3x2 @VirtaWeeruska @jakivisto Hyvä Jari! Just näin. Ei oo kaikki kotona. :D\n",
            "Malta ei täytä enää koronan matkustuskriteeriä Siellä ilmaantuvuus on 27 tapausta / 100 000 asukasta / 14 päivää (Saisi olla max 8 tai max 10) (Suomen oma tilanne 2,4) Eri mittari kuvassa alla @KirsiVarhila @mika_salminen #koronafi https://t.co/9nJxWO93sg https://t.co/IB3bA8IwbR\n",
            "Olen väsynyt https://t.co/Ujj4DUibE5\n",
            "Hollanti ei täytä enää koronan matkustuskriteeriä Siellä ilmaantuvuus on 15 tapausta / 100 000 asukasta / 14 päivää (Saisi olla max 8 tai max 10) (Suomen oma tilanne 2,4) Eri mittari kuvassa alla @KirsiVarhila @mika_salminen #koronafi https://t.co/M3FOUDj3SO https://t.co/F2qyLyLwFq\n",
            "@marijkuusisto Joskus tainnut hyvää huomenta toivottaa kotona kylpyhuoneen peilin edessä hiljaisilla huokauksilla höystettynä.\n"
          ]
        }
      ]
    }
  ]
}