show-attend-and-tell-caio-pinho-pt-br.ipynb

{"cells":[{"cell_type":"code","execution_count":1,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:03.083795Z","iopub.status.busy":"2021-11-18T10:15:03.083471Z","iopub.status.idle":"2021-11-18T10:15:03.805172Z","shell.execute_reply":"2021-11-18T10:15:03.804395Z","shell.execute_reply.started":"2021-11-18T10:15:03.083705Z"},"trusted":true},"outputs":[],"source":["! nvidia-smi"]},{"cell_type":"markdown","metadata":{},"source":["# Imports\n","The following packages are imported:\n","- tensorflow\n","- matplotlib\n","- numpy\n","- IPython\n"]},{"cell_type":"code","execution_count":2,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:03.808821Z","iopub.status.busy":"2021-11-18T10:15:03.808604Z","iopub.status.idle":"2021-11-18T10:15:08.582695Z","shell.execute_reply":"2021-11-18T10:15:08.581736Z","shell.execute_reply.started":"2021-11-18T10:15:03.808792Z"},"trusted":true},"outputs":[],"source":["import tensorflow as tf\n","from tensorflow.keras.layers import *\n","\n","import matplotlib.pyplot as plt\n","import pandas as pd\n","import numpy as np\n","\n","import IPython"]},{"cell_type":"markdown","metadata":{},"source":["# Data\n","- data_dir is the path to the images and the `results.csv`\n","- image_dir is the path exculsively to the images\n","- csv_file is the path to the `results.csv` file"]},{"cell_type":"code","execution_count":3,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:08.585456Z","iopub.status.busy":"2021-11-18T10:15:08.585046Z","iopub.status.idle":"2021-11-18T10:15:08.591686Z","shell.execute_reply":"2021-11-18T10:15:08.590979Z","shell.execute_reply.started":"2021-11-18T10:15:08.585418Z"},"trusted":true},"outputs":[],"source":["data_dir = '../input/flickr-image-dataset/flickr30k_images'\n","image_dir = f'{data_dir}/flickr30k_images'\n","csv_file = '../input/translatedcsv/translatedCSV.csv'"]},{"cell_type":"markdown","metadata":{},"source":["Here we read the csv file as a dataframe and make some observations from it.\n","For a quick EDA we are going to \n","- check the shape of the dataframe\n","- check the names of the columns\n","- find out the unique image names there are"]},{"cell_type":"code","execution_count":4,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:08.597445Z","iopub.status.busy":"2021-11-18T10:15:08.597137Z","iopub.status.idle":"2021-11-18T10:15:09.077776Z","shell.execute_reply":"2021-11-18T10:15:09.076127Z","shell.execute_reply.started":"2021-11-18T10:15:08.597408Z"},"trusted":true},"outputs":[],"source":["df = pd.read_csv(csv_file, delimiter='|')\n","\n","print(f'[INFO] The shape of dataframe: {df.shape}')\n","print(f'[INFO] The columns in the dataframe: {df.columns}')\n","print(f'[INFO] Unique image names: {len(pd.unique(df[\"image_name\"]))}')"]},{"cell_type":"markdown","metadata":{},"source":["A quick observation here is to see that the dataframe has `158915` elements but only `31783` image names. This means that there is a duplicacy involved. On further inspection we will see that each image has 5 unique captions attached to it ($31783\\times 5=158915$)\n","\n","While looking into the dataframe I found out that `19999` had some messed up entries. This has led me to manually change the entries in that row."]},{"cell_type":"code","execution_count":5,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:09.079337Z","iopub.status.busy":"2021-11-18T10:15:09.079084Z","iopub.status.idle":"2021-11-18T10:15:09.177908Z","shell.execute_reply":"2021-11-18T10:15:09.177138Z","shell.execute_reply.started":"2021-11-18T10:15:09.079310Z"},"trusted":true},"outputs":[],"source":["df.columns = ['image_name', 'comment_number', 'comment']\n","del df['comment_number']\n","\n","# Under scrutiny I had found that 19999 had a messed up entry\n","#df['comment'][19999] = ' A dog runs across the grass .'\n","\n","# Image names now correspond to the absolute position\n","df['image_name'] = image_dir+'/'+df['image_name']\n","\n","# <start> comment <end>\n","df['comment'] = \"<start> \"+df['comment']+\" <end>\""]},{"cell_type":"code","execution_count":6,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:09.179624Z","iopub.status.busy":"2021-11-18T10:15:09.179373Z","iopub.status.idle":"2021-11-18T10:15:09.228006Z","shell.execute_reply":"2021-11-18T10:15:09.227288Z","shell.execute_reply.started":"2021-11-18T10:15:09.179590Z"},"trusted":true},"outputs":[],"source":["# Shuffle the dataframe\n","df = df.sample(frac=1).reset_index(drop=True)\n","df.head()"]},{"cell_type":"code","execution_count":7,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:09.229709Z","iopub.status.busy":"2021-11-18T10:15:09.229292Z","iopub.status.idle":"2021-11-18T10:15:09.234110Z","shell.execute_reply":"2021-11-18T10:15:09.233264Z","shell.execute_reply.started":"2021-11-18T10:15:09.229663Z"},"trusted":true},"outputs":[],"source":["# SIZE = len(df)\n","\n","# train_size = int(0.7* SIZE) \n","# val_size = int(0.1* SIZE)\n","# test_size = int(0.2* SIZE)\n","\n","# train_size, val_size, test_size\n","\n","train_size = 60_000 \n","val_size = 10_000\n","test_size = 20_000"]},{"cell_type":"markdown","metadata":{},"source":["Splitting the dataframe accordingly"]},{"cell_type":"code","execution_count":8,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:09.235906Z","iopub.status.busy":"2021-11-18T10:15:09.235624Z","iopub.status.idle":"2021-11-18T10:15:09.244911Z","shell.execute_reply":"2021-11-18T10:15:09.244001Z","shell.execute_reply.started":"2021-11-18T10:15:09.235870Z"},"trusted":true},"outputs":[],"source":["train_df = df.iloc[:train_size,:]\n","val_df = df.iloc[train_size:train_size+val_size,:]\n","test_df = df.iloc[train_size+val_size:train_size+val_size+test_size,:]\n","\n","train_df.shape, val_df.shape, test_df.shape"]},{"cell_type":"code","execution_count":9,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:09.246593Z","iopub.status.busy":"2021-11-18T10:15:09.246266Z","iopub.status.idle":"2021-11-18T10:15:09.277891Z","shell.execute_reply":"2021-11-18T10:15:09.277262Z","shell.execute_reply.started":"2021-11-18T10:15:09.246558Z"},"trusted":true},"outputs":[],"source":["# Enter different indices.\n","index = 200\n","\n","image_name = train_df['image_name'][index]\n","comment = train_df['comment'][index]\n","\n","print(comment)\n","\n","IPython.display.Image(filename=image_name)"]},{"cell_type":"markdown","metadata":{},"source":["# Text Handling\n","- Defined the size of the vocab which is `5000`.\n","- Initialized the Tokenizer class.\n","    - Standardized (all to lower case)\n","    - Filters the punctuations\n","    - Splits the text\n","    - Creates the vocabulary (`<start>, <end> and <unk>` is defined)"]},{"cell_type":"code","execution_count":10,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:09.281345Z","iopub.status.busy":"2021-11-18T10:15:09.280814Z","iopub.status.idle":"2021-11-18T10:15:09.286387Z","shell.execute_reply":"2021-11-18T10:15:09.285550Z","shell.execute_reply.started":"2021-11-18T10:15:09.281310Z"},"trusted":true},"outputs":[],"source":["# Choose the top 10000 words from the vocabulary\n","top_k = 10000\n","tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=top_k,\n","                                                  oov_token=\"<unk>\",\n","                                                  filters='!\"#$%&()*+.,-/:;=?@[\\]^_`{|}~')"]},{"cell_type":"markdown","metadata":{},"source":["Here we fit the `tokenizer` object on the captions. This helps in the updation of the vocab that the `tokenizer` object might have.\n","\n","In the first iteration the vocabulary does not start from `0`. Both the dictionaries have 1 as the key or value."]},{"cell_type":"code","execution_count":11,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:09.288431Z","iopub.status.busy":"2021-11-18T10:15:09.287832Z","iopub.status.idle":"2021-11-18T10:15:10.829031Z","shell.execute_reply":"2021-11-18T10:15:10.828293Z","shell.execute_reply.started":"2021-11-18T10:15:09.288394Z"},"trusted":true},"outputs":[],"source":["# build the vocabulary\n","tokenizer.fit_on_texts(train_df['comment'])"]},{"cell_type":"code","execution_count":12,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:10.830461Z","iopub.status.busy":"2021-11-18T10:15:10.830200Z","iopub.status.idle":"2021-11-18T10:15:10.838831Z","shell.execute_reply":"2021-11-18T10:15:10.837868Z","shell.execute_reply.started":"2021-11-18T10:15:10.830427Z"},"trusted":true},"outputs":[],"source":["# This is a sanity check function\n","def check_vocab(word):\n","    i = tokenizer.word_index[word]\n","    print(f\"The index of the word: {i}\")\n","    print(f\"Index {i} is word {tokenizer.index_word[i]}\")\n","    \n","check_vocab(\"pijama\")"]},{"cell_type":"markdown","metadata":{},"source":["Here we are padding the sentences so that each of the sentences are of the same length."]},{"cell_type":"code","execution_count":13,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:10.840908Z","iopub.status.busy":"2021-11-18T10:15:10.840588Z","iopub.status.idle":"2021-11-18T10:15:10.846855Z","shell.execute_reply":"2021-11-18T10:15:10.846183Z","shell.execute_reply.started":"2021-11-18T10:15:10.840873Z"},"trusted":true},"outputs":[],"source":["tokenizer.word_index['<pad>'] = 0\n","tokenizer.index_word[0] = '<pad>'"]},{"cell_type":"code","execution_count":14,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:10.848569Z","iopub.status.busy":"2021-11-18T10:15:10.848283Z","iopub.status.idle":"2021-11-18T10:15:12.624004Z","shell.execute_reply":"2021-11-18T10:15:12.623232Z","shell.execute_reply.started":"2021-11-18T10:15:10.848505Z"},"trusted":true},"outputs":[],"source":["# Create the tokenized vectors\n","train_seqs = tokenizer.texts_to_sequences(train_df['comment'])\n","val_seqs = tokenizer.texts_to_sequences(val_df['comment'])\n","test_seqs = tokenizer.texts_to_sequences(test_df['comment'])"]},{"cell_type":"code","execution_count":15,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:12.625344Z","iopub.status.busy":"2021-11-18T10:15:12.625104Z","iopub.status.idle":"2021-11-18T10:15:13.384864Z","shell.execute_reply":"2021-11-18T10:15:13.384091Z","shell.execute_reply.started":"2021-11-18T10:15:12.625311Z"},"trusted":true},"outputs":[],"source":["# Pad each vector to the max_length of the captions\n","# If you do not provide a max_length value, pad_sequences calculates it automatically\n","train_cap_vector = tf.keras.preprocessing.sequence.pad_sequences(train_seqs, padding='post')\n","val_cap_vector = tf.keras.preprocessing.sequence.pad_sequences(val_seqs, padding='post')\n","test_cap_vector = tf.keras.preprocessing.sequence.pad_sequences(test_seqs, padding='post')"]},{"cell_type":"code","execution_count":16,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:13.386475Z","iopub.status.busy":"2021-11-18T10:15:13.386238Z","iopub.status.idle":"2021-11-18T10:15:13.391986Z","shell.execute_reply":"2021-11-18T10:15:13.391318Z","shell.execute_reply.started":"2021-11-18T10:15:13.386443Z"},"trusted":true},"outputs":[],"source":["# Caption vector\n","train_cap_vector.shape, val_cap_vector.shape, test_cap_vector.shape"]},{"cell_type":"code","execution_count":17,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:13.394056Z","iopub.status.busy":"2021-11-18T10:15:13.393505Z","iopub.status.idle":"2021-11-18T10:15:15.778024Z","shell.execute_reply":"2021-11-18T10:15:15.776011Z","shell.execute_reply.started":"2021-11-18T10:15:13.394021Z"},"trusted":true},"outputs":[],"source":["train_cap_ds = tf.data.Dataset.from_tensor_slices(train_cap_vector)\n","val_cap_ds = tf.data.Dataset.from_tensor_slices(val_cap_vector)\n","test_cap_ds = tf.data.Dataset.from_tensor_slices(test_cap_vector)"]},{"cell_type":"markdown","metadata":{},"source":["# Image Handling\n","- Load the image\n","- decode jpeg\n","- resize\n","- standardize"]},{"cell_type":"code","execution_count":18,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:15.779713Z","iopub.status.busy":"2021-11-18T10:15:15.779297Z","iopub.status.idle":"2021-11-18T10:15:15.785824Z","shell.execute_reply":"2021-11-18T10:15:15.784972Z","shell.execute_reply.started":"2021-11-18T10:15:15.779670Z"},"trusted":true},"outputs":[],"source":["@tf.function\n","def load_img(image_path):\n","    img = tf.io.read_file(image_path)\n","    img = tf.image.decode_jpeg(img, channels=3)\n","    img = tf.image.resize(img, (299, 299))\n","    img = tf.keras.applications.inception_v3.preprocess_input(img)\n","    return img"]},{"cell_type":"code","execution_count":19,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:15.787740Z","iopub.status.busy":"2021-11-18T10:15:15.787444Z","iopub.status.idle":"2021-11-18T10:15:15.796364Z","shell.execute_reply":"2021-11-18T10:15:15.795526Z","shell.execute_reply.started":"2021-11-18T10:15:15.787706Z"},"trusted":true},"outputs":[],"source":["train_img_name = train_df['image_name'].values\n","val_img_name = val_df['image_name'].values\n","test_img_name = test_df['image_name'].values"]},{"cell_type":"code","execution_count":20,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:15.798243Z","iopub.status.busy":"2021-11-18T10:15:15.797924Z","iopub.status.idle":"2021-11-18T10:15:15.927844Z","shell.execute_reply":"2021-11-18T10:15:15.927108Z","shell.execute_reply.started":"2021-11-18T10:15:15.798182Z"},"trusted":true},"outputs":[],"source":["train_img_ds = tf.data.Dataset.from_tensor_slices(train_img_name).map(load_img)\n","val_img_ds = tf.data.Dataset.from_tensor_slices(val_img_name).map(load_img)\n","test_img_ds = tf.data.Dataset.from_tensor_slices(test_img_name).map(load_img)"]},{"cell_type":"markdown","metadata":{},"source":["# Joint data"]},{"cell_type":"code","execution_count":21,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:15.929336Z","iopub.status.busy":"2021-11-18T10:15:15.929089Z","iopub.status.idle":"2021-11-18T10:15:15.944232Z","shell.execute_reply":"2021-11-18T10:15:15.943510Z","shell.execute_reply.started":"2021-11-18T10:15:15.929302Z"},"trusted":true},"outputs":[],"source":["# prefecth and batch the dataset\n","AUTOTUNE = tf.data.experimental.AUTOTUNE\n","BATCH_SIZE = 512\n","\n","train_ds = tf.data.Dataset.zip((train_img_ds, train_cap_ds)).batch(BATCH_SIZE,drop_remainder=True).prefetch(buffer_size=AUTOTUNE)\n","val_ds = tf.data.Dataset.zip((val_img_ds, val_cap_ds)).batch(BATCH_SIZE,drop_remainder=True).prefetch(buffer_size=AUTOTUNE)\n","test_ds = tf.data.Dataset.zip((test_img_ds, test_cap_ds)).batch(BATCH_SIZE,drop_remainder=True).prefetch(buffer_size=AUTOTUNE)"]},{"cell_type":"markdown","metadata":{},"source":["Sanity check for the division of datasets"]},{"cell_type":"markdown","metadata":{},"source":["## Model"]},{"cell_type":"code","execution_count":22,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:15.946767Z","iopub.status.busy":"2021-11-18T10:15:15.945932Z","iopub.status.idle":"2021-11-18T10:15:15.951158Z","shell.execute_reply":"2021-11-18T10:15:15.950177Z","shell.execute_reply.started":"2021-11-18T10:15:15.946728Z"},"trusted":true},"outputs":[],"source":["# Some global variables\n","EMBEDDING_DIM = 512\n","VOCAB_SIZE = top_k+1\n","UNITS = 256\n","KERNEL = 64\n","FEATURES = 2048"]},{"cell_type":"code","execution_count":23,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:15.952979Z","iopub.status.busy":"2021-11-18T10:15:15.952520Z","iopub.status.idle":"2021-11-18T10:15:15.963247Z","shell.execute_reply":"2021-11-18T10:15:15.961166Z","shell.execute_reply.started":"2021-11-18T10:15:15.952933Z"},"trusted":true},"outputs":[],"source":["class CNN_Encoder(tf.keras.Model):\n","    \n","    def __init__(self, embedding_dim, batch_size):\n","        super(CNN_Encoder, self).__init__()\n","        self.batch_size = batch_size\n","        self.embedding_dim = embedding_dim\n","        \n","    def build(self, input_shape):\n","        self.image_model = tf.keras.applications.InceptionV3(include_top=False,\n","                                                weights='imagenet')\n","        self.new_input = self.image_model.input\n","        self.hidden_layer = self.image_model.layers[-1].output\n","        self.image_features_extract_model = tf.keras.Model(self.new_input, self.hidden_layer)\n","        self.image_features_extract_model.trainable = False\n","        \n","        self.reshape = tf.keras.layers.Reshape(target_shape=(KERNEL,FEATURES))\n","        self.fc = Dense(units=self.embedding_dim,\n","                        activation='relu')\n","        \n","    def call(self, x):\n","        x = self.image_features_extract_model(x)\n","        x = self.reshape(x)\n","        x = self.fc(x)\n","        return x"]},{"cell_type":"code","execution_count":24,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:15.965299Z","iopub.status.busy":"2021-11-18T10:15:15.964574Z","iopub.status.idle":"2021-11-18T10:15:32.312832Z","shell.execute_reply":"2021-11-18T10:15:32.311195Z","shell.execute_reply.started":"2021-11-18T10:15:15.965263Z"},"trusted":true},"outputs":[],"source":["# Test the encoder\n","encoder = CNN_Encoder(EMBEDDING_DIM, BATCH_SIZE)\n","for image, caption in train_ds.take(1):\n","    features = encoder(image)\n","    print(f\"ENCODER OUTPUT: {features.shape}\")"]},{"cell_type":"code","execution_count":25,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:32.314639Z","iopub.status.busy":"2021-11-18T10:15:32.314356Z","iopub.status.idle":"2021-11-18T10:15:32.323249Z","shell.execute_reply":"2021-11-18T10:15:32.322426Z","shell.execute_reply.started":"2021-11-18T10:15:32.314602Z"},"trusted":true},"outputs":[],"source":["class BahdanauAttention(tf.keras.Model):\n","    def __init__(self, units):\n","        super(BahdanauAttention, self).__init__()\n","        self.W1 = tf.keras.layers.Dense(units)\n","        self.W2 = tf.keras.layers.Dense(units)\n","        self.V = tf.keras.layers.Dense(1)\n","\n","    def call(self, annotations, hidden):\n","        #                                           64          64  256\n","        # annotations(CNN_encoder output) shape == (batch_size, 64, embedding_dim)\n","        #                  64          512\n","        # hidden shape == (batch_size, units)\n","        #                                 64          1  512\n","        # hidden_with_time_axis shape == (batch_size, 1, units)\n","        hidden_with_time_axis = tf.expand_dims(hidden, 1)\n","        #                                  64          64  512\n","        # attention_hidden_layer shape == (batch_size, 64, units)\n","        attention_hidden_layer = (tf.nn.tanh(self.W1(annotations) +\n","                                             self.W2(hidden_with_time_axis)))\n","        #                 64          64  1\n","        # score shape == (batch_size, 64, 1)\n","        # This gives you an unnormalized score for each image feature.\n","        score = self.V(attention_hidden_layer)\n","        #                             64          64  1\n","        # attention_weights shape == (batch_size, 64, 1)\n","        attention_weights = tf.nn.softmax(score, axis=1)\n","        #                          64          64  256\n","        # context_vector shape == (batch_size, 64, embedding_dim)\n","        #                                    64          256\n","        # context_vector shape after sum == (batch_size, embedding_dim)\n","        context_vector = attention_weights * annotations\n","        context_vector = tf.reduce_sum(context_vector, axis=1) # thinking: average?\n","\n","        return context_vector, attention_weights"]},{"cell_type":"code","execution_count":26,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:32.324977Z","iopub.status.busy":"2021-11-18T10:15:32.324715Z","iopub.status.idle":"2021-11-18T10:15:32.336775Z","shell.execute_reply":"2021-11-18T10:15:32.336021Z","shell.execute_reply.started":"2021-11-18T10:15:32.324932Z"},"trusted":true},"outputs":[],"source":["class RNN_Decoder(tf.keras.Model):\n","    def __init__(self, embedding_dim, units, vocab_size, batch_size):\n","        super(RNN_Decoder, self).__init__()\n","        self.batch_size = batch_size\n","        self.units = units\n","\n","        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n","        self.gru = tf.keras.layers.GRU(self.units,\n","                                       recurrent_initializer='glorot_uniform')\n","        # self.fc1 = tf.keras.layers.Dense(self.units)\n","        self.fc2 = tf.keras.layers.Dense(vocab_size)\n","\n","        self.attention = BahdanauAttention(self.units)\n","\n","    def call(self, x, annotations, hidden):\n","        # defining attention as a separate model\n","        #                          64     256\n","        # context_vector shape == (batch, embedding_shape)\n","        #                             64     64  1\n","        # attention_weights shape == (batch, 64, 1)\n","        context_vector, attention_weights = self.attention(annotations, hidden)\n","        #                                            64           1  256\n","        # x shape after passing through embedding == (batch_size, 1, embedding_dim)\n","        x = self.embedding(x)\n","        #                                 64          1  256+256\n","        # x shape after concatenation == (batch_size, 1, embedding_dim + embedding_shape)\n","        x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)\n","\n","        # passing the concatenated vector to the GRU\n","        output = self.gru(x)\n","        state = output\n","        #           64          1  512\n","        # shape == (batch_size, 1, units)\n","        x = self.fc2(output)\n","        # #            64               512\n","        # # x shape == (batch_size * 1, units)\n","        # x = tf.reshape(x, (-1, x.shape[2]))\n","        # #                  64              vocab\n","        # # output shape == (batch_size * 1, vocab)\n","        # x = self.fc2(x)\n","\n","        return x, output, attention_weights\n","\n","    def reset_state(self):\n","        return tf.zeros((self.batch_size, self.units))"]},{"cell_type":"code","execution_count":27,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:32.338493Z","iopub.status.busy":"2021-11-18T10:15:32.338028Z","iopub.status.idle":"2021-11-18T10:15:39.003123Z","shell.execute_reply":"2021-11-18T10:15:39.002367Z","shell.execute_reply.started":"2021-11-18T10:15:32.338430Z"},"trusted":true},"outputs":[],"source":["# Test the decoder\n","encoder = CNN_Encoder(EMBEDDING_DIM, BATCH_SIZE)\n","decoder = RNN_Decoder(EMBEDDING_DIM, UNITS, VOCAB_SIZE, BATCH_SIZE)\n","\n","for image, caption in train_ds.take(1):\n","    features = encoder(image)\n","    print(f\"ENCODER OUTPUT: {features.shape}\")\n","    hidden = decoder.reset_state()\n","    dec_input = tf.expand_dims([tokenizer.word_index['<start>']] * caption.shape[0], 1)\n","    predictions, hidden, attn_weights = decoder(dec_input, features, hidden)\n","    print(f\"PREDICTION: {predictions.shape}\")\n","    print(f\"HIDDEN: {hidden.shape}\")\n","    print(f\"ATTENTION: {attn_weights.shape}\")"]},{"cell_type":"markdown","metadata":{},"source":["# Wrapping the Gradient Tape in Model Class"]},{"cell_type":"code","execution_count":28,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:39.007575Z","iopub.status.busy":"2021-11-18T10:15:39.007371Z","iopub.status.idle":"2021-11-18T10:15:40.881501Z","shell.execute_reply":"2021-11-18T10:15:40.880673Z","shell.execute_reply.started":"2021-11-18T10:15:39.007549Z"},"trusted":true},"outputs":[],"source":["for image, caption in train_ds.take(1):\n","    print(image.shape)\n","    print(caption.shape)"]},{"cell_type":"code","execution_count":29,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:15:40.883594Z","iopub.status.busy":"2021-11-18T10:15:40.882885Z","iopub.status.idle":"2021-11-18T10:15:40.897089Z","shell.execute_reply":"2021-11-18T10:15:40.896045Z","shell.execute_reply.started":"2021-11-18T10:15:40.883555Z"},"trusted":true},"outputs":[],"source":["class Image_Caption_Gen(tf.keras.Model):\n","    def __init__(self, encoder, decoder):\n","        super(Image_Caption_Gen, self).__init__()\n","        self.encoder = encoder\n","        self.decoder = decoder\n","\n","    def train_step(self, data):\n","        img_tensor, target = data\n","        \n","        loss = 0\n","        \n","        # initializing the hidden state for each batch\n","        # because the captions are not related from image to image\n","        hidden = self.decoder.reset_state()\n","        \n","        dec_input = tf.expand_dims([tokenizer.word_index['<start>']] * BATCH_SIZE, 1)\n","        \n","        with tf.GradientTape() as tape:\n","            features = self.encoder(img_tensor)\n","            \n","            for i in range(1, target.shape[1]):\n","                # passing the features through the decoder\n","                predictions, hidden, _ = self.decoder(dec_input, features, hidden)\n","                \n","                loss += loss_function(target[:, i], predictions)\n","                \n","                # using teacher forcing\n","                dec_input = tf.expand_dims(target[:, i], 1)\n","                \n","        total_loss = (loss / int(target.shape[1]))\n","        trainable_variables = self.encoder.trainable_variables + self.decoder.trainable_variables\n","        gradients = tape.gradient(loss, trainable_variables)\n","        optimizer.apply_gradients(zip(gradients, trainable_variables))\n","        return {\"custom_loss\": total_loss}\n","    \n","    def test_step(self, data):\n","        img_tensor, target = data\n","        \n","        loss = 0\n","        \n","        # initializing the hidden state for each batch\n","        # because the captions are not related from image to image\n","        hidden = self.decoder.reset_state()\n","        \n","        dec_input = tf.expand_dims([tokenizer.word_index['<start>']] * BATCH_SIZE, 1)\n","        \n","        features = self.encoder(img_tensor)\n","            \n","        for i in range(1, target.shape[1]):\n","            # passing the features through the decoder\n","            predictions, hidden, _ = self.decoder(dec_input, features, hidden)\n","\n","            loss += loss_function(target[:, i], predictions)\n","\n","            # using teacher forcing\n","            dec_input = tf.expand_dims(target[:, i], 1)\n","                \n","        total_loss = (loss / int(target.shape[1]))\n","        return {\"custom_loss\": total_loss}"]},{"cell_type":"markdown","metadata":{},"source":["We use `Adam` as the optimizer.\n","\n","The loss is `SparseCategoricalCrossentropy`, because here it would be inefficient to use one-hot-encoders are the ground truth. We will also use mask to help mask the `<pad>` so that we do not let the sequence model learn to overfit on the same."]},{"cell_type":"code","execution_count":33,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:18:41.040078Z","iopub.status.busy":"2021-11-18T10:18:41.039512Z","iopub.status.idle":"2021-11-18T10:18:41.044810Z","shell.execute_reply":"2021-11-18T10:18:41.043812Z","shell.execute_reply.started":"2021-11-18T10:18:41.040037Z"},"trusted":true},"outputs":[],"source":["# Early Stopping to prevent overfitting\n","es = tf.keras.callbacks.EarlyStopping(monitor=\"val_custom_loss\", patience=5, verbose=2, restore_best_weights=True)"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2021-11-18T10:18:43.727155Z","iopub.status.busy":"2021-11-18T10:18:43.726562Z"},"trusted":true},"outputs":[],"source":["EPOCHS=100\n","# Test the decoder\n","encoder = CNN_Encoder(EMBEDDING_DIM, BATCH_SIZE)\n","decoder = RNN_Decoder(EMBEDDING_DIM, UNITS, VOCAB_SIZE, BATCH_SIZE)\n","\n","optimizer = tf.keras.optimizers.Adam()\n","loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n","    from_logits=True, reduction='none')\n","\n","def loss_function(real, pred):\n","    mask = tf.math.logical_not(tf.math.equal(real, 0))\n","    loss_ = loss_object(real, pred)\n","\n","    mask = tf.cast(mask, dtype=loss_.dtype)\n","    loss_ *= mask\n","\n","    return tf.reduce_mean(loss_)\n","\n","main_model = Image_Caption_Gen(encoder, decoder)\n","main_model.compile(loss=loss_function, optimizer=optimizer)\n","\n","history = main_model.fit(\n","    train_ds,\n","    validation_data=val_ds,\n","    callbacks = [es],\n","    epochs=EPOCHS)"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2021-11-18T10:18:36.414946Z","iopub.status.idle":"2021-11-18T10:18:36.417303Z","shell.execute_reply":"2021-11-18T10:18:36.417077Z","shell.execute_reply.started":"2021-11-18T10:18:36.417043Z"},"trusted":true},"outputs":[],"source":["custom_test_loss = main_model.evaluate(test_ds)\n","print(f'[INFO] Test Loss: {custom_test_loss}')"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2021-11-18T10:18:36.421587Z","iopub.status.idle":"2021-11-18T10:18:36.421883Z","shell.execute_reply":"2021-11-18T10:18:36.421753Z","shell.execute_reply.started":"2021-11-18T10:18:36.421736Z"},"trusted":true},"outputs":[],"source":["#! pip install wandb -qqq\n","#import wandb"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2021-11-18T10:18:36.426929Z","iopub.status.idle":"2021-11-18T10:18:36.427389Z","shell.execute_reply":"2021-11-18T10:18:36.427202Z","shell.execute_reply.started":"2021-11-18T10:18:36.427180Z"},"trusted":true},"outputs":[],"source":["#run = wandb.init(entity=\"caiopinho\",\n","#                 project=\"under-attention\",\n","#                 group=\"Show_Attend_Tell\",\n","#                 name=\"SAT-Baseline\")"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2021-11-18T10:18:36.431843Z","iopub.status.idle":"2021-11-18T10:18:36.432302Z","shell.execute_reply":"2021-11-18T10:18:36.432119Z","shell.execute_reply.started":"2021-11-18T10:18:36.432097Z"},"trusted":true},"outputs":[],"source":["plt.plot(history.history[\"custom_loss\"], label=\"train_loss\")\n","plt.plot(history.history[\"val_custom_loss\"], label=\"val_loss\")\n","plt.title(\"Loss vs. Epoch\")\n","plt.xlabel(\"Epoch #\")\n","plt.ylabel(\"Loss\")\n","plt.legend(loc=\"lower left\")\n","\n","plt.savefig(\"loss.png\")\n","plt.show()"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2021-11-18T10:18:36.433731Z","iopub.status.idle":"2021-11-18T10:18:36.434213Z","shell.execute_reply":"2021-11-18T10:18:36.433917Z","shell.execute_reply.started":"2021-11-18T10:18:36.433898Z"},"trusted":true},"outputs":[],"source":["#run.log({\"Loss\":wandb.Image(\"loss.png\")})"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2021-11-18T10:18:36.435929Z","iopub.status.idle":"2021-11-18T10:18:36.436433Z","shell.execute_reply":"2021-11-18T10:18:36.436221Z","shell.execute_reply.started":"2021-11-18T10:18:36.436197Z"},"trusted":true},"outputs":[],"source":["# Save the weights of the model for better reproducibility\n","main_model.encoder.save_weights(\"encoderPT2.h5\")\n","main_model.decoder.save_weights(\"decoderPT2.h5\")\n","\n","#artifact = wandb.Artifact('model-weights', type='model')\n","\n","# Add a file to the artifact's contents\n","#artifact.add_file('encoder.h5')\n","#artifact.add_file('decoder.h5')\n","\n","# Save the artifact version to W&B and mark it as the output of this run\n","#run.log_artifact(artifact)"]},{"cell_type":"markdown","metadata":{},"source":["# Catpions"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2021-11-18T10:18:36.438250Z","iopub.status.idle":"2021-11-18T10:18:36.438912Z","shell.execute_reply":"2021-11-18T10:18:36.438535Z","shell.execute_reply.started":"2021-11-18T10:18:36.438507Z"},"trusted":true},"outputs":[],"source":["# Test the decoder\n","encoder = CNN_Encoder(EMBEDDING_DIM, 1)\n","decoder = RNN_Decoder(EMBEDDING_DIM, UNITS, VOCAB_SIZE, 1)\n","\n","for image, caption in train_ds.take(1):\n","    features = encoder(tf.expand_dims(image[1],0))\n","    print(f\"ENCODER OUTPUT: {features.shape}\")\n","    hidden = decoder.reset_state()\n","    dec_input = tf.expand_dims([tokenizer.word_index['<start>']], 1)\n","    predictions, hidden, attn_weights = decoder(dec_input, features, hidden)\n","    print(f\"PREDICTION: {predictions.shape}\")\n","    print(f\"HIDDEN: {hidden.shape}\")\n","    print(f\"ATTENTION: {attn_weights.shape}\")"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2021-11-18T10:18:36.440444Z","iopub.status.idle":"2021-11-18T10:18:36.440847Z","shell.execute_reply":"2021-11-18T10:18:36.440642Z","shell.execute_reply.started":"2021-11-18T10:18:36.440621Z"},"trusted":true},"outputs":[],"source":["encoder.load_weights(\"../input/encdec1/encoderPT.h5\")\n","decoder.load_weights(\"../input/encdec1/decoderPT.h5\")"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2021-11-18T10:18:36.442062Z","iopub.status.idle":"2021-11-18T10:18:36.442420Z","shell.execute_reply":"2021-11-18T10:18:36.442249Z","shell.execute_reply.started":"2021-11-18T10:18:36.442230Z"},"trusted":true},"outputs":[],"source":["def evaluate(image):\n","    #                          max_length  64\n","    attention_plot = np.zeros((72, KERNEL))\n","\n","    hidden = decoder.reset_state()\n","\n","    img = tf.expand_dims(load_img(image), 0)\n","    features = encoder(img)\n","\n","    dec_input = tf.expand_dims([tokenizer.word_index['<start>']], 0)\n","    result = []\n","\n","    for i in range(72):\n","        predictions, hidden, attention_weights = decoder(dec_input, features, hidden)\n","\n","        attention_plot[i] = tf.reshape(attention_weights, (-1, )).numpy()\n","\n","        predicted_id = tf.random.categorical(predictions, 1)[0][0].numpy()\n","        result.append(tokenizer.index_word[predicted_id])\n","\n","        if tokenizer.index_word[predicted_id] == '<end>':\n","            return result, attention_plot\n","\n","        dec_input = tf.expand_dims([predicted_id], 0)\n","\n","    attention_plot = attention_plot[:len(result), :]\n","    return result, attention_plot"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2021-11-18T10:18:36.443589Z","iopub.status.idle":"2021-11-18T10:18:36.443985Z","shell.execute_reply":"2021-11-18T10:18:36.443798Z","shell.execute_reply.started":"2021-11-18T10:18:36.443778Z"},"trusted":true},"outputs":[],"source":["def plot_attention(image, result, attention_plot):\n","    temp_image = np.array(Image.open(image))\n","\n","    fig = plt.figure(figsize=(10, 10))\n","\n","    len_result = len(result)\n","    for l in range(len_result):\n","        temp_att = np.resize(attention_plot[l], (8, 8))\n","        ax = fig.add_subplot(len_result//2, len_result//2, l+1)\n","        ax.set_title(result[l])\n","        img = ax.imshow(temp_image)\n","        ax.imshow(temp_att, cmap='gray', alpha=0.6, extent=img.get_extent())\n","\n","    plt.tight_layout()\n","    plt.show()"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2021-11-18T10:18:36.445173Z","iopub.status.idle":"2021-11-18T10:18:36.445544Z","shell.execute_reply":"2021-11-18T10:18:36.445373Z","shell.execute_reply.started":"2021-11-18T10:18:36.445354Z"},"trusted":true},"outputs":[],"source":["from PIL import Image"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2021-11-18T10:18:36.446746Z","iopub.status.idle":"2021-11-18T10:18:36.447143Z","shell.execute_reply":"2021-11-18T10:18:36.446967Z","shell.execute_reply.started":"2021-11-18T10:18:36.446936Z"},"trusted":true},"outputs":[],"source":["image_url = 'https://cdn.crello.com/api/media/medium/384647790/stock-photo-smiling-young-man-reading-newspaper?token='\n","image_extension = image_url[-4:]\n","image_path = tf.keras.utils.get_file('image4'+image_extension,\n","                                     origin=image_url)\n","print(image_extension)\n","print(image_path)\n","result, attention_plot = evaluate(image_path)\n","print ('Prediction Caption:', ' '.join(result))\n","#plot_attention(image_path, result, attention_plot)\n","# opening the image\n","Image.open(image_path)"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2021-11-18T10:18:36.448435Z","iopub.status.idle":"2021-11-18T10:18:36.448858Z","shell.execute_reply":"2021-11-18T10:18:36.448650Z","shell.execute_reply.started":"2021-11-18T10:18:36.448628Z"},"trusted":true},"outputs":[],"source":["image_url = 'https://image.freepik.com/fotos-gratis/foto-horizontal-de-uma-linda-mulher-de-pele-escura-com-penteado-afro-sorriso-largo-dentes-brancos-mostra-algo-legal-para-um-amigo-aponta-para-o-canto-superior-direito-encosta-na-parede_273609-16442.jpg'\n","image_extension = image_url[-4:]\n","image_path = tf.keras.utils.get_file('image5'+image_extension,\n","                                     origin=image_url)\n","\n","result, attention_plot = evaluate(image_path)\n","print ('Prediction Caption:', ' '.join(result))\n","plot_attention(image_path, result, attention_plot)\n","# opening the image\n","Image.open(image_path)"]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.10"}},"nbformat":4,"nbformat_minor":4}