## 2 – Neural machine translation with attention

- If you had to translate a book’s paragraph from French to English, you would not read the whole paragraph, then close the book and translate.
- Even during the translation process, you would read/re-read and focus on the parts of the French paragraph corresponding to the parts of the English you are writing down.
- The attention mechanism tells a Neural Machine Translation model where it should pay attention to at any step.

### 2.1 – Attention mechanism

In this part, you will implement the attention mechanism presented in the lecture videos.

- Here is a figure to remind you how the model works.
- The diagram on the left shows the attention model.
- The diagram on the right shows what one “attention” step does to calculate the attention variables \(\alpha^{\langle t, t’ \rangle}\).
- The attention variables \(\alpha^{\langle t, t’ \rangle}\) are used to compute the context variable \(context^{\langle t \rangle}\) for each timestep in the output (\(t=1, \ldots, T_y\)).

Here are some properties of the model that you may notice:

#### Pre-attention and Post-attention LSTMs on both sides of the attention mechanism

- There are two separate LSTMs in this model (see diagram on the left): pre-attention and post-attention LSTMs.
*Pre-attention*Bi-LSTM is the one at the bottom of the picture is a Bi-directional LSTM and comes*before*the attention mechanism.- The attention mechanism is shown in the middle of the left-hand diagram.
- The pre-attention Bi-LSTM goes through \(T_x\) time steps

*Post-attention*LSTM: at the top of the diagram comes*after*the attention mechanism.- The post-attention LSTM goes through \(T_y\) time steps.

- The post-attention LSTM passes the hidden state \(s^{\langle t \rangle}\) and cell state \(c^{\langle t \rangle}\) from one time step to the next.

#### An LSTM has both a hidden state and cell state

- In the lecture videos, we were using only a basic RNN for the post-attention sequence model
- This means that the state captured by the RNN was outputting only the hidden state \(s^{\langle t\rangle}\).

- In this assignment, we are using an LSTM instead of a basic RNN.
- So the LSTM has both the hidden state \(s^{\langle t\rangle}\) and the cell state \(c^{\langle t\rangle}\).

#### Each time step does not use predictions from the previous time step

- Unlike previous text generation examples earlier in the course, in this model, the post-attention LSTM at time t does not take the previous time step’s prediction \(y^{\langle t-1 \rangle}\) as input.
- The post-attention LSTM at time ‘t’ only takes the hidden state \(s^{\langle t\rangle}\) and cell state \(c^{\langle t\rangle}\) as input.
- We have designed the model this way because unlike language generation (where adjacent characters are highly correlated) there isn’t as strong a dependency between the previous character and the next character in a YYYY-MM-DD date.

#### Concatenation of hidden states from the forward and backward pre-attention LSTMs

- \(\overrightarrow{a}^{\langle t \rangle}\): hidden state of the forward-direction, pre-attention LSTM.
- \(\overleftarrow{a}^{\langle t \rangle}\): hidden state of the backward-direction, pre-attention LSTM.
- \(a^{\langle t \rangle} = [\overrightarrow{a}^{\langle t \rangle}, \overleftarrow{a}^{\langle t \rangle}]\): the concatenation of the activations of both the forward-direction \(\overrightarrow{a}^{\langle t \rangle}\) and backward-directions \(\overleftarrow{a}^{\langle t \rangle}\) of the pre-attention Bi-LSTM.

#### Computing “energies” \(e^{\langle t, t’ \rangle}\) as a function of \(s^{\langle t-1 \rangle}\) and \(a^{\langle t’ \rangle}\)

- Recall in the lesson videos “Attention Model”, at time 6:45 to 8:16, the definition of “e” as a function of \(s^{\langle t-1 \rangle}\) and \(a^{\langle t \rangle}\).
- “e” is called the “energies” variable.
- \(s^{\langle t-1 \rangle}\) is the hidden state of the post-attention LSTM
- \(a^{\langle t’ \rangle}\) is the hidden state of the pre-attention LSTM.
- \(s^{\langle t-1 \rangle}\) and \(a^{\langle t \rangle}\) are fed into a simple neural network, which learns the function to output \(e^{\langle t, t’ \rangle}\).
- \(e^{\langle t, t’ \rangle}\) is then used when computing the attention \(a^{\langle t, t’ \rangle}\) that \(y^{\langle t \rangle}\) should pay to \(a^{\langle t’ \rangle}\).

- The diagram on the right of figure 1 uses a
`RepeatVector`

node to copy \(s^{\langle t-1 \rangle}\)’s value \(T_x\) times. - Then it uses
`Concatenation`

to concatenate \(s^{\langle t-1 \rangle}\) and \(a^{\langle t \rangle}\). - The concatenation of \(s^{\langle t-1 \rangle}\) and \(a^{\langle t \rangle}\) is fed into a “Dense” layer, which computes \(e^{\langle t, t’ \rangle}\).
- \(e^{\langle t, t’ \rangle}\) is then passed through a softmax to compute \(\alpha^{\langle t, t’ \rangle}\).
- Note that the diagram doesn’t explicitly show variable \(e^{\langle t, t’ \rangle}\), but \(e^{\langle t, t’ \rangle}\) is above the Dense layer and below the Softmax layer in the diagram in the right half of figure 1.
- We’ll explain how to use
`RepeatVector`

and`Concatenation`

in Keras below.

### Implementation Details

Let’s implement this neural translator. You will start by implementing two functions: `one_step_attention()`

and `model()`

.

#### one_step_attention

- The inputs to the one_step_attention at time step t are:
- \([a^{<1>},a^{<2>}, …, a^{}]\): all hidden states of the pre-attention Bi-LSTM.
- \(s^{t-1}\): the previous hidden state of the post-attention LSTM

- one_step_attention computes:
- \([\alpha^{<t,1>},\alpha^{<t,2>}, …, \alpha^{<t,T_x>}]\): the attention weights
- \(context^{ \langle t \rangle }\): the context vector:

$$context^{<t>} = \sum_{t’ = 1}^{T_x} \alpha^{<t,t’>}a^{<t’>}\tag{1}$$

##### Clarifying ‘context’ and ‘c’

- In the lecture videos, the context was denoted \(c^{\langle t \rangle}\)
- In the assignment, we are calling the context \(context^{\langle t \rangle}\).
- This is to avoid confusion with the post-attention LSTM’s internal memory cell variable, which is also denoted \(c^{\langle t \rangle}\).

#### Implement `one_step_attention`

**Exercise**: Implement `one_step_attention()`

.

- The function
`model()`

will call the layers in`one_step_attention()`

\(T_y\) using a for-loop. - It is important that all \(T_y\) copies have the same weights.
- It should not reinitialize the weights every time.
- In other words, all \(T_y\) steps should have shared weights.

- Here’s how you can implement layers with shareable weights in Keras:
- Define the layer objects in a variable scope that is outside of the
`one_step_attention`

function. For example, defining the objects as global variables would work.- Note that defining these variables inside the scope of the function
`model`

would technically work, since`model`

will then call the`one_step_attention`

function. For the purposes of making grading and troubleshooting easier, we are defining these as global variables. Note that the automatic grader will expect these to be global variables as well.

- Note that defining these variables inside the scope of the function
- Call these objects when propagating the input.

- Define the layer objects in a variable scope that is outside of the
- We have defined the layers you need as global variables.
- Please run the following cells to create them.
- Please note that the automatic grader expects these global variables with the given variable names. For grading purposes, please do not rename the global variables.

- Please check the Keras documentation to learn more about these layers. The layers are functions. Below are examples of how to call these functions.
- RepeatVector()

var_repeated = repeat_layer(var1)

concatenated_vars = concatenate_layer([var1,var2,var3])

var_out = dense_layer(var_in)

activation = activation_layer(var_in)

dot_product = dot_layer([var1,var2])

# GRADED FUNCTION: one_step_attention def one_step_attention(a, s_prev): """ Performs one step of attention: Outputs a context vector computed as a dot product of the attention weights "alphas" and the hidden states "a" of the Bi-LSTM. Arguments: a -- hidden state output of the Bi-LSTM, numpy-array of shape (m, Tx, 2*n_a) s_prev -- previous hidden state of the (post-attention) LSTM, numpy-array of shape (m, n_s) Returns: context -- context vector, input of the next (post-attention) LSTM cell """ ### START CODE HERE ### # Use repeator to repeat s_prev to be of shape (m, Tx, n_s) so that you can concatenate it with all hidden states "a" (≈ 1 line) s_prev = repeator(s_prev) # Use concatenator to concatenate a and s_prev on the last axis (≈ 1 line) # For grading purposes, please list 'a' first and 's_prev' second, in this order. concat = concatenator([a, s_prev]) # Use densor1 to propagate concat through a small fully-connected neural network to compute the "intermediate energies" variable e. (≈1 lines) e = densor1(concat) # Use densor2 to propagate e through a small fully-connected neural network to compute the "energies" variable energies. (≈1 lines) energies = densor2(e) # Use "activator" on "energies" to compute the attention weights "alphas" (≈ 1 line) alphas = activator(energies) # Use dotor together with "alphas" and "a" to compute the context vector to be given to the next (post-attention) LSTM-cell (≈ 1 line) context = dotor([alphas,a]) ### END CODE HERE ### return context

Now you can use these layers $T_y$ times in a `for`

loop to generate the outputs, and their parameters will not be reinitialized. You will have to carry out the following steps:

- Propagate the input
`X`

into a bi-directional LSTM.- Bidirectional
- LSTM
- Remember that we want the LSTM to return a full sequence instead of just the last hidden state.

Sample code:

sequence_of_hidden_states = Bidirectional(LSTM(units=..., return_sequences=...))(the_input_X)

- Iterate for \(t = 0, \cdots, T_y-1\):
- Call
`one_step_attention()`

, passing in the sequence of hidden states \([a^{\langle 1 \rangle},a^{\langle 2 \rangle}, …, a^{ \langle T_x \rangle}]\) from the pre-attention bi-directional LSTM, and the previous hidden state $s^{}$ from the post-attention LSTM to calculate the context vector $context^{}$. - Give \(context^{<t>}\) to the post-attention LSTM cell.
- Remember to pass in the previous hidden-state \(s^{\langle t-1\rangle}\) and cell-states \(c^{\langle t-1\rangle}\) of this LSTM
- This outputs the new hidden state \(s^{<t>}\) and the new cell state \(c^{<t>}\).Sample code:
`next_hidden_state, _ , next_cell_state = post_activation_LSTM_cell(inputs=..., initial_state=[prev_hidden_state, prev_cell_state])`

Please note that the layer is actually the “post attention LSTM cell”. For the purposes of passing the automatic grader, please do not modify the naming of this global variable. This will be fixed when we deploy updates to the automatic grader.

- Apply a dense, softmax layer to \(s^{<t>}\), get the output.

Sample code:`Python output = output_layer(inputs=...)`

- Save the output by adding it to the list of outputs.

- Call
- Create your Keras model instance.
- It should have three inputs:
`X`

, the one-hot encoded inputs to the model, of shape (\(T_{x}, humanVocabSize)\)- \(s^{\langle 0 \rangle}\), the initial hidden state of the post-attention LSTM
- \(c^{\langle 0 \rangle}\)), the initial cell state of the post-attention LSTM

- The output is the list of outputs.

Sample code`Python model = Model(inputs=[...,...,...], outputs=...)`

- It should have three inputs:

# GRADED FUNCTION: model def model(Tx, Ty, n_a, n_s, human_vocab_size, machine_vocab_size): """ Arguments: Tx -- length of the input sequence Ty -- length of the output sequence n_a -- hidden state size of the Bi-LSTM n_s -- hidden state size of the post-attention LSTM human_vocab_size -- size of the python dictionary "human_vocab" machine_vocab_size -- size of the python dictionary "machine_vocab" Returns: model -- Keras model instance """ # Define the inputs of your model with a shape (Tx,) # Define s0 (initial hidden state) and c0 (initial cell state) # for the decoder LSTM with shape (n_s,) X = Input(shape=(Tx, human_vocab_size)) s0 = Input(shape=(n_s,), name='s0') c0 = Input(shape=(n_s,), name='c0') s = s0 c = c0 # Initialize empty list of outputs outputs = [] ### START CODE HERE ### # Step 1: Define your pre-attention Bi-LSTM. (≈ 1 line) a = Bidirectional(LSTM(n_a, return_sequences=True))(X) # Step 2: Iterate for Ty steps for t in range(Ty): # Step 2.A: Perform one step of the attention mechanism to get back the context vector at step t (≈ 1 line) context = one_step_attention(a, s) # Step 2.B: Apply the post-attention LSTM cell to the "context" vector. # Don't forget to pass: initial_state = [hidden state, cell state] (≈ 1 line) s, _, c = post_activation_LSTM_cell(context,initial_state=[s, c]) # Step 2.C: Apply Dense layer to the hidden state output of the post-attention LSTM (≈ 1 line) out = output_layer(s) # Step 2.D: Append "out" to the "outputs" list (≈ 1 line) outputs.append(out) # Step 3: Create model instance taking three inputs and returning the list of outputs. (≈ 1 line) model = Model(inputs=[X, s0, c0], outputs=outputs) ### END CODE HERE ### return model

#### Compile the model

- After creating your model in Keras, you need to compile it and define the loss function, optimizer and metrics you want to use.

Sample code

optimizer = Adam(lr=..., beta_1=..., beta_2=..., decay=...) model.compile(optimizer=..., loss=..., metrics=[...])

### START CODE HERE ### (≈2 lines) opt = Adam(lr=0.005, beta_1=0.9, beta_2=0.999, decay=0.01) model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy']) ### END CODE HERE ###

While training you can see the loss as well as the accuracy on each of the 10 positions of the output. The table below gives you an example of what the accuracies could be if the batch had 2 examples:

We have run this model for longer, and saved the weights. Run the next cell to load our weights. (By training a model for several minutes, you should be able to obtain a model of similar accuracy, but loading our model will save you time.)

EXAMPLES = ['3 May 1979', '5 April 09', '21th of August 2016', 'Tue 10 Jul 2007', 'Saturday May 9 2018', 'March 3 2001', 'March 3rd 2001', '1 March 2001'] for example in EXAMPLES: source = string_to_int(example, Tx, human_vocab) source = np.array(list(map(lambda x: to_categorical(x, num_classes=len(human_vocab)), source))).swapaxes(0,1) prediction = model.predict([source, s0, c0]) prediction = np.argmax(prediction, axis = -1) output = [inv_machine_vocab[int(i)] for i in prediction] print("source:", example) print("output:", ''.join(output),"\n")

source: 3 May 1979 output: 1979-05-03 source: 5 April 09 output: 2009-05-05 source: 21th of August 2016 output: 2016-08-21 source: Tue 10 Jul 2007 output: 2007-07-10 source: Saturday May 9 2018 output: 2018-05-09 source: March 3 2001 output: 2001-03-03 source: March 3rd 2001 output: 2001-03-03 source: 1 March 2001 output: 2001-03-01

## 3 – Visualizing Attention (Optional / Ungraded)

def plot_attention_map(model, input_vocabulary, inv_output_vocabulary, text, n_s = 128, num = 6, Tx = 30, Ty = 10): """ Plot the attention map. """ attention_map = np.zeros((10, 30)) Ty, Tx = attention_map.shape s0 = np.zeros((1, n_s)) c0 = np.zeros((1, n_s)) layer = model.layers[num] encoded = np.array(string_to_int(text, Tx, input_vocabulary)).reshape((1, 30)) encoded = np.array(list(map(lambda x: to_categorical(x, num_classes=len(input_vocabulary)), encoded))) f = K.function(model.inputs, [layer.get_output_at(t) for t in range(Ty)]) r = f([encoded, s0, c0]) for t in range(Ty): for t_prime in range(Tx): attention_map[t][t_prime] = r[t][0,t_prime,0] # Normalize attention map # row_max = attention_map.max(axis=1) # attention_map = attention_map / row_max[:, None] prediction = model.predict([encoded, s0, c0]) predicted_text = [] for i in range(len(prediction)): predicted_text.append(int(np.argmax(prediction[i], axis=1))) predicted_text = list(predicted_text) predicted_text = int_to_string(predicted_text, inv_output_vocabulary) text_ = list(text) # get the lengths of the string input_length = len(text) output_length = Ty # Plot the attention_map plt.clf() f = plt.figure(figsize=(8, 8.5)) ax = f.add_subplot(1, 1, 1) # add image i = ax.imshow(attention_map, interpolation='nearest', cmap='Blues') # add colorbar cbaxes = f.add_axes([0.2, 0, 0.6, 0.03]) cbar = f.colorbar(i, cax=cbaxes, orientation='horizontal') cbar.ax.set_xlabel('Alpha value (Probability output of the "softmax")', labelpad=2) # add labels ax.set_yticks(range(output_length)) ax.set_yticklabels(predicted_text[:output_length]) ax.set_xticks(range(input_length)) ax.set_xticklabels(text_[:input_length], rotation=45) ax.set_xlabel('Input Sequence') ax.set_ylabel('Output Sequence') # add grid and legend ax.grid() #f.show() return attention_map attention_map = plot_attention_map(model, human_vocab, inv_machine_vocab, "Tuesday 09 Oct 1993", num = 7, n_s = 64);