less than 1 minute read

Tags: , , ,

RNN: Backpropagation through time

Imgur

RNN: Different types of RNNs

Imgur

Imgur

Summary RNN architectures

Imgur

RNN: Language model and sequence generation

What is language modelling?

  • Speech recongnition

    • The apple and pair salad.
    • The apple and pear salad.
        P(The apple and pair salad) = 3.2 x 10 ^ -13
        P(The apple and pear salad) = 5.7 x 10 ^ -10
        P(Sentences) = ?
    

Language modelling with an RNN

  • Training set: large corpus (large body or vary large set of english text of eenglish sentences) of english text.

Imgur

RNN: Sampling novel sequences

Sampling a sequence from a trained RNN

Imgur

Character-level language model

Imgur

Sequence Generation

Imgur

RNN: Vanishing gradients with RNNs

Imgur

從英文來說,單數 (cat —> was),複數 (cats —> were),中間的子句有時候很長時,會容易造成,Vanishing gradients

RNN: gated recurrent unit (GRU)

Imgur

GRU(simplified)

Imgur

Full GRU

Imgur

RNN: long short term memory (LSTM)

GRU and LSTM

Imgur

Imgur

LSTM

Imgur

RNN: bidirectionisal RNN

  • disadvantage:
    • need the entire sequence of data before you can make predictions anywhere
    • ex: real time speech recognition system

Imgur

RNN: deep RNNs

Imgur

Imgur