3232 < main class ="container ">
3333 < section id ="welcome ">
3434 < h1 > Module 1: Recurrent Neural Networks and LSTM</ h1 >
35+
3536 < div class ="content-box ">
3637 < h2 > Module Overview</ h2 >
3738 < p > This module introduces Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks,
@@ -58,6 +59,325 @@ <h2>Learning Objectives</h2>
5859 </ div >
5960 </ section >
6061
62+ < section class ="content-box ">
63+ < h2 > Objective 01 - Describe Neural Networks Used for Modeling Sequences</ h2 >
64+ < h3 > Overview</ h3 >
65+ < p > We have reached the last sprint of the core data science curriculum! In this unit so far, we have created
66+ and trained feed-forward neural networks. While we can do a lot with this type of neural network, some
67+ types of data work better with different architectures.</ p >
68+ < p > This module will explore recurrent neural networks (RNN), and a type of RNN called a long short-term
69+ memory (LSTM) network. These architectures are well suited for processing sequences and using them for
70+ many natural language processing tasks.</ p >
71+
72+ < h3 > Sequence</ h3 >
73+ < p > A sequence is a collection of objects (integers, floats, characters, tokens, and other data types) where
74+ you can repeat the order of matter and objects. A Python list is an example, as well as NumPy arrays.
75+ Many of the data structures we use are built on basic sequences.</ p >
76+
77+ < h3 > Time Series</ h3 >
78+ < p > A time series is a data where you have not just the order but some actual continuous marker for where the
79+ points lie “in time” - this could be a date, a timestamp, Unix time, or something else. Of course, all
80+ time series are also sequences, and for some techniques, you might consider the order of the sequence
81+ and not the separation (in time) of the entries.</ p >
82+
83+ < h3 > Recursion</ h3 >
84+ < p > In mathematics, recursion is defining objects based on previously defined other objects of the same type.
85+ In other words, recursion is something that happens when a thing calls itself one or more times.</ p >
86+ < p > For example, a recursive function calls itself and uses its previous terms to define subsequent terms.
87+ < a href ="https://en.wikipedia.org/wiki/Pascal%27s_triangle " target ="_blank "
88+ rel ="noopener noreferrer "> Pascal's Triangle</ a > is an example of using previous terms to calculate
89+ subsequent terms: each number is the sum of the two numbers directly above it.
90+ </ p >
91+ < p > In computer science, a recursive function calls itself from within its code.</ p >
92+
93+ < h3 > Recurrent Neural Networks (RNN)</ h3 >
94+ < p > Remember that a feed-forward neural network has an input layer and then some number of hidden layers. The
95+ output from each layer is fed into the next layer without any feedback. In contrast, with a recurrent
96+ neural network, there is a layer where the output from the nodes feeds back into itself. This layer is
97+ called the recurrent layer.</ p >
98+ < p > Simple RNNs have a weakness called the vanishing gradient problem: the recursive aspect sometimes results
99+ in the back-propagation gradients either exploding or becoming very small (vanishing). So what can we
100+ do?</ p >
101+
102+ < h3 > Long short-term memory (LSTM) network</ h3 >
103+ < p > To prevent the vanishing gradient problem, we can create a memory state within the network that adds to
104+ the gradients; this prevents them from becoming too small. You can learn more about the structure of the
105+ LSTM network in < a
106+ href ="https://adventuresinmachinelearning.com/recurrent-neural-networks-lstm-tutorial-tensorflow/ "
107+ target ="_blank " rel ="noopener noreferrer "> this article</ a > . For now, we'll focus on how to implement
108+ them
109+ and what types of problems they are suitable for.</ p >
110+
111+ < h3 > Follow Along</ h3 >
112+ < p > In this section, we'll first look at the option in Keras for creating a simple neural network with a
113+ recurrent layer. The < code > keras.layers.SimpleRNN</ code > is a fully connected RNN where the output from
114+ the previous time step is fed to the next time step.</ p >
115+ < pre > < code > # Example: https://keras.io/guides/working_with_rnns/
116+
117+ # Imports
118+ import numpy as np
119+ import tensorflow as tf
120+ from tensorflow import keras
121+ from tensorflow.keras import layers
122+
123+ # Instantiate the model
124+ model = keras.Sequential()
125+ model.add(layers.Embedding(input_dim=1000, output_dim=64))
126+
127+ # The output of SimpleRNN will be a 2D tensor of shape (batch_size, 128)
128+ model.add(layers.SimpleRNN(128))
129+
130+ # Add an additional hidden layer
131+ model.add(layers.Dense(10))
132+
133+ # View the architecture
134+ model.summary()</ code > </ pre >
135+ < pre > < code > Model: "sequential"
136+ _________________________________________________________________
137+ Layer (type) Output Shape Param #
138+ =================================================================
139+ embedding (Embedding) (None, None, 64) 64000
140+ _________________________________________________________________
141+ simple_rnn (SimpleRNN) (None, 128) 24704
142+ _________________________________________________________________
143+ dense (Dense) (None, 10) 1290
144+ =================================================================
145+ Total params: 89,994
146+ Trainable params: 89,994
147+ Non-trainable params: 0
148+ _________________________________________________________________
149+ </ code > </ pre >
150+ < p > Next, we can also create a network with a LSTM layer.</ p >
151+ < pre > < code > # Example: https://keras.io/guides/working_with_rnns/
152+
153+ # LSTM network example
154+ model = keras.Sequential()
155+ # Add an Embedding layer expecting input vocab of size 1000, and
156+ # output embedding dimension of size 64.
157+ model.add(layers.Embedding(input_dim=1000, output_dim=64))
158+
159+ # Add a LSTM layer with 128 internal units.
160+ model.add(layers.LSTM(128))
161+
162+ # Add a Dense layer with 10 units.
163+ model.add(layers.Dense(10))
164+
165+ model.summary()</ code > </ pre >
166+ < pre > < code > Model: "sequential_1"
167+ _________________________________________________________________
168+ Layer (type) Output Shape Param #
169+ =================================================================
170+ embedding_1 (Embedding) (None, None, 64) 64000
171+ _________________________________________________________________
172+ lstm (LSTM) (None, 128) 98816
173+ _________________________________________________________________
174+ dense_1 (Dense) (None, 10) 1290
175+ =================================================================
176+ Total params: 164,106
177+ Trainable params: 164,106
178+ Non-trainable params: 0
179+ _________________________________________________________________
180+ </ code > </ pre >
181+ < h3 > Challenge</ h3 >
182+ < p > Before class time, it would be good to review the Keras: Working with RNNs documentation. Ensure you know
183+ how to add a recurrent layer and the difference between a simple RNN and LSTM.</ p >
184+
185+ < h3 > Additional Resources</ h3 >
186+ < ul >
187+ < li > < a href ="https://keras.io/guides/working_with_rnns/ " target ="_blank "
188+ rel ="noopener noreferrer "> Keras: Working with RNNs</ a > </ li >
189+ < li > < a href ="https://adventuresinmachinelearning.com/recurrent-neural-networks-lstm-tutorial-tensorflow/ "
190+ target ="_blank " rel ="noopener noreferrer "> Recurrent Neural Networks: LSTM Tutorial</ a > </ li >
191+ </ ul >
192+ </ section >
193+
194+ < section class ="content-box ">
195+ < h2 > Objective 02 - Apply an LSTM to a Text Generation Problem Using Keras</ h2 >
196+ < h3 > Overview</ h3 >
197+ < p > In the first part of this module, we generally learned why recurrent neural networks are a good choice
198+ for working with sequential data, such as text. Now, we will implement a specific type of RNN called a
199+ long short-term memory network (LSTM) to make text predictions.</ p >
200+ < p > LSTM networks are suitable for text prediction and generation because they can remember long sequences of
201+ data. So, let's test out how to implement an LSTM network with text prediction.</ p >
202+ < h3 > Follow Along</ h3 >
203+ < p > We'll use text from < a href ="https://www.gutenberg.org/ " target ="_blank "
204+ rel ="noopener noreferrer "> Project Gutenberg</ a > and use a portion of it to train the neural
205+ network. The novel is the Adventures of Sherlock Holmes by Arthur Conan Doyle; the shortened text used
206+ in the following analysis is also available < a
207+ href ="https://raw.githubusercontent.com/bloominstituteoftechnology/data-science-practice-datasets/main/unit_4/sherlock.txt "
208+ target ="_blank " rel ="noopener noreferrer "> here</ a > .</ p >
209+ < pre > < code > # Load the text
210+ import requests
211+
212+ url = "https://raw.githubusercontent.com/bloominstituteoftechnology/data-science-practice-datasets/main/unit_4/sherlock.txt"
213+ response = requests.get(url)
214+ text = response.text
215+
216+ # Strip the \r\n characters
217+ text = text.replace('\r\n', ' ')
218+ </ code > </ pre >
219+ < p > We now have a single string of text. However, the neural network input needs to be numeric, so we must
220+ convert or encode the text as characters. We can create two look-up tables: character to integer and
221+ integer to character (to make predictions after training).</ p >
222+ < pre > < code > # Encode Data as Chars
223+
224+ # Find the unique characters
225+ chars = list(set(text))
226+
227+ # Lookup tables
228+ char_int = {c:i for i, c in enumerate(chars)}
229+ int_char = {i:c for i, c in enumerate(chars)}
230+
231+ print('The number of unique characters in the text:', len(chars))
232+ </ code > </ pre >
233+ < pre > < code > The number of unique characters in the text: 91</ code > </ pre >
234+ < p > Now we need to create sequences of the characters to train on.</ p >
235+ < pre > < code > # Create the sequence data
236+ maxlen = 40
237+ step = 5
238+
239+ # Encode the characters using the lookup tables
240+ encoded = [char_int[c] for c in text]
241+
242+ # Initialize empty lists to hold the sequences
243+ sequences = [] # Each element is 40 chars long
244+ next_char = [] # One element for each sequence
245+
246+ # Loop through the entire text
247+ for i in range(0, len(encoded) - maxlen, step):
248+ sequences.append(encoded[i : i + maxlen])
249+ next_char.append(encoded[i + maxlen])
250+
251+ print('sequences: ', len(sequences))
252+ </ code > </ pre >
253+ < pre > < code > sequences: 54974</ code > </ pre >
254+ < p > And now that the text is processed, we can build our model! We'll use a Keras utility to pad our
255+ sequences, so they are all the same length up to the maximum we specify. Then, we'll create our feature
256+ and target arrays:</ p >
257+ < pre > < code > import tensorflow as tf
258+ from tensorflow.keras.preprocessing import sequence
259+
260+ # Pad sequences so all are equal
261+ seq = tf.keras.preprocessing.sequence.pad_sequences(sequences, maxlen=40)
262+
263+ # Create x & y
264+ import numpy as np
265+
266+ # Create arrays of zeros (False)
267+ x = np.zeros((len(sequences), maxlen, len(chars)), dtype=np.bool)
268+ y = np.zeros((len(sequences), len(chars)), dtype=np.bool)
269+
270+ # Turn on the location (set to True) when the character is present
271+ for i, sequence in enumerate(sequences):
272+ for t, char in enumerate(sequence):
273+ x[i,t,char] = 1
274+
275+ y[i, next_char[i]] = 1
276+ </ code > </ pre >
277+ < p > The model we will use has an input layer equal to the number of characters in our text, a hidden layer of
278+ 64 nodes, an LSTM layer of 64 nodes, and an output layer equal to the character set's size. We are
279+ predicting one of the characters, so we need to reflect that in the output.</ p >
280+ < pre > < code > # Build the model: a single LSTM
281+ from keras.models import Sequential
282+ from tensorflow.keras.layers import Dense, LSTM
283+ from tensorflow.keras.layers import Bidirectional, Embedding
284+
285+ model = Sequential()
286+ model.add(Embedding(output_dim=64, input_dim=len(chars)))
287+ model.add(Bidirectional(LSTM(64)))
288+ model.add(Dense(len(chars), activation='softmax'))
289+
290+ model.compile(loss='categorical_crossentropy', optimizer='adam')
291+ </ code > </ pre >
292+ < p > Finally, let's fit the model! We will choose a lower number of epochs for this text run because neural
293+ networks usually take some time to train. We can adjust the epochs later to see how our results change.
294+ </ p >
295+ < pre > < code > # Fit the model
296+ model.fit(seq, y, batch_size=32,
297+ epochs=5, verbose=2)
298+ </ code > </ pre >
299+ < pre > < code > Epoch 1/5
300+ 1718/1718 - 59s - loss: 2.5776
301+ Epoch 2/5
302+ 1718/1718 - 59s - loss: 2.2019
303+ Epoch 3/5
304+ 1718/1718 - 60s - loss: 2.0714
305+ Epoch 4/5
306+ 1718/1718 - 59s - loss: 1.9853
307+ Epoch 5/5
308+ 1718/1718 - 60s - loss: 1.9195
309+
310+ < tensorflow .python.keras.callbacks.History at 0x7f2e49e5fb70 >
311+ </ code > </ pre >
312+ < p > Once we fit the model, we need to convert the numeric predictions back into characters, so that we can
313+ read it. We'll create a function to do this.</ p >
314+ < pre > < code > # Predict and convert text back into characters
315+ def generate_text(model, seed, length):
316+
317+ encoded = [char_int[c] for c in seed]
318+
319+ generated = ''
320+ generated += seed
321+ model.reset_states()
322+
323+ start_index = 0
324+
325+ for _ in range(length):
326+
327+ sample = encoded[start_index:start_index+10]
328+ sample = np.array(sample)
329+ sample = np.expand_dims(sample,0)
330+
331+ pred = model.predict(sample)
332+ pred = tf.squeeze(pred, 0)
333+ next_char = np.argmax(pred)
334+ encoded.append(next_char)
335+ generated += int_char[next_char]
336+
337+ start_index += 1
338+
339+ return generated</ code > </ pre >
340+ < pre > < code > # Set the seed text which the model will use to generate the predicted text
341+ seed_text = "I have no data yet it is a capital mistake to theorise before one has data insensibly one begins to twist facts to suit theories"
342+
343+ generate_text(model, seed_text, 400)</ code > </ pre >
344+ < pre > < code > 'I have no data yet it is a capital mistake to theorise before one has data insensibly one begins to twist facts to suit theoriestoyhraov an an a lomenenlaent ne th the k nedae are tf tav aovhnan ertenee af aeaeng ah thet aoske ah thrneahe k nd r eenneandt dt sane tdtytd aovtheohe sntov rdane ahathhset nee rgavtirtddtn th rdt t ahe a“ n edt aheee r dtntoatheavdtodrd aootttd aheo ea“ ne erd dtoooneteosd an n e d aovdteate ne ee eahetheoothh“ th ftetveaah ddteteoointeerre r eeah nn e etn dnthvrftovtvtaaeonkk '</ code > </ pre >
345+ < p > Well, that is interesting! We have something resembling language, but the words don't make any sense - I
346+ don't know what an “aootttd” is, but it could be exciting! There also isn't any punctuation or other
347+ structure in the text. But, we only trained the network for five epochs, which isn't very many.</ p >
348+ < p > Let's increase that to 100 epochs and compare the output, using the same seed text.</ p >
349+ < pre > < code > # Train with more epochs
350+ model.fit(seq, y, batch_size=32,
351+ epochs=100, verbose=0)</ code > </ pre >
352+ < pre > < code > <tensorflow.python.keras.callbacks.History at 0x7f2e463b94e0></ code > </ pre >
353+ < pre > < code > # Set the seed text which the model will use to generate the predicted text
354+ seed_text = "I have no data yet it is a capital mistake to theorise before one has data insensibly one begins to twist facts to suit theories"
355+
356+ generate_text(model, seed_text, 400)</ code > </ pre >
357+ </ code > </ pre >
358+ < pre > < code > 'I have no data yet it is a capital mistake to theorise before one has data insensibly one begins to twist facts to suit theoriesdoiinwoktis asty fa-eeiclwegtrgssah bhe nt.nrlfc-rrtdxoed GevccGsatrtin!y ing prlosa,IoIwoectiiocc.-ihIcpez bhe cs,.nrrgin?hj—ffcr trmc!séBe ,eoit-l suent ew E_ eTeoaiebmiL4aelay4ve:img” wseuWeoocet t t s onn”y”“j”]g i IeenoTTlJ ” ana”,e”'oeeoIieaepaovP kt HeCtrt i xii vO'zllr1mcsasg?b! '' e dn e hh lnhdnnr rs o h eLcn. Oa rtt ddzt eeoIdT ddc s snnnn”n£sJFsœe,aT e-Meee ioS s e'</ code > </ pre >
359+ < p > Now we can see that the text is starting to develop some structure, with punctuation and even a few words
360+ that seem more like words?</ p >
361+ < p > We kept this example simple so that you could see how to set up an LSTM for generating text. Usually, you
362+ would use more layers to capture the structure of the text better.</ p >
363+ < h3 > Challenge</ h3 >
364+ < p > Now it is up to you! Using the exact text and code above, add additional layers to the network and see if
365+ you can improve the text prediction.</ p >
366+ < p > You can even take it a step further and source a new text, load it, and process it in the same way, and
367+ see what your network can generate.</ p >
368+
369+ < h3 > Additional Resources</ h3 >
370+ < ul >
371+ < li > < a href ="https://colah.github.io/posts/2015-08-Understanding-LSTMs/ " target ="_blank "
372+ rel ="noopener noreferrer "> Understanding LSTMs</ a > </ li >
373+ < li > < a href ="https://www.analyticsvidhya.com/blog/2018/03/text-generation-using-python-nlp/ "
374+ target ="_blank " rel ="noopener noreferrer "> Text Generation Using Python</ a > </ li >
375+ < li > < a href ="https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/ "
376+ target ="_blank " rel ="noopener noreferrer "> Text Generation With LSTM Recurrent Neural Networks in
377+ Python with Keras</ a > </ li >
378+ </ ul >
379+ </ section >
380+
61381 < section id ="guided-project ">
62382 < div class ="content-box ">
63383 < h2 > Guided Project</ h2 >
0 commit comments