10 - Recurrent Neural Networks

Summary

Up to this point, we have only discussed neural approaches for text classification (e.g., review and news classification) that handle the text as a bag of words. That is, we aggregate the words either by representing them as explicit features in a feature vector or by averaging their numerical representations (i.e., embeddings). Although this strategy completely ignores the order in which words occur in a sentence, it has been repeatedly shown to be a good solution for many practical natural language processing applications that are driven by text classification. Nevertheless, for many natural language processing tasks such as part-of-speech tagging, we need to capture the word-order information more explicitly. Sequence models capture exactly this scenario, where classification decisions must be made using not only the current information but also the context in which it appears. In particular, we discuss several types of recurrent neural networks, including stacked (or deep) recurrent neural networks, bidirectional recurrent neural networks, and long short-term memory networks. Last, we introduced conditional random fields, which extend recurrent neural networks with an extra layer that explicitly models transition probabilities between two cells.