- Time: Wednesday 06:30PM-09:00PM, fall semester, 2020
- Meeting Location: Zoom Meetings
- Instructor: Yue Ning
- Office: Gateway South 448
- Office Hours: Mondays 8:00AM - 10:00AM
- Teaching assistant: Kun Wu firstname.lastname@example.org
- Course details: We will be using Canvas for online discussion, announcements, and homework submission. You are encouraged to ask and answer questions on the forum as long as you do not give away solutions to homework problems. Participation in the online forum will count towards class participation.
What is this course about?Natural language processing (NLP) is one of the most important technologies referring to automatic computational processing of human languages. This includes algorithms that take human-produced text as input or produce text as output. People communicate almost everything in language: emails, phone calls, language translation, web searches, reports, books, social media, etc. Human language is symbolic in nature and also highly ambiguous and variable. Comprehending human language is a crucial and challenging part of artificial intelligence. There are a large variety of underlying tasks and machine learning models behind NLP applications. Recently, deep learning approaches have been studied and achieved high performance in many NLP tasks. The course provides an introduction to machine learning and deep learning research applied to NLP. We will cover topics including word vector representations, neural networks, recurrent neural networks, convolutional neural networks, seq2seq models, as well as some attention-based models.
PrerequisitesThis course is fast-paced and covers a lot of ground, so it is important that you have a solid foundation on both the theoretical and empirical fronts. You should have background in python programming, probability theory, linear algebra, calculus, and foundations of machine learning.
- Ian Goodfellow and Yoshua Bengio and Aaron Courville. Deep Learning (DL). 2016. MIT Press. We will cover topics including basic neural networks, back propagation, RNNs and CNNs.
- Dan Jurafsky and James H. Martin. Speech and Language Processing (3rd ed. draft) (SLP). 2018.
- Yoav Goldberg. Neural Network Methods for Natural Language Processing (NNLP). 2017.
CourseworkSubmissions: All assignments (homework problems and project milestones) must be submitted on Canvas by 6:30 PM on the due dates.
- Homework (50%): There will be bi-weekly homework assignments with both written and programming parts. Each assignment is centered around an application and will also deepen your understanding of the theoretical concepts.
- Midterm Exam (15%): The midterm exam is to evaluate your understanding of the course so far.
- Project (25%): The final project provides an opportunity for you to use the tools from class to build something interesting of your choice. You need to make a presentation in class and submit a report.
- Participation (5%): Discussions on Canvas and attending classes.
- Quizzes (5%): Online pop quizzes.
|1||Sep. 2||Introduction to NLP||CH1-3 SLP; CH1 NNLP|
|2||Sep. 9||Machine Learning Basics & Neural Networks||CH2-5 DL; CH2-5 NNLP|
|3||Sep. 16||Vector Semantics: TFIDF, CBOW, Skip-gram, Glove||
CH 6 SLP; CH6-8, CH10-11 NNLP
- Mikolov et al. Efficient Estimation of Word Representations in Vector Space. 2013 (word2vec)
- Mikolov et al. Distributed Representations of Words and Phrases and their Compositionality. 2013 (Negative sampling)
- Pennington et al. GloVe: Global Vectors for Word Representation. 2014
|4||Sep. 23||Deep Feedforward Networks||
- Wager et al. Dropout Training as Adaptive Regularization. 2013
- Ning Qian. On the momentum term in gradient descent learning algorithms
- Ruder et al. An overview of gradient descent optimization algorithms. 2017
|5||Sep. 30||Language Modeling||
CH 3 SLP; CH9 NNLP
- Bengio et al. A Neural Probabilistic Language Model. 2003
|6||Oct. 7||Recurrent Neural Networks (RNNs)||
CH 9, 13 SLP; CH 14 NNLP
- Afshine Amidi and Shervine Amidi. Recurrent Neural Networks cheatsheet. 2019
|7||Oct. 14||More on RNNs (vanishing gradients)||
CH10 DL; CH15-16 NNLP
- Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. 1997 (LSTM)
- Cho et al. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. 2014
|8||Oct. 21||Convolutional Neural Networks (CNNs)||
CH9 DL; CH 13 NNLP
- LeCun et al. Gradient-based learning applied to document recognition. 1998
|9||Oct. 28||Midterm exam|
|10||Nov. 4||Machine Translation, Seq2seq models, Attention models||
CH 10 DL; CH 17 NNLP
- Sutskever et al. Sequence to Sequence Learning with Neural Networks. 2014
|11||Nov. 11||Natural Language Generation||
- Rush et al. A Neural Attention Model for Abstractive Sentence Summarization. 2015
- Yue Dong. A Survey on Neural Network-Based Summarization Methods. 2018
|12||Nov. 18||Tree Recursive Neural Networks and Constituency Parsing||
CH 18 NNLP
- Socher et al. Parsing with Compositional Vector Grammars. 2013
- Socher et al. Semantic Compositionality through Recursive Matrix-Vector Spaces. 2012
- Socher et al. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. 2013
- Kitaev et al. Constituency Parsing with a Self-Attentive Encoder. 2018
|13||Nov. 25||Thanksgiving - No class|
|14||Dec. 2||Dependency Parsing||
- Chen and Manning. A Fast and Accurate Dependency Parser using Neural Networks. 2014
|15||Dec. 9||Project presentation||-|
|16||Dec. 16||Project presentation||-||Project report due|