Knowledge-driven Natural Language Generation

• Time: Wednesday and Friday 2pm-3:15pm
• Place: 1131 Sibel
• Instructor: Prof. Heng Ji (Email: hengji@illinois.edu; Office: 3318 Sibel; Office hours: Wednesday 3:15pm-4:15pm)
• TA: Ying Lin (Email:yinglin8@illinois.edu; Office: Siebel 1115)

Lectures and Schedules

Date	Lecture	Reading	Assignment
01/22	Overview	Neural Generation Models
01/24	Neural Natural Language Generation:seq2seq+attention	Reading 2
01/29	Neural Natural Language Generation (Cont'): Encoding and Decoding, copy mechanism	Reading 3
01/31	Controlled Natural Language Generation: Application 1: Encoding Tables and Knowledge Graphs
02/05	Encoding Knowledge Graphs with Graph Neural Networks	Reading 4
02/07	Relational Knowledge Generation, Application 2: News Timeline Generation		Assignment 1 (due on 02/21)
02/12	Hands-on Generation tools practice (by Qingyun Wang)
02/14	Application 3: Knowledgeable Dialog Generation
02/26	Application 4: Image/Video Captioning and Storytelling
02/26 6pm make-up class Room 3403	Application 4: Image/Video Captioning and Storytelling
02/28	Application 4: Knowledge-driven Image/Video Captioning
03/04	Application 1: Conditioned Table-to-Text		Assignment 2 (due on 03/25)
03/04 6pm make-up class Room 3403	Application 1: Knowledge Graph to Text and Topic-to-text
03/06	Application 5: Scientific Hypotheses Generation; Increase diversity in generation
03/11	Application 3: Discourse-aware Dialog Generation
03/13	Application 4: Image/Video to Text and the other way around (Cont')
03/25	Project proposal presentation
03/27	Project proposal presentation (Cont')
04/01	Abstractive Summary Generation		Assignment 3 (due on 04/15)
04/03	Abstractive Summary Generation (Cont')
04/08	Abstractive Summary Generation (Cont')
04/10	Counter-Argument Narrative Generation
04/15	Narrative and Creative Generation
04/17	Narrative Generation		First Paper Draft due
04/22	Hypothesis Generation
04/24	Knowledge Graph Generation
04/29	Final Project Presentations
05/01	Final Project Presentations
05/06	Final Project Presentations		Final Paper Due

Course Description

In this course we will teach machines to describe knowledge they have learned from data. We will develop a set of intelligent systems which can transform structured knowledge bases into natural language, which is an opposite direction of Information Extraction. The topics will cover both of the conventional template filling based approaches and modern neural networks based generation approaches. We will dive deep into various technical components: how to represent knowledge, how to feed knowledge into a generation model, how to evaluate generation results? We will do three project-based assignments and a final term project interesting applications including:

1. News image and video caption generation to describe entities and events
2. Generate scripts for news videos
3. Generate scientific ideas and write technical papers
4. Write technical reviews from various perspectives
5. Write a bio for all kinds of professionals, artists and paintings
6. Write a news article about scientific discovery results and perspectives about events
7. Write a history book a news timeline based on event-centric knowledge bases
8. Make Alexa smarter by feeding information from background and real-time news streams
9. Or the other way around: Generate paintings, news videos and yoga instructional videos
This is an advanced graduate-level course, and the prerequisites include Natural Language Processing and Machine Learning course.

 

Grading

• The instructor gives an overview tutorial (2pm-2:50pm), followed by Q/A and discussions (2:50pm-3:15pm).
• 3 Assignments (15pts each): participate in shared tasks and submit through Codalab, the grades will be based on your system’s rank in the class. If a submission is successfully complete then it will get 10 basic score, and then the additional score is based on the system's relative rank in the class between [0, 5]
• Pick one projet from five applications or one theory problem, final Paper submitted to EMNLP2020/CIKM2020/AACL2020/COLING2020 (55pts): you can either continue working on one of the assignments and focus on novel methods, or choose a new topic to work on. we will host open-house poster session and select a paper for best paper.

Course Materials

The instructor will write most of the slides and hand-outs, give survey about the best papers from

• ACL Papers: ACL Anthology.

The following books may provide some useful background:

• Speech and Language Processing, Daniel Jurafsky and James Martin, Prentice-Hall (second edition).
• Christopher D. Manning and Hinrich Schutze. Foundations of Statistical Natural Language Processing. MIT Press.
• Christopher M. Bishop. Pattern Recognition and Machine Learning.
• Trevor Hastie, Robert Tibshirani and Jerome Friedman. The Elements of Statistical Learning.
• Jiawei Han, Micheline Kamber, and Jian Pei, Data Mining: Concepts and Techniques, 3^rd edition, Morgan Kaufmann

Prerequisites:

• Programming in Python
• Already took NLP and machine learning courses
• Familiar with Probability and Statistics
• Research background in NLP or related fields; it's targeted for students who work on related areas. Extensive research background is expected.
• Interests in languages and lingusitics
• Solid background in algorithms, probabilities
• Good programming skills
• Sufficient mathematical background

Class Policy

• Restricted Academic Integrity
• NO Incompleteness are accepted
• NO Late Assignments are accepted
• NO cell phones and internet surfing are allowed in the classroom
• Don't come late to the class, if you are late more than 10 minutes, simply skip it in order not to disturb the lecture

Mailing List

• nlg-2020@lists.cs.illinois.edu