Knowledge-driven Natural Language Generation

• Time: Wednesday and Friday 2pm-3:15pm
• Place: Zoom
• Instructor: Prof. Heng Ji (Email: hengji@illinois.edu; Office hours: Wednesday and Friday 3:15pm-3:45pm)
• TA: Manling Li (manling2@illinois.edu; Office hours: Wednesday and Friay 3:15pm-3:45pm)

Lectures and Schedules

Date	Lecture	Reading	Assignment
01/19	Overview
01/21	Neural Natural Language Generation:seq2seq+attention
01/26	Neural Natural Language Generation (Cont'): Encoding and Decoding, pointer network+copy mechanism; Knowledge Controlled NLG Benchmarks	GNN
01/28	Controlled Natural Language Generation: Graph Neural Networks and Memory Network	Language Models
02/02	Neural Language Modeling	Knowledge Acquisition and Inference	Possible Project Topics
02/04	Knowledge Acquisition and Representation	Prompt Learning
02/09	Prompt Learning	Knowledge Selection for Dialogues
02/11	Application 1: Knowledge Selection for Dialog
02/16	Application 2: Knowledge Graph/Table to Text Generation, Evaluation Metrics Discussion	Abstractive Summarization
02/18	Application 3: Abstractive Summarization: basic techniques and addressing hallucination
02/23	Application 3: Abstractive Summarization: Multi-view Summarization and Argumentation Generation
02/25	Application 3: Abstractive Summarization (Cont'): Timeline Generation (Guest Lecture by Manling Li)	Long Document Representation	Assignment 1 (due on 03/11)
03/02	Application 3: Abstractive Summarization (Cont'): Summarizing Long Documents (Guest Lecture by Vicki Qi Zeng)	Generation for Fake News Detection
03/04	Application 4: Generation for Fake News Detection
03/09	Application 4: Generation for Fake News Detection (Cont')	Schema Induction	Assignment 2 (due on 03/30)
03/11	Application 5: NLG for Schema Induction; Proposal discussions
03/23	No Class
03/25	Project proposal presentation (extended class including office hour)
03/30	Application 5: NLG for Schema Induction (Cont')	NLG for IE
04/01	Application 6: NLG for Information Extraction		Caption Generation
04/06	Application 7: Background Knowledge Guided Image/Video Captioning
04/08	Application 7: Background Knowledge Guided Image/Video Captioning (Cont')
04/13	Application 8: Visual Words Generation from Images and Videos
04/15	NLG Evaluation (Guest Lecture by Ken Chan)	Assignment 3 (due on 04/29) Evaluation Metrics	First Paper Draft due
04/20	NLG Remaining Challenges
04/22	No Class
04/27	Final Project Presentations
04/29	Final Project Presentations, Special Lecture at 10am CT by Roy Bar-Haim on AI Debating
04/29	Final Project Presentations
05/04	Final Project Presentations		Final Paper Due

Course Description

In this course we will teach machines to describe knowledge they have learned from data. We will develop a set of intelligent systems which can transform structured knowledge bases into natural language, which is an opposite direction of Information Extraction. The topics will cover both of the conventional template filling based approaches and modern neural networks based generation approaches. We will dive deep into various technical components: how to represent knowledge, how to feed knowledge into a generation model, how to generate natural language controlled and guided by the acquired knowledge, how to evaluate generation results? We will do three project-based assignments and a final term project interesting applications including:

1. News image and video caption generation to describe entities and events
2. Generate scripts for news videos
3. Generate scientific ideas and write technical papers
4. Write technical reviews from various perspectives
5. Write a bio for all kinds of professionals, artists and paintings
6. Write a news article about scientific discovery results and perspectives about events
7. Write a history book a news timeline based on event-centric knowledge bases
8. Make Alexa smarter by feeding information from background and real-time news streams
9. Or the other way around: Generate paintings, news videos and yoga instructional videos
This is an advanced graduate-level course, and the prerequisites include Natural Language Processing and Machine Learning course.

 

Grading

• 3 Assignments (15pts each): participate in shared tasks and submit through Codalab.
• Term Project (45 pts in total, 5pts proposal, 15pts project implementation, 25pts final report writing/presentation) Team work is encouraged, but a team cannot be more than 4 students. Everyone is encouraged to submit papers to EMNLP2022/COLING2022 based on the term projects.
• Reading assignment (5 points): Each student is required to sign up one paper critique session, 3mins with or without slides.
• Class Participation (5 points): Extra points to students who actively participate in discussions.

Course Materials

The instructor will write most of the slides and hand-outs, give survey about the best papers from this survey paper: Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji and Meng Jiang. 2022. A Survey of Knowledge-Enhanced Text Generation. ACM Computing Survey. and

• ACL Papers: ACL Anthology.

The following books may provide some useful background:

• Speech and Language Processing, Daniel Jurafsky and James Martin, Prentice-Hall (second edition).
• Christopher D. Manning and Hinrich Schutze. Foundations of Statistical Natural Language Processing. MIT Press.
• Ian Goodfellow and Yoshua Bengio and Aaron Courville. Deep Learning. MIT Press.

Prerequisites:

• Programming in Python
• Already took NLP and machine learning courses
• Familiar with Probability and Statistics
• Research background in NLP or related fields; it's targeted for PhD students who work on related areas. Extensive research background is expected.
• Interests in languages and lingusitics
• Solid background in algorithms, probabilities
• Good programming skills
• Sufficient mathematical background

Class Policy

• Restricted Academic Integrity
• NO Incompleteness are accepted
• NO Late Assignments are accepted
• Don't come late to the class, if you are late more than 10 minutes, simply skip it in order not to disturb the lecture

Mailing List

• cs598-nlg-sp22@googlegroups.com