CSE-895: Selected Topics on Large Language Models

News

Course Description

"Selected Topics on Large Language Models" is a graduate-level course designed to explore the technical foundations of both statistical and neural language modeling, with a primary focus on the latest generation of large language models. These models have significantly impacted not only the field of natural language processing but also various AI domains, contributing to the development of intelligent systems capable of processing multiple modalities such as language, vision, speech, and code. The course begins by delving into the classical concept of language modeling based on n-grams and statistical modeling. Subsequently, it covers distributed word representations and extends these to context-dependent representations. The basics of attention mechanism and transformers are discussed, leading to an exploration of encoder-based language models, including pre-training and fine-tuning paradigms. The course also addresses encoder-decoder and generative language models, touching on concepts like prompt engineering, soft prompt learning, and the emergence of in-context learning in generative auto-regressive language models. After establishing these foundational concepts, the course delves into state-of-the-art literature that leverages language models to address various AI challenges, with a particular emphasis on multi-modal language models. The analysis covers the capabilities of Large Language Models (LLMs) from both theoretical and experimental perspectives, examining different facets of intelligence, including various types of reasoning such as mathematical reasoning, compositional reasoning, and reasoning on spatial and temporal information. As part of the course, students engage in a hands-on project involving a variety of language models, including members of the BERT family and older generative models like T5 and GPT-2. We exercise with new versions of LLaMA, GPT and mutimodal ones such as LLaVA and Qwen-VL. Computational resources are provided specific to this class to facilitate the successful completion of the project for the students.

Topics to be covered

1. Basics of statistical language modeling, n-grams, next-word prediction, evaluation metrics.
2. Representation Learning: Word representations, Distributed Representations, Contextualized Language representations.
3. Encoder-decoder models, attention mechanism, transformer architecture.
4. Encoder-based Language models, pre-training, fine-tuning.
5. Encoder-decoder (Generative) Language models, prompt engineering, soft-prompt learning.
6. In-context learning, auto-regressive decoding.
7. Application of language models in solving real-world problems, QA systems, image and video captioning, code generation, virtual assistants, search and information retrieval.
8. Capabilities of language models in capturing linguistic information: syntax and semantics.
9. Reasoning capabilities of language models, commonsense reasoning, mathematical reasoning, compositional reasoning, spatial and temporal reasoning.
10. Multi-modal language models capturing semantics of vision, speech, language and code.
11. Fairness and bias in language models and other open challenges.

Course Evaluation

• Attendance is required
• Project 35%
• Seminar 15%
• Homework including coding assignments 15%
• Quizzes 15%
• Paper-reading, reviews and participation discussions 20%

Grading scheme:

90-100 => 4; 85-90 => 3.5; 80-85=> 3; 75-80 => 2.5; 70-75 => 2; 65-70=> 1.5; 59-65=> 1; Total<58 => Fail/0.

Suggested Paper Reading

• TBD

General Natural Language Processing References (text books)

• Jurafsky and Martin: Speech and Language Processing (2nd Edition), Prentice Hall (2008), Amazon; 3rd Edition (free draft).
• Yoav Goldberg: Neural Network Methods for NLP, Morgan and Claypool (2017), Online version.
• Manning and Schuetze, Foundations of Statistical Natural Language Processing, MIT Press (1999), Online version.
• We will look into research papers and those will be posted during the semester.

Prerequisites (at least two items of below list should be fulfilled)

• An introductory machine learning course
• An introductory deep learning course
• Knowledge about the basics of probability theory and linear algebra
• Programming skills, the project and assignments will require a working familiarity with Python.
• An introductory natural language processing course

Academic Honesty

Students are responsible for adhering to the MSU Academic Honesty (see link) policy and violation will result in a grade of 0 for the course. Furthermore, copying, paraphrasing, or plagiarizing someone else's work, including their code, or allowing your own work to be copied or paraphrased, even if only in part, is not allowed and will result in an automatic grade of 0 for the copied assignment. Using websites that allow you to get help from or pay someone else to write your code or solve the homework assignments are also prohibited and will result in a grade of 0 for the assignment. If students collaborate on an assignment, i.e., discuss the assignment in any way, the names of students and what was discussed must be included in the homework write-up.

Acknowledgment