Course at a glance
- Format
- In-person, participation-driven research seminar with paired seminal/frontier readings and a team research project.
- Reading model
- Each research session connects a foundational paper to a modern frontier paper; the frontier paper is normally the core review paper.
- Assessment
- 15% participation · 20% weekly reviews · 15% paired-paper presentation/discussion · 50% research project.
- Project verification
- Code/audit packet freezes Nov. 17; rebuttal and TA project/code interview occur before final project evaluation.
Instructional team
Instructor & course staff
Course design
Foundations first. Frontier next.
The course follows the evolution from representation learning and multimodal fusion toward models that reason, use tools, control interfaces, and act in the physical world.
What we will study
Multimodal foundation models now span far more than image captioning and VQA. We will connect the architectural and training ideas that made modern VLMs possible to current work in reasoning, grounding, video, agents, autonomous driving, robotics, and predictive world models.
- Vision and vision-language representation learning
- VLM architectures, instruction tuning, unified understanding/generation
- Grounding, hallucination, spatial/3D and temporal reasoning
- Web agents, GUI/computer-use agents, memory and agent reliability
- Autonomous driving, VLAs, real-time control, and world-action models
The paired-paper format
Each research meeting has two anchors. Presenters should teach both papers as one intellectual story—not as two disconnected summaries.
- Seminal anchor: the idea, abstraction, or architecture that established the line of work.
- Frontier anchor: a recent paper that changes the scale, objective, architecture, evaluation, or deployment setting.
- Discussion should explicitly ask: what persisted, what changed, and why?
- The frontier anchor is the default core-review paper unless Canvas says otherwise.
Living syllabusBecause this area moves quickly, a frontier anchor may be replaced by a major new result during the semester. Changes will be announced on Ed/Canvas.
- Recommended background
- Prior coursework or research experience in machine learning and deep learning is expected, including familiarity with Transformers. Background in computer vision, NLP, robotics, or another relevant modality is useful, but deep expertise in every modality is not required.
- Course materials
- There is no required textbook. Readings, lecture slides where applicable, assignment templates/rubrics, and supporting resources will be linked from the course page or Canvas.
- Course staff / office hours
- Teaching assistants: Junhyun Kim and Brisa Maneechotesuwan. Office hours and course-specific contact channels will be posted on Canvas once the Fall 2026 course shell is finalized.
Fall 2026
Schedule & paper pairings
The schedule is the single source of truth for class meetings, paper pairings, presenters, graded deliverables, and required checkpoints. Hover or focus a colored requirement label for details.
Week
Date
Topic
Seminal anchor
Frontier anchor · core review
Presenters
Requirements
Ongoing
Everyclass / assigned session
↻
Recurring course requirementsWeekly reviews, one paired-paper presentation/discussion, and active class participation continue throughout the semester.
W1
Aug 25Tuesday
01
Introduction, logistics & the VLM → agent arcCourse framing, paper-review workshop, project examples, and presentation sign-up process.
Zsolt Kira
W1
Aug 27Thursday
Transformers & multimodal tokenizationAttention, token interfaces, spatial/temporal position; VLM landscape, alignment, and evaluation.Foundations
Zsolt Kira
W2
Aug 31Monday · deadline
P
Paper presentation sign-upChoose your paired-paper presentation/discussion slot on the sign-up sheet.
Sign-up · RequiredSign-up sheet
W2
Sep 1Tuesday
Vision encoders as foundation modelsPatch tokens, scaling, self-supervision, dense visual features.Foundations
Zsolt Kira
W2
Sep 3Thursday
Vision–language alignment & open-vocabulary representationsContrastive learning, retrieval, zero-shot transfer, multilingual and dense alignment.Foundations
W3
Sep 8Tuesday
From frozen backbones to large VLMsCross-attention, resamplers, adapters, scaling, visual resolution routing.VLM architectures
W3
Sep 10Thursday
Instruction-tuned VLMs from pretrained backbonesVisual instruction tuning, data mixtures, native multimodal pretraining, long-context image/video.VLM architectures
Team roster · Required
W4
Sep 15Tuesday
Open-vocabulary detection, grounding & segmentationLanguage-conditioned localization from boxes to dense masks.Grounding
W4
Sep 17Thursday
P
Project proposal presentationsTeam proposal presentations; detailed expectations are in the requirement marker.
Proposal · 5%Template
W5
Sep 22Tuesday
Unified multimodal understanding + generationOne model for perception, generation, editing, interleaved sequences, and emerging world-model behavior.Unified models
W5
Sep 24Thursday
Grounded image generation & editingSpatial conditioning, controllability, context-aware generation and editing.Generation
W6
Sep 29Tuesday
Video VLMs & spatiotemporal reasoningTemporal attention, frame selection, latent temporal reasoning, long-video constraints.Video
W6
Oct 1Thursday
Multimodal chain-of-thought & reasoning modelsReasoning supervision, perception-reasoning coupling, compact open reasoning models.Reasoning
W7
Oct 6Tuesday
—
Fall Break — no classGeorgia Tech Fall Break is Oct. 5–6, 2026.
W7
Oct 8Thursday
Post-training & RL for multimodal reasoningOutcome-based RL and GRPO, emergent reasoning behavior, and perception-aware objectives unique to multimodal RL.Reasoning
W8
Oct 13Tuesday
Hallucination, grounding & visual reliabilityLanguage-prior and temporal hallucination, visual illusion, attention/grounding diagnostics, mitigation.Reliability
W8
Oct 15Thursday
M
Project midterm presentationsProject midterm presentations; detailed expectations are in the requirement marker.
W9
Oct 20Tuesday
Spatial / 3D reasoning & geometric structureMetric spatial understanding, reconstructive supervision, scene structure, and diagnosing visual-spatial intelligence.Geometry
W9
Oct 22Thursday
Multimodal tool use & modular agentsProgram synthesis, visual modules, tool routing, GUI + code hybrid execution.Agents
W10
Oct 27Tuesday
Web agents: from structured pages to visual action policiesBrowser tasks, screenshot-only control, online evaluation, open web-agent data.Web agents
W10
Oct 29Thursday
Computer-use / GUI agentsScreenshot grounding, mouse/keyboard action spaces, long-horizon desktop workflows, realistic evaluation.Computer use
W11
Nov 3Tuesday
Agent memory, planning & safe computer useReason+act loops, visual memory, recovery, observation/action mismatch, safety-aware execution.Agent reliability
W11
Nov 5Thursday
Autonomous driving as a multimodal decision problemLanguage reasoning, trajectory planning, dense 3D geometry, action-grounded driving evaluation.Driving
W12
Nov 10Tuesday
Embodied VLMs → Vision-Language-Action modelsGrounding language in robot state/action, embodied reasoning, continuous control.VLA
W12
Nov 12Thursday
Generalist VLAs, action representations & data scalingLatent actions from video, cross-embodiment data, action representations, massive real-world trajectory pretraining.VLA scaling
W13
Nov 17Tuesday
Fast, continuous VLA controlDiffusion/flow policies, action chunks, asynchronous inference, reaction latency.Real-time VLA
W13
Nov 19Thursday
World models + action: World Action ModelsPredictive imagination, latent dynamics, future states, action-conditioned world modeling.World-action models
W13
Nov 20Friday · returned
A
AI-assisted project audit/review returnedTeams receive the implementation/correctness audit and research critique to evaluate before rebuttal.
Audit returned · Review input
W14
Nov 24Tuesday
↗
Project clinic + rebuttal dueProject clinic; evidence-based rebuttal due by 11:59 PM ET.
W14
Nov 26Thursday
—
Thanksgiving — no classInstitute holiday / Thanksgiving recess.
W15
Nov 30–Dec 4TA interview window
I
TA project/code interviewsVerification interviews occur during this window; individual scheduling details will be posted on Canvas.
Interview · Verification gate
W15
Dec 1Tuesday
F
Final project presentations · IFinal project presentations; each team presents once on its assigned date.
Final presentation · 10%Rubric
W15
Dec 3Thursday
F
Final project presentations · IIFinal project presentations; each team presents once on its assigned date.
Final presentation · 10%Rubric
W16
Dec 8Tuesday
F
Final project presentations · III + course synthesisFinal project presentations; each team presents once on its assigned date. Course synthesis follows presentations.
Final presentation · 10%Rubric
Final
Canvasdeadline
R
Final project reportSubmit the conference-style final report by the deadline posted on Canvas.
Final report · 15%Template
Grading scale
The standard letter-grade scale will be used; no curve is planned. Borderline totals may be rounded at the instructor’s discretion.
| Grade | Course total |
|---|---|
| A | 90% and above |
| B | 80%–89% |
| C | 70%–79% |
| D | 60%–69% |
| F | Below 60% |
Course logistics & policies
Communication, conduct, attendance, integrity, and support
These policies restore the logistical guidance from prior offerings while updating it for the Fall 2026 project and AI-use workflow. Georgia Tech policy and approved accommodations supersede this summary when applicable.
Course format, attendance, and participation
This is a participation-driven, in-person research seminar; there is no routine remote option. Regular attendance, preparation, and engagement are part of the 15% participation grade. If you are ill, do not attend in person. Contact the teaching team as soon as possible, particularly if you will miss a presentation, project interview, or other scheduled responsibility. Institute-approved absences and verified emergencies will be handled consistent with Georgia Tech policy.
Communication policy
Students are responsible for information posted on this course page and Canvas, and for announcements sent through Ed Discussion or Georgia Tech email. Check Georgia Tech email and course announcements at least once each school day. Unless an announcement states otherwise, students will not be held responsible for time-sensitive changes until 24 hours after the message is sent. Use Ed for questions that may help the class; use private messages/email for individual matters.
Late and make-up work
Deadlines are firm unless an assignment explicitly states a late policy. Weekly paper reviews are due at 11:59 PM ET the day before class and late reviews are not accepted; the lowest three eligible review scores are dropped in part to absorb ordinary conflicts. For illness, family emergency, or another significant circumstance, notify the teaching team as early as possible and use the Dean of Students Request Assistance process when verification or coordinated faculty notification is needed. Do not send medical records or other sensitive documentation to the instructor or TAs. Approved accommodations, Institute Approved Absences, and verified emergency/illness requests will be honored under Georgia Tech policy.
Academic integrity, plagiarism, collaboration, and AI use
All work is governed by the Georgia Tech Academic Honor Code. Discussion and scientific collaboration are encouraged, but submitted work must follow the assignment-specific authorship rules and all borrowed text, figures, code, ideas, datasets, models, and other resources must be appropriately cited or attributed.
- Weekly paper reviews: the review must be your own reading and synthesis. Generative AI/LLM assistance may not be used to summarize, draft, rewrite, critique, or otherwise produce the review.
- Paired-paper presentations and discussion materials: presenters may discuss the papers with others and use cited source material, but submitted slides, synthesis, and discussion questions must be the presenters’ own work.
- Research project implementation: external codebases, pretrained models, tools, and AI coding assistants/agents may be used when permitted, but must be attributed. Material AI/agent use must be documented in the repository-root
AGENT_USAGE.md, including what was used, for what purpose, what files/experiments were materially affected, and how the output was validated. - Project ownership: the research question/contribution, experimental decisions, execution, interpretation, and understanding must be the team’s own. Students remain responsible for correctness, licensing, citations, and all submitted material. The code interview is used to verify authorship, contribution, and understanding.
Suspected academic-integrity violations may be referred to the Office of Student Integrity in accordance with Institute procedures.
Professional and online conduct
Discussion should be rigorous, collegial, and respectful. Read existing Ed threads before posting, avoid language that is needlessly hostile or likely to be misread, protect classmates’ privacy, and do not post or distribute another student’s personal information or work without permission. Course staff may moderate discussion posts that are off-topic, inappropriate, or inconsistent with a respectful academic environment.
Disability accommodations, illness, and emergencies
Students who require disability-related accommodations should work with the Office of Disability Services and provide the appropriate accommodation notice. Students affected by significant illness or personal emergencies may use the Dean of Students Request Assistance process. Please notify the instructor that you are working through the appropriate office, but do not send personal medical details or documentation directly to course staff.
Mental health and student support
Georgia Tech’s Center for Mental Health Care & Resources provides counseling, crisis support, assessment, referral, workshops, and related services for students. For urgent mental-health support, students may contact the Center at 404-894-2575; after hours, the same number provides an after-hours counselor option. In an immediate emergency, call 911 or Georgia Tech Police at 404-894-2500. The national 988 Suicide & Crisis Lifeline is available by calling or texting 988.
Student–faculty expectations
This course follows Georgia Tech’s Student–Faculty Expectations. Students should expect a respectful and engaged academic environment, regularly scheduled instruction, clear course requirements, and reasonable communication and feedback. The instructional staff expects students to attend regularly and on time, prepare for discussion, meet stated deadlines, communicate professionally, and contribute to a respectful academic environment.
Final assessment and final instructional days
There is no traditional final exam. The research project serves as the major culminating assessment. Project interviews, final presentations, and the final report occur near the end of the semester as listed in the schedule. Any change to final-assessment logistics will be announced through the normal course communication channels.
Subject to change
Because this is a seminar in a rapidly evolving research area, the reading list and detailed schedule may change to incorporate important new work. Changes to papers, deadlines, or course logistics will be communicated through the course page and Ed/Canvas announcements. The published course page and Canvas assignment pages are authoritative when a development draft or template differs.
Resources
Useful links
Instructor homepage ↗Research, publications, RIPL, and contact information.GT Academic Calendar ↗Official semester dates and Institute holidays.Academic Honor Code ↗Institute academic-integrity requirements.Student–Faculty Expectations ↗Georgia Tech expectations for the academic environment.Dean of Students — Request Assistance ↗Emergency, illness, and personal-assistance process.Office of Disability Services ↗Accommodation process and student resources.Center for Mental Health Care & Resources ↗Counseling, crisis support, referral, and mental-health resources.CS231n ↗Computer-vision background.CS224n ↗NLP and Transformer background.Course AI-use disclosureHow AI assisted the Fall 2026 course-material redesign; student projects maintain their own separate AGENT_USAGE.md.