No required textbook. This field moves faster than textbooks do. Every reading is a research paper, linked directly from the schedule below: two required papers and one optional paper per lecture.
If you have never built or studied an AI agent before, these two will get you to the starting line:
This course requires a basic understanding of machine learning:
Your final grade for this course will be based on the following scheme:
| Date | Mode | Topics | Due | Objectives and Materials |
|---|---|---|---|---|
| Part I: Foundations of Agentic AI Systems | ||||
| Wed. 09/23 |
In-person | Introduction | Sign-up opens |
What turns an LLM into an agent? The agent loop, reasoning, planning, tool use. (Required) ReAct: Synergizing Reasoning and Acting in Language Models (Required) Toolformer: LMs Can Teach Themselves to Use Tools (Optional) Tree of Thoughts: Deliberate Problem Solving with LLMs |
| Mon. 09/28 |
In-person | Agent Architectures | - |
Retrieval, short- and long-term memory, reflection, single- vs. multi-agent systems, delegated actions. (Required) Generative Agents: Interactive Simulacra of Human Behavior (Required) Reflexion: Language Agents with Verbal RL (Optional) AgentBench: Evaluating LLMs as Agents |
| Part II: Threat Modeling Agentic AI Systems | ||||
| Wed. 09/30 |
In-person | Threat Modeling I |
Team sign-up Disc. sign-up |
Assets, adversaries, trust boundaries, untrusted observations, agent privileges. (Required) Agent Security Bench (ASB) (Required) Identifying the Risks of LM Agents with an LM-Emulated Sandbox (ToolEmu) (Optional) AI Agents Under Threat: A Survey of Key Security Challenges |
| Mon. 10/05 |
In-person | Threat Modeling II | - |
Delegated authority, persistent state, security properties, unsafe actions vs. incorrect outputs. (Required) AgentHarm: Measuring Harmfulness of LLM Agents (Required) Attacking Vision-Language Computer Agents via Pop-ups (Optional) BadChain: Backdoor Chain-of-Thought Prompting |
| Part III: Attacks on Agentic AI Systems | ||||
| Wed. 10/07 |
In-person | Prompt Injection I | - |
Instruction/data confusion, goal hijacking, direct injection. (Required) Not What You've Signed Up For: Indirect Prompt Injection (Required) Formalizing and Benchmarking Prompt Injection Attacks and Defenses (Optional) Defending Against Indirect Prompt Injection with Spotlighting |
| Mon. 10/12 |
In-person | Prompt Injection II | - |
Indirect and environmental attacks, retrieval attacks, data exfiltration, adaptive attacks. (Required) InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Agents (Required) AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses (Optional) EIA: Environmental Injection Attack on Generalist Web Agents |
| Wed. 10/14 |
No class | - | Project prep | No class. Use this session to prepare Checkpoint Presentation 1. |
| Mon. 10/19 |
In-person | Group Project | Slides | Checkpoint Presentation 1 — 10 min + 5 min Q&A per team |
| Wed. 10/21 |
In-person | Tools, Privileges, and Authorization | - |
Tool invocation, capabilities, excessive agency, over-privileged agents, authentication vs. authorization, confused-deputy problems, least privilege, irreversible actions, human approval. (Required) Model Context Protocol (MCP): Landscape and Security Threats (Required) Defeating Prompt Injections by Design (CaMeL) (Optional) AdvAgent: Controllable Blackbox Red-teaming on Web Agents |
| Mon. 10/26 |
In-person | Memory, State, and Persistent Attacks | - |
Long-term memory and retrieval corpora, memory poisoning, persistent compromise across sessions, provenance, recovery. (Required) AgentPoison: Red-Teaming LLM Agents via Poisoning Memory or Knowledge Bases (Required) Watch Out for Your Agents! Backdoor Threats to LLM-Based Agents (Optional) PoisonedRAG: Knowledge Corruption Attacks to RAG |
| Wed. 10/28 |
In-person | Multi-Agent Security | - |
Agent-to-agent communication, identity, delegation, transitive trust, cascading failure, containment. Unauthorized communication vs. malicious behavior by an authorized agent. (Required) Agent Smith: A Single Image Can Jailbreak One Million Multimodal Agents (Required) On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents (Optional) Flooding Spread of Manipulated Knowledge in LLM Multi-Agent Communities |
| Part IV: Defending Agentic AI Systems | ||||
| Mon. 11/02 |
In-person | Defenses I: Architecture | - |
Where should enforcement live? Isolation, sandboxing, tool mediation, information-flow restriction. (Required) IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems (Required) Firewalls to Secure Dynamic LLM Agentic Networks (Optional) Securing AI Agents with Information-Flow Control |
| Wed. 11/04 |
In-person | Defenses II: Model and Runtime | - |
Training-time hardening, runtime monitors, network-level mediation. Prevention vs. detection vs. containment. (Required) StruQ: Defending Against Prompt Injection with Structured Queries (Required) SecAlign: Defending Against Prompt Injection with Preference Optimization (Optional) Jatmo: Prompt Injection Defense by Task-Specific Finetuning |
| Part V: Evaluation, Oversight, and Frontier Problems | ||||
| Mon. 11/09 |
In-person | Evaluating Agent Security | - |
Threat-model validity, attack success rate, task utility, the security–utility tradeoff, adaptive attacks, benchmark realism, reproducibility. (Required) Agent Security Bench (ASB) (Revisit) (Required) Dissecting Adversarial Robustness of Multimodal LM Agents (Optional) AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses (revisit: read the defense evaluation, not the attacks) |
| Wed. 11/11 |
No class | - | - | Veterans Day |
| Mon. 11/16 |
In-person | Group Project | Slides | Checkpoint Presentation 2 — 10 min + 5 min Q&A per team |
| Wed. 11/18 |
In-person | Human Oversight and Agentic-System Literacy | - |
Human-in-the-loop vs. on-the-loop, situational awareness, calibrated trust, automation bias, escalation, auditability. (Required) Guidelines for Human-AI Interaction (Required) Visibility into AI Agents (Optional) Trust in Automation: Designing for Appropriate Reliance |
| Mon. 11/23 |
In-person | Frontier Problems in Secure Agentic AI | - |
Long-horizon autonomy, agent identity, self-modifying agents, machine-to-machine trust, agent ecosystems, oversight at scale, open questions. (Required) AI Control: Improving Safety Despite Intentional Subversion (Required) Infrastructure for AI Agents (Optional) Multi-Agent Risks from Advanced AI |
| Wed. 11/25 |
No class | - | - | Thanksgiving Break |
| Mon. 11/30 |
No class | Project prep | - | No class. Use this session to prepare the Final Presentation. |
| Wed. 12/02 |
In-person | Group Project | Slides | Final Presentation — 15 min + 5 min Q&A per team. |
| Finals Week (12/07 – 12/11) | ||||
| Mon. 12/07 – Wed. 12/09 |
Online | Final Exam | - | - |
| Fri. 12/11 |
No class | Project report due | - | Submit the final report on Canvas by 6:00 pm [Details]. |