Kevin Pu

PhD Candidate · Computer Science · University of Toronto

Advised by Prof. Tovi Grossman at the Dynamic Graphics Group and Prof. Yan Chen

My research interests lie in intersection of human-computer interaction and AI. My past works have focused on aligning AI support with human capabilities to build harmonious mixed-initiative interaction, enhancing human-AI collaboration workflows while balancing automation and human agency.

News

Apr 2026 I am traveling to CHI 2026 in Barcelona, Spain! Looking forward to connecting with folks!

Mar 2026 Cocoa has received a 🏆 Best Paper Award (top 1% of submissions)! Congratulations to first author Kevin Feng and all collaborators!

Feb 2026 I am visiting Bjoern Hartmann's lab at UC Berkeley from January to May 2026 as a visiting scholar! Looking forward to new collaborations and connections!

Jan 2026 One co-authored paper is conditionally accepted to CHI 2026! Congrats to Kevin Feng and other collaborators!

Sep 2025 Super excited to attend UIST 2025 in Busan! I will be presenting my Meta internship paper on proactive assistant in smart glasses and a co-authored paper on interactive co-design with AI!

Sep 2025 It was my pleasure to visit KAIST and give a research talk at the KIXLAB. Thank you for hosting me!

Jul 2025 Two papers are conditionally accepted to UIST 2025! Excited to present soon in Busan, South Korea!

May 2025 Invited to give a research talk at the VisLab at HKUST. Thank you for hosting me!

Apr 2025 Attending CHI 2025 in Yokohama, Japan! I will be presenting two papers on proactive programming tools and interactive research idea development!

Mar 2025 Wrapped up my internship at Meta Reality Labs and submitted to UIST 2025. Fingers crossed!

Jan 2025 Two papers conditionally accepted to CHI 2025!! See you in Yokohama, Japan!

Oct 2024 Started my internship as a Research Scientist intern at Meta Reality Labs in Toronto!

Oct 2024 Excited to serve as an SV again at UIST 2024 in Pittsburgh! Xiaohang is going to present our collaborative work VizGroup.

Sep 2024 Wrapped up my internship at Ai2 Semantic Scholars and submitted to CHI 2025_. Had a great time in Seattle thanks to my mentor Pao Siangliulue and other collaborators!

May 2024 Started my internship at Ai2 Semantic Scholars. Excited to spend my summer in Seattle!

May 2024 Attending CHI 2024 in Honolulu, Hawaii! One co-authored paper got accepted!

Oct 2023 Attending UIST 2023 at San Francisco as a Student Volunteer and presenting DiLogics!

Oct 2022 I will be traveling to Bend, Oregon to attend the UIST 2022 conference and present SemanticOn! SemanticOn received the 🏅 Honorable Mention Award at UIST 2022_!

Selected Publications

Show all →

Mixed-Initiative Interaction

Human-AI Collaboration

Neurosymbolic Program Synthesis

CHI 2026 🏆 Best Paper

Cocoa: Co-Planning and Co-Execution with AI Agents

K. J. Kevin Feng , Kevin Pu , Matt Latzke , Tal August , Pao Siangliulue , Jonathan Bragg , Daniel S. Weld , Amy X. Zhang , Joseph Chee Chang

As AI agents take on increasingly long-running tasks involving sophisticated planning and execution, there is a corresponding need for novel interaction designs that enable deeper human-agent collaboration. However, most prior works leverage human interaction to fix "autonomous" workflows that have yet to become fully autonomous or rigidly treat planning and execution as separate stages. Based on a formative study with 9 researchers using AI to support their work, we propose a design that affords greater flexibility in collaboration, so that users can 1) delegate agency to the user or agent via a collaborative plan where individual steps can be assigned; and 2) interleave planning and execution so that plans can adjust after partial execution. We introduce Cocoa, a system that takes design inspiration from computational notebooks to support complex research tasks. A lab study (n=16) found that Cocoa enabled steerability without sacrificing ease-of-use, and a week-long field deployment (n=7) showed how researchers collaborated with Cocoa to accomplish real-world tasks.

📄 PDF 📝 arXiv

Teaser figure for Cocoa: Co-Planning and Co-Execution with AI Agents

UIST 2025

ProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Modal Wearable Devices

Kevin Pu , Ting Zhang , Naveen Sendhilnathan , Sebastian Freitag , Raj Sodhi , Tanya R. Jonker

Wearable AI systems aim to provide timely assistance in daily life, but existing approaches often rely on user initiation or predefined task knowledge, neglecting users' current mental states. We introduce ProMemAssist, a smart glasses system that models a user's working memory (WM) in real-time using multi-modal sensor signals. Grounded in cognitive theories of WM, our system represents perceived information as memory items and episodes with encoding mechanisms, such as displacement and interference. This WM model informs a timing predictor that balances the value of assistance with the cost of interruption. In a user study with 12 participants completing cognitively demanding tasks, ProMemAssist delivered more selective assistance and received higher engagement compared to an LLM baseline system. Qualitative feedback highlights the benefits of WM modeling for nuanced, context-sensitive support, offering design implications for more attentive and user-aware proactive agents.

📄 PDF 🎬 Video 📝 arXiv

Teaser figure for ProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Modal Wearable Devices

UIST 2025

StoryEnsemble: Enabling Dynamic Exploration & Iteration in the Design Process with AI and Forward-Backward Propagation

Sangho Suh , Michael Lai , Kevin Pu , Steven P. Dow , Tovi Grossman

Design processes involve exploration, iteration, and movement across interconnected stages such as persona creation, problem framing, solution ideation, and prototyping. However, time and resource constraints often hinder designers from exploring broadly, collecting feedback, and revisiting earlier assumptions—making it difficult to uphold core design principles in practice. To better understand these challenges, we conducted a formative study with 15 participants—comprised of UX practitioners, students, and instructors. Based on the findings, we developed StoryEnsemble, a tool that integrates AI into a node-link interface and leverages forward and backward propagation to support dynamic exploration and iteration across the design process. A user study with 10 participants showed that StoryEnsemble enables rapid, multi-directional iteration and flexible navigation across design stages. This work advances our understanding of how AI can foster more iterative design practices by introducing novel interactions that make exploration and iteration more fluid, accessible, and engaging.

📄 PDF 🎬 Video

Teaser figure for StoryEnsemble: Enabling Dynamic Exploration & Iteration in the Design Process with AI and Forward-Backward Propagation

CHI 2025

Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Support

Kevin Pu , Daniel Lazaro , Ian Arawjo , Haijun Xia , Ziang Xiao , Tovi Grossman , Yan Chen

AI programming tools enable powerful code generation, and recent prototypes attempt to reduce user effort with proactive AI agents, but their impact on programming workflows remains unexplored. We introduce and evaluate Codellaborator, a design probe LLM agent that initiates programming assistance based on editor activities and task context. We explored three interface variants to assess trade-offs between increasingly salient AI support: prompt-only, proactive agent, and proactive agent with presence and context (Codellaborator). In a within-subject study (N = 18), we find that proactive agents increase efficiency compared to prompt-only paradigm, but also incur workflow disruptions. However, presence indicators and interaction context support alleviated disruptions and improved users' awareness of AI processes. We underscore trade-offs of Codellaborator on user control, ownership, and code understanding, emphasizing the need to adapt proactivity to programming processes. Our research contributes to the design exploration and evaluation of proactive AI systems, presenting design implications on AI-integrated programming workflow.

📄 PDF 🎬 Video 📝 arXiv

Teaser figure for Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Support

CHI 2025

IdeaSynth: Iterative Research Idea Development Through Evolving and Composing Idea Facets with Literature-Grounded Feedback

Kevin Pu , K. J. Kevin Feng , Tovi Grossman , Tom Hope , Bhavana Dalvi Mishra , Matt Latzke , Jonathan Bragg , Joseph Chee Chang , Pao Siangliulue

Research ideation involves broad exploring and deep refining ideas. Both require deep engagement with literature. Existing tools focus primarily on broad idea generation, yet offer little support for iterative specification, refinement, and evaluation needed to further develop initial ideas. To bridge this gap, we introduce IdeaSynth, a research idea development system that uses LLMs to provide literature-grounded feedback for articulating research problems, solutions, evaluations, and contributions. IdeaSynth represents these idea facets as nodes on a canvas, and allow researchers to iteratively refine them by creating and exploring variations and combinations. Our lab study (N = 20) showed that participants, while using IdeaSynth, explored more alternative ideas and expanded initial ideas with more details compared to a strong LLM-based baseline. Our deployment study (N = 7) demonstrated that participants effectively used IdeaSynth for real-world research projects at various ideation stages from developing initial ideas to revising framings of mature manuscripts, highlighting the possibilities to adopt IdeaSynth in researcher's workflows.

📄 PDF 🎬 Video 📝 arXiv

Teaser figure for IdeaSynth: Iterative Research Idea Development Through Evolving and Composing Idea Facets with Literature-Grounded Feedback

UIST 2024

VizGroup: An AI-assisted Event-driven System for Collaborative Programming Learning Analytics

Xiaohang Tang , Sam Wong , Kevin Pu , Xi Chen , Yalong Yang , Yan Chen

Programming instructors often conduct collaborative learning activities, like Peer Instruction, to foster a deeper understanding in students and enhance their engagement with learning. These activities, however, may not always yield productive outcomes due to the diversity of student mental models and their ineffective collaboration. In this work, we introduce VizGroup, an AI-assisted system that enables programming instructors to easily oversee students' real-time collaborative learning behaviors during large programming courses. VizGroup leverages Large Language Models (LLMs) to recommend event specifications for instructors so that they can simultaneously track and receive alerts about key correlation patterns between various collaboration metrics and ongoing coding tasks. We evaluated VizGroup with 12 instructors in a comparison study using a dataset collected from a Peer Instruction activity that was conducted in a large programming lecture. The results showed that VizGroup helped instructors effectively overview, narrow down, and track nuances throughout students' behaviors.

📄 PDF 🎬 Video

Teaser figure for VizGroup: An AI-assisted Event-driven System for Collaborative Programming Learning Analytics

CHI 2024

Behind the Pup-ularity Curtain: Understanding the Motivations, Challenges, and Work Performed in Creating and Managing Pet Influencer Accounts

Suhyeon Yoo , Kevin Pu , Khai N. Truong

Creating dedicated accounts to post users' pet content is a growing trend on Instagram. While these account owners derive joy from this pursuit, they may also struggle with criticisms and challenges. Yet, there remains a knowledge gap on how pet account owners manage their pets' online presence and navigate these obstacles successfully. Drawing from interviews with 21 Instagram pet account owners, we uncover the motivations behind pet account creation, spanning personal, altruistic, and commercial goals. We learn about the strategies employed for crafting their pets' online identities and personas, as well as the challenges faced by both owners and their pets in navigating the complexities of digital identity management. We discuss the evolving dynamics between humans and their pets, positioning pet identity cultivation as a form of collaborative work, akin to the "third shift", highlighting the need to design interfaces that support this unique identity management process.

📄 PDF

Teaser figure for Behind the Pup-ularity Curtain: Understanding the Motivations, Challenges, and Work Performed in Creating and Managing Pet Influencer Accounts

UIST 2023

DiLogics: Creating Web Automation Programs with Diverse Logics

Kevin Pu , Jim Yang , Angel Yuan , Minyi Ma , Rui Dong , Xinyu Wang , Yan Chen , Tovi Grossman

Knowledge workers frequently encounter repetitive web data entry tasks, like updating records or placing orders. Web automation increases productivity, but translating tasks to web actions accurately and extending to new specifications is challenging. Existing tools can automate tasks that perform the same logical trace of UI actions (e.g., input text in each field in order), but do not support tasks requiring different executions based on varied input conditions. We present DiLogics, a programming-by-demonstration system that utilizes NLP to assist users in creating web automation programs that handle diverse specifications. DiLogics first semantically segments input data to structured task steps. By recording user demonstrations for each step, DiLogics generalizes the web macros to novel but semantically similar task requirements. Our evaluation showed that non-experts can effectively use DiLogics to create automation programs that fulfill diverse input instructions. DiLogics provides an efficient, intuitive, and expressive method for developing web automation programs satisfying diverse specifications.

📄 PDF 🎬 Video

Teaser figure for DiLogics: Creating Web Automation Programs with Diverse Logics

UIST 2022 Best Paper Honorable Mention

SemanticOn: Specifying Content-Based Semantic Conditions for Web Automation Programs

Kevin Pu , Rainey Fu , Rui Dong , Xinyu Wang , Yan Chen , Tovi Grossman

Data scientists, researchers, and clerks often create web automation programs to perform repetitive yet essential tasks, such as data scraping and data entry. However, existing web automation systems lack mechanisms for defining conditional behaviors where the system can intelligently filter candidate content based on semantic filters (e.g., extract texts based on key ideas or images based on entity relationships). We introduce SemanticOn, a system that enables users to specify, refine, and incorporate visual and textual semantic conditions in web automation programs via two methods: natural language description via prompts or information highlighting. Users can coordinate with SemanticOn to refine the conditions as the program continuously executes or reclaim manual control to repair errors. In a user study, participants completed a series of conditional web automation tasks. They reported that SemanticOn helped them effectively express and refine their semantic intent by utilizing visual and textual conditions.

📄 PDF 🎬 Video

Teaser figure for SemanticOn: Specifying Content-Based Semantic Conditions for Web Automation Programs