I’m an Assistant Professor of Artificial Intelligence at Yonsei University, focusing on natural language processing, computer vision, and multimodal learning for AI agents. Before joining Yonsei, I was a researcher at the Allen Institute for AI working with Yejin Choi. I received my Ph.D. in Computer Science from Seoul National University, advised by Gunhee Kim.

My research focuses on curating web, simulation, and real-world datasets while advancing multimodal large language models to build AI systems that are not only capable of understanding but also socially responsible. Along the way, I’ve been honored with the Best Paper Award at NAACL 2022, Outstanding Paper Awards at EMNLP 2023 and ACL 2024, as well as the Okawa Foundation Research Grant.

Towards Visual Text Design Transfer Across Languages
Yejin Choi, Jiwan Chung, Sumin Shim, Giyeong Oh, Youngjae Yu
In NeurIPS 2024

Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!
Jiwan Chung, Seungwon Lim, Jaehyun Jeon, Seungbeen Lee, Youngjae Yu
In EMNLP 2024

ActionSwitch: Class-agnostic Detection of Simultaneous Actions in Streaming Videos
Hyolim Kang, Jeongseok Hyun, Joungbin An, Youngjae Yu, Seon Joo Kim
In ECCV 2024

CACTUS: Towards Psychological Counseling Conversations using Cognitive Behavioral Theory
Suyeon Lee, Sunghwan Kim, Minju Kim, Dongjin Kang, Dongil Yang, Harim Kim, Minseok Kang, Dayi Jung, Min Hee Kim, Seungbeen Lee, Kyoung-Mee Chung, Youngjae Yu, Dongha Lee, Jinyoung Yeo
In EMNLP 2024 (Findings)

How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models
Jaeyoung Lee, Ximing Lu, Jack Hessel, Faeze Brahman, Youngjae Yu, Yonatan Bisk, Yejin Choi, Saadia Gabriel
In EMNLP 2024 (Findings)

i-SRT: Aligning Large Multimodal Models for Videos by Iterative Self-Retrospective Judgment
Daechul Ahn, Yura Choi, San Kim, Youngjae Yu, Dongyeop Kang, Jonghyun Choi
In Arxiv

Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding
Jiwan Chung, Sungjae Lee, Minseo Kim, Seungju Han, Ashkan Yousefpour, Jack Hessel, Youngjae Yu
In EMNLP 2024

Scalp Diagnostic System With Label-Free Segmentation and Training-Free Image Translation
Youngmin Kim, Saejin Kim, Hoyeon Moon, Youngjae Yu, Junhyug Noh
In Arxiv

TRAIT: Personality Testset designed for LLMs with Psychometrics
Seungbeen Lee, Seungwon Lim, Seungju Han, Giyeong Oh, Jiwan Chung, Minju Kim, Yeonsoo Lee, Dongha Lee, Jinyoung Yeo, Youngjae Yu
In Arxiv

Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback
Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, Jonghyun Choi
In ACL 2024 (Oral)

Aligning Large Language Models by On-Policy Self-Judgment
Sangkyu Lee, Sungdong Kim, Ashkan Yousefpour, Minjoon Seo, Kang Min Yoo, Youngjae Yu
In ACL 2024

Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Dataset
Minjin Kim, Minju Kim, Hana Kim, Beong-woo Kwak, Soyeon Chun, Hyunseo Kim, SeongKu Kang, Youngjae Yu, Jinyoung Yeo, Dongha Lee
In ACL 2024

Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation
Dongjin Kang, Sunghwan Kim, Taeyoon Kwon, Seungjun Moon, Hyunsouk Cho, Youngjae Yu, Dongha Lee, Jinyoung Yeo
In ACL 2024 (Outstanding)

HyperCLOVA X Technical Report
Jiwan Chung, Sangkyu Lee, Youngjae Yu contributed
In Arxiv

Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models
Hyungjoo Chae, Yeonghyeon Kim, Seungone Kim, Kai Tzu-iunn Ong, Beong-woo Kwak, Seonghwan Kim, Taeyoon Kwon, Jiwan Chung, Youngjae Yu, Jinyoung Yeo
In EMNLP 2024

SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models
Hyun Lee, Kim Sung-Bin, Seungju Han, Youngjae Yu, Tae-Hyun Oh
In NAACL 2024

Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation
Shin-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard B W Yang, Giyeong Oh, Yanmin Gong, Youngjae Yu
In ICLR 2024

Localized Symbolic Knowledge Distillation for Visual Commonsense Models
Jae Sung Park, Jack Hessel, Khyathi Chandu, Paul Pu Liang, Ximing Lu, Youngjae Yu, Qiuyuan Huang, Peter West, Jianfeng Gao, Ali Farhadi, Yejin Choi
In NeurIPS 2023

Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms
Seungju Han, Junhyeok Kim, Jack Hessel, Liwei Jiang, Jiwan Chung, Yejin Son, Yejin Choi, Youngjae Yu
In EMNLP 2023 (Oral)

VLIS: Unimodal Language Models Guide Multimodal Language Generation
Jiwan Chung, Youngjae Yu
In EMNLP 2023

SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, Yejin Choi
In EMNLP 2023 (Outstanding)

Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents
Hyungjoo Chae, Yongho Song, Kai Tzu-iunn Ong, Taeyoon Kwon, Minjin Kim, Youngjae Yu, Dongha Lee, Dongyeop Kang, Jinyoung Yeo
In EMNLP 2023

Long Story Short: a Summarize-then-Search Method for Long Video Question Answering
Jiwan Chung, Youngjae Yu
In BMVC 2023

CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agents
Jeongeun Park, Seungwon Lim, Joonhyung Lee, Sangbeom Park, Minsuk Chang, Youngjae Yu, Sungjoon Choi
In RA-L/ICRA 2024

Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, Yejin Choi
In ACL 2023

Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Youngjae Yu, Ludwig Schmidt, William Yang Wang, Yejin Choi
In NeurIPS 2023

CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos
Seungju Han, Jack Hessel, Nouha Dziri, Yejin Choi, Youngjae Yu
In ICCV 2023

Fusing Pre-trained Language Models with Multimodal Prompts through Reinforcement Learning
Youngjae Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, Jae Sung Park, Ximing Lu, Prithviraj Ammanabrolu, Rowan Zellers, Ronan Le Bras, Gunhee Kim, Yejin Choi
In CVPR 2023

Zero-shot Active Visual Search (ZAVIS): Intelligent Object Search for Robotic Assistants
Jeongeun Park, Taerim Yoon, Jejoon Hong, Youngjae Yu, Matthew Pan, Sungjoon Choi
In ICRA 2023

PROSOCIALDIALOG: A Prosocial Backbone for Conversational Agents
Youngjae Yu *, Hyunwoo Kim *, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, Maarten Sap
In EMNLP 2022

MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, Yejin Choi
In CVPR 2022 (Oral) Link

Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer
Yanpeng Zhao, Jack Hessel, Youngjae Yu, Ximing Lu, Rowan Zellers, Yejin Choi
In NAACL 2022 Link

NeuroLogic A* esque Decoding: Constrained Text Generation with Lookahead Heuristics
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, Yejin Choi
In NAACL 2022 Link

MERLOT: Multimodal Neural Script Knowledge Models
Rowan Zellers*, Ximing Lu*, Jack Hessel*, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, Yejin Choi (* equal contribution)
In NeurIPS 2021 Link

Automatic Curation of Large-Scale Datasets for Audio-Visual Representation Learning
Sangho Lee*, Jiwan Chung*, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, Yale Song (* equal contribution)
In ICCV 2021 Link

Pano-AVQA: Grounded Audio-Visual Question Answering on 360° Videos
Heeseung Yun, Youngjae Yu, Wonsuk Yang, Gunhee Kim
In ICCV 2021 Link

Transitional Adaptation of Pretrained Models for Visual Storytelling
Youngjae Yu*, Jiwan Chung*, Jongseok Kim, Heeseung Yun, Gunhee Kim (* equal contribution)
In CVPR 2021 Link

Self-Supervised Learning of Compressed Video Representations
Youngjae Yu, Sangho Lee, Gunhee Kim, Yale Song
In ICLR 2021 Link

Parameter Efficient Multimodal Transformers for Video Representation Learning
Sangho Lee, Youngjae Yu, Thomas Breuel, Jan Kautz, Gunhee Kim, Yale Song
In ICLR 2021 Link

Dual Compositional Learning in Interactive Image Retrieval
Jongseok Kim, Youngjae Yu, Hoeseong Kim, Gunhee Kim
In AAAI 2021 Link

Character Grounding and Re-Identification in Story of Videos and Text Descriptions
Youngjae Yu, Jongseok Kim, Heeseung Yun, Jiwan Chung, Gunhee Kim
In ECCV 2020 (Spotlight) Link

A Joint Sequence Fusion Model for Video Question Answering and Retrieval
Youngjae Yu, Jongseok Kim, Gunhee Kim
In ECCV 2018 Link

A Memory Network Approach for Story-based Temporal Summarization of 360° Videos
Sangho Lee, Jinyoung Sung, Youngjae Yu, Gunhee Kim
In CVPR 2018 Link

A Deep Ranking Model for Spatio-Temporal Highlight Detection from a 360° Video
Youngjae Yu,Sangho Lee, Joonil Na, Jaeyun Kang, Gunhee Kim
In AAAI 2018 Link

Supervising Neural Attention Models for Video Captioning by Human Gaze Data
Youngjae Yu, Jongwook Choi, Yeonhwa Kim, Kyoung Yu, Sang-Hun Lee, Gunhee Kim
In CVPR 2017 Link

End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering
Youngjae Yu,Hyungjin Ko, Jongwook Choi, Gunhee Kim
In CVPR 2017 (Spotlight) Link

TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, Gunhee Kim
In CVPR 2017 (Spotlight) Link

International Journal

Video Question Answering with Spatio-Temporal Reasoning
Yunseok Jang, Yale Song, Chris Dongju Kim, Youngjae Yu, Youngjin Kim, Gunhee Kim
In IJCV 2019 Link

TimesVector: A Vectorized Clustering Approach to the Analysis of Time Series Transcriptome Data from Multiple Phenotypes
Inuk Jung, Kyuri Jo, Hyejin Kang, Hongryul Ahn, Youngjae Yu, Sun Kim
In Bioinformatics 2017 Link

Workshop

Cycled Compositional Learning between Images and Text
Jongseok Kim*, Youngjae Yu*, Seunghwan Lee, Gunhee Kim
In CVPR 2020, Third Workshop on Computer Vision for Fashion, Art and Design Link

Augmenting Data for Sarcasm Detection with Unlabeled Conversation Context
Hankyol Lee, Youngjae Yu, Gunhee Kim
In ACL 2020, Second Workshop on Figurative Language Processing Link

CurlingNet: Compositional Learning between Images and Text for Fashion IQ Data
Youngjae Yu, Seunghwan Lee, Yuncheol Choi, Gunhee Kim
In ICCV 2019, Second Workshop on Computer Vision for Fashion, Art and Design Link

Encoding Video and Label Priors for Multi-label Video Classification on YouTube-8M dataset
Seil Na, Youngjae Yu, Sangho Lee, Jisung Kim, Gunhee Kim
In CVPR 2017, YouTube-8M Large-Scale Video Understanding Link

Video Captioning and Retrieval Models with Semantic Attention
Youngjae Yu, Hyungjin Ko, Jongwook Choi, Gunhee Kim
In ECCV 2016, LSMDC & VisStory Link