Dark and Bright Side of Participatory Red-Teaming with Targets of Stereotyping for Eliciting Harmful Behaviors from Large Language Models
Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, 2026 (CHI 2026)🏆 Honorable Mention
Abstract
Warning: This article contains stereotypical and offensive content.Red-teaming—where adversarial prompts are crafted to expose harmful behaviors and assess risks—offers a dynamic approach to surfacing underlying stereotypical bias in large language models. Because such subtle harms are best recognized by those with lived experience, involving targets of stereotyping as red-teamers is essential. However, critical challenges remain in leveraging their lived experience for red-teaming while safeguarding psychological well-being. We conducted an empirical study of participatory red-teaming with 20 individuals stigmatized by stereotypes against non-prestigious college graduates in South Korea’s rigid educational meritocracy. Through mixed-methods analysis, we found participants transformed experienced discrimination into strategic expertise for identifying biases, while facing psychological costs such as stress and negative reflections on group identity. Notably, red-team participation enhanced their sense of agency and empowerment through their role as guardians of the AI ecosystem. We discuss the implications for designing participatory red-teaming that prioritizes both the ethical treatment and the empowerment of stigmatized groups.
BibTeX
@inproceedings{10.1145/3772318.3790820,
author = {Kim, Sieun and Jo, Yeeun and Na, Sungmin and Lim, Hyunseung and Lee, Eunchae and Choi, Yu Min and Cho, Soohyun and Hong, Hwajung},
title = {Dark and Bright Side of Participatory Red-Teaming with Targets of Stereotyping for Eliciting Harmful Behaviors from Large Language Models},
year = {2026},
isbn = {9798400722783},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3772318.3790820},
doi = {10.1145/3772318.3790820},
abstract = {Warning: This article contains stereotypical and offensive content.Red-teaming—where adversarial prompts are crafted to expose harmful behaviors and assess risks—offers a dynamic approach to surfacing underlying stereotypical bias in large language models. Because such subtle harms are best recognized by those with lived experience, involving targets of stereotyping as red-teamers is essential. However, critical challenges remain in leveraging their lived experience for red-teaming while safeguarding psychological well-being. We conducted an empirical study of participatory red-teaming with 20 individuals stigmatized by stereotypes against non-prestigious college graduates in South Korea’s rigid educational meritocracy. Through mixed-methods analysis, we found participants transformed experienced discrimination into strategic expertise for identifying biases, while facing psychological costs such as stress and negative reflections on group identity. Notably, red-team participation enhanced their sense of agency and empowerment through their role as guardians of the AI ecosystem. We discuss the implications for designing participatory red-teaming that prioritizes both the ethical treatment and the empowerment of stigmatized groups.},
booktitle = {Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems},
articleno = {1389},
numpages = {20},
keywords = {generative artificial intelligence, participatory red-teaming, stereotype bias, mental health, AI safety},
location = {
},
series = {CHI '26}
}