Auditing GPT's Content Moderation Guardrails: Can ChatGPT Write Your Favorite TV Show?

Large language models (LLMs) are increasingly appearing in consumer-facing products. To prevent problematic use, the organizations behind these systems have put content moderation guardrails in place that prevent the models from generating content they consider harmful. However, most of these enforcement standards and processes are opaque. Although they play a major role in the user experience of these tools, automated content moderation tools have received relatively less attention than other aspects of the models. This study undertakes an algorithm audit of OpenAI’s ChatGPT with the goal of better understanding its content moderation guardrails and their potential biases. To evaluate performance on a broad cultural range of content, we generate a dataset of 100 popular United States television shows with one to three synopses for each episode in the first season of each show (3,309 total synopses). We probe GPT’s content moderation endpoint (ME) to identify violating content both in the synopses themselves, and in GPT’s own outputs when asked to generate a script based on each synopsis, also comparing with ME outputs on 81 real scripts from the same TV shows (269,578 total ME outputs). Our findings show that a large number of GPT-generated and real scripts flag as content violations (about 18% of GPT scripts and 69% of real ones). Using metadata, we find that TV maturity ratings, as well as certain genres (Animation, Crime, Fantasy, and others) are statistically significantly related to a script’s likelihood of flagging. We conclude by discussing the implications of LLM self-censorship and directions for future research on their moderation procedures.