Publications

You can also find my articles on my Google Scholar profile.

Peer-reviewed publications

Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI

International AAAI Conference on Web and Social Media (ICWSM), 2025

Automated text annotation is a compelling use case for generative large language models (LLMs) in social media re-search. Recent work suggests that LLMs can achieve strongperformance on annotation tasks; however, these studies evaluate LLMs on a small number of tasks and likely suffer from contamination due to a reliance on public benchmark datasets. Here, we test a human-centered frameworkfor responsibly evaluating artificial intelligence tools usedin automated annotation. We use GPT-4 to replicate 27 annotation tasks across 11 password-protected datasets fromrecently published computational social science articles in high-impact journals. For each task, we compare GPT-4 an-notations against human-annotated ground-truth labels andagainst annotations from separate supervised classificationmodels fine-tuned on human-generated labels. Although thequality of LLM labels is generally high, we find significant variation in LLM performance across tasks, even withindatasets. Our findings underscore the importance of a human-centered workflow and careful evaluation standards: Automated annotations significantly diverge from human judgment in numerous scenarios, despite various optimizationstrategies such as prompt tuning. Grounding automated annotation in validation labels generated by humans is essentialfor responsible evaluation.

Nicholas Pangakis and Samuel Wolken (2024). Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI. Proceedings of the International AAAI Conference on Web and Social Media.

The Rise of and Demand for Identity-Oriented Media Coverage

American Journal of Political Science, 2024

While some assert that social identities have become more salient in American media coverage, existing evidence is largely anecdotal. An increased emphasis on social identities has important political implications, including for polarization and representation. We first document the rising salience of different social identities using natural language processing tools to analyze all tweets from 19 media outlets (2008–2021) alongside 553,078 URLs shared on Facebook. We then examine one potential mechanism: Outlets may highlight meaningful social identities—race/ethnicity, gender, religion, or partisanship—to attract readers through various social and psychological pathways. We find that identity cues are associated with increases in some forms of engagement on social media. To probe causality, we analyze 3,828 randomized headline experiments conducted via Upworthy. Headlines mentioning racial/ethnic identities generated more engagement than headlines that did not, with suggestive evidence for other identities. Identity-oriented media coverage is growing and rooted partly in audience demand.

Hopkins, D. J., Lelkes, Y., & Wolken, S. (2024). The rise of and demand for identity‐oriented media coverage. American Journal of Political Science.
Download Paper

Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels

Proceedings of the Sixth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS), 2024

Computational social science (CSS) practitioners often rely on human-labeled data to fine-tune supervised text classifiers. We assess the potential for researchers to augment or replace human-generated training data with surrogate training labels from generative large language models (LLMs). We introduce a recommended workflow and test this LLM application by replicating 14 classification tasks and measuring performance. We employ a novel corpus of English-language text classification data sets from recent CSS articles in high-impact journals. Because these data sets are stored in password-protected archives, our analyses are less prone to issues of contamination. For each task, we compare supervised classifiers fine-tuned using GPT-4 labels against classifiers fine-tuned with human annotations and against labels from GPT-4 and Mistral-7B with few-shot in-context learning. Our findings indicate that supervised classification models fine-tuned on LLM-generated labels perform comparably to models fine-tuned with labels from human annotators. Fine-tuning models using LLM-generated labels can be a fast, efficient and cost-effective method of building supervised text classifiers.

Nicholas Pangakis and Samuel Wolken. 2024. Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels. In Proceedings of the Sixth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS 2024), pages 113–131, Mexico City, Mexico. Association for Computational Linguistics.
Download Paper

Working papers

The diminishing state of shared reality on US television news

Under review, 2024

The potential for a large, diverse population to coexist peacefully is thought to depend on the existence of a “shared reality:” a public sphere in which participants are exposed to similar facts about similar topics. A generation ago, broadcast television news was widely considered to serve this function; however, since the rise of cable news in the 1990s, critics and scholars have worried that the corresponding fragmentation and segregation of audiences along partisan lines has caused this shared reality to be lost. Here we examine this concern using a unique combination of data sets tracking the production (since 2012) and consumption (since 2016) of television news content on the three largest cable and broadcast networks respectively. With regard to production, we find strong evidence for the ``loss of shared reality hypothesis:’ while broadcast continues to cover similar topics with similar language, cable news networks have become increasingly distinct, both from broadcast news and each other, diverging both in terms of content and language. With regard to consumption, we find more mixed evidence: while broadcast news has indeed declined in popularity, it remains the dominant source of news for roughly 50% more Americans than does cable; moreover, its decline, while somewhat attributable to cable, appears driven more by a shift away from news consumption altogether than a growth in cable consumption. We conclude that shared reality on US television news is indeed diminishing, but is more robust than previously thought and is declining for somewhat different reasons.

With Homa Hosseinmardi, David Rothschild, and Duncan Watts.
Download Paper

Increased Fox News Viewership Is Not Associated with Heightened Anti-Black Prejudice

Under review, 2023

Today’s media environment provides Americans with unparalleled choice in how or whether to watch political TV news. Prior studies have focused on the impacts of the growing range of ideological slants on vote choice. But the fragmentation of the media landscape may also increase variation in the coverage of race-related topics. With a large audience and programs that even some employees thought conveyed racism, Fox News provides a valuable case study. We use a population-based panel 2008–2020 to measure the associations between changes in self-reported Fox News viewership and race-related attitudes and thus bound Fox News’ likely effects assuming positive selection. Difference-in-difference models demonstrate that increased Fox News watching is not strongly associated with increases in Whites’ anti-Black prejudice or opposition to government assistance targeting Black Americans. However, those whose Fox News watching increased grew increasingly anti-immigration. These results indicate the limits of Fox News’ impacts on racial prejudice.

With Daniel J. Hopkins and Yphtach Lelkes.
Download Paper

Select works in progress

Political Narratives in Evangelical Sermons

In progress, 2024

Commentators have regularly noted the increasing involvement of evangelical movements in American politics. Despite this consensus, there is limited understanding of how evangelical leaders engage with politics within their congregations and potentially shape their congregants’ political views. To examine the prevalence and content of political speech in evangelical communities, we construct a dataset of approximately 50,000 sermon recordings around the 2022 midterm elections. We employ large language models (LLMs) to identify policy discussion and vote instructions in transcribed sermons. Our findings reveal that political references in evangelical sermons are frequent, especially around elections. By contrast, direct endorsements of political candidates remain rare, suggesting a maintained division between church and state, with political discourse in evangelical churches focusing on social issues rather than specific candidates.

With Marc Jacobs, Sean Westwood, and Yphtach Lelkes.

Sam Wolken