AI-generated content labels don't affect public persuasion on policy issues

A comprehensive survey experiment involving over 1,500 American participants revealed that the presence or absence of a label indicating content as AI-generated or human-authored had no significant bearing on its persuasive efficacy regarding public policy messages. Intriguingly, even though the majority of participants accepted these labels as truthful, the messages consistently influenced their policy views by an average of almost 10 percentage points. This investigation's findings were documented in the academic journal PNAS Nexus.

The increasing prevalence of generative artificial intelligence (AI) in political communication presents a complex challenge. While AI offers efficient means for creating persuasive political content on a vast scale, its dual nature allows for both constructive dialogue and the rapid dissemination of misinformation and deceptive practices. This capability enables smaller entities to amplify misleading narratives online, potentially creating a false sense of broad public consensus. The difficulty people face in distinguishing AI-produced text from human writing exacerbates this risk, raising concerns about a potential erosion of trust in the information landscape due to a surge of synthetic content.

One proposed remedy involves mandatory labeling of AI-generated content. Legislative frameworks in both the European Union and the United States are beginning to incorporate such disclosure requirements. However, the actual impact of these AI labels on the persuasive power of the messages remains an open question. It's plausible that people might be skeptical of labeled AI content, given a general preference for the credibility, accuracy, and authenticity often associated with human authorship. Conversely, if AI is perceived as a source of advanced knowledge, such labels could inadvertently enhance persuasion.

To delve into this phenomenon, researcher Isabel O. Gallegos and her team conducted a survey experiment to assess how different authorship labels influence opinions on public policies across four distinct areas: geoengineering, pharmaceutical import regulations, compensation for college athletes, and the accountability of social media platforms.

The study enrolled 1,601 English-speaking U.S. residents through Prolific, with an average age of 40 and 53% identifying as women. Politically, 49% supported Democrats, 20% Republicans, and 25% identified as independent, with the remainder unaffiliated. Participants engaged in an online experiment where they read a text message concerning a specific public policy. This text was randomly presented with one of three labels: explicitly stating authorship by a human expert in U.S. policy, by an expert AI model trained in U.S. policy, or with no authorship details provided. The policy proposals were carefully selected from a previous study, focusing on less polarizing topics to maximize the potential for persuasion among participants.

The messages, which were all generated by AI but manually corrected for any inaccuracies, included statements such as: “Geoengineering presents too many hazards and should not be considered,” “Drug importation compromises safety controls and the domestic pharmaceutical industry,” “Collegiate athletes should receive compensation,” and “Social media platforms ought to be held accountable for harmful content posted by users.” Each statement was reinforced with a concise paragraph containing supporting arguments.

Prior to exposure to the messages, participants evaluated their existing knowledge, agreement, and confidence regarding the policy topic they were about to encounter. Following the text, they re-assessed their level of agreement, confidence in their response, willingness to share the information, and their perception of the information's accuracy. Additionally, demographic data, AI experience, belief in the authorship label, and news consumption habits were collected.

The study's outcomes demonstrated that the messages generally held persuasive power, shifting participants' support for the presented policies by an average of 9.74 percentage points. Nevertheless, the authorship label—whether attributing the message to AI, a human expert, or providing no attribution—did not significantly alter the message's persuasiveness. Furthermore, there were no notable differences in how participants judged the accuracy of the message or their inclination to share it.

Remarkably, this outcome persisted despite 92% of participants indicating their belief in the authorship label. The researchers observed that this finding, regarding the labels' lack of influence on persuasiveness, remained consistent across various participant characteristics, including their prior knowledge of the policy, previous experience with AI, political affiliation, and educational background. However, older individuals did show a tendency to react more negatively to AI-labeled content compared to human-labeled content.

The study's authors concluded that, “Considering the current level of public confidence in AI-generated content, these findings suggest that while authorship labels could improve transparency, they are unlikely to substantially diminish the persuasive impact of such content. This underscores the necessity for exploring alternative strategies to manage the challenges presented by AI-generated information.”

This research significantly contributes to the academic understanding of public trust in AI-generated information. However, it is crucial to recognize that perceptions and trust in AI content are not static and are subject to change as individuals gain more experience with AI technologies. Consequently, these results offer a snapshot of how Americans interacted with AI in 2024, the period of data collection for this study. Future studies in different cultural contexts or at later times might yield varying results. Moreover, the fact that the AI-generated texts were meticulously crafted to be fact-based and logical might have made them unusually resistant to the typical skepticism often directed at AI. The paper, titled “Labeling messages as AI-generated does not reduce their persuasive effects,” was co-authored by Isabel O. Gallegos, Chen Shani, Weiyan Shi, Federico Bianchi, Izzy Gainsburg, Dan Jurafsky, and Robb Willer.