Trust in the Machine: Dissecting AI Authorship in News Credibility

Prompt

AI Authorship Disclosure and News Topics: Examining Machine Heuristics in Audience Perceptions of News Credibility Ziyao Zhang – Washington State University (WSU) Introduction / Background • AI in News: News organizations (AP, Bloomberg, Forbes) now use AI to generate content at scale. This raises the question: Will audiences trust news knowing an AI wrote it? Credibility is central to journalism, so any drop in trust is problematic. • Mixed Credibility Findings: Studies show mixed results – some found AI-written articles as credible as human-written, or even more credible, while others found credibility diminishes if an article is known to be machine-authored. This inconsistency suggests the impact of AI authorship may depend on context cues (e.g. topic, writing style) and audience preconceptions. • Transparency vs. Trust: Audiences want transparency – 94% of news consumers favor disclosure if AI is used in journalism. However, disclosing “AI-authored” could either reassure (nothing is hidden) or raise skepticism about the article. Journalism ethicists recommend transparency despite this dilemma. • Machine Heuristics: This study applies machine heuristic (MH) theory to explain audience judgments. Machine heuristic = a mental shortcut where people assume machines are more objective and reliable than humans. For example, if a story is labeled as AI-written, some readers might trust its data accuracy (seeing the AI as unbiased), but others might doubt its creativity or insight. The MH framework will guide expectations about when AI authorship helps or hurts credibility. Research Questions & Hypotheses • RQ1: AI Disclosure Effect – Does labeling a news piece as AI-authored (vs. human-authored) change audience perceptions of its credibility? o H1: AI Authorship will slightly lower perceived credibility. News labeled “Generated by AI” will be judged less credible than the same content labeled “By a human reporter”. This prediction is based on a modest negative bias from MH (people may be wary of AI judgment), consistent with prior findings of a small credibility drop for machine-written news. • H2: Topic Moderation – The effect of AI disclosure on credibility will vary by news topic. (Certain topics might amplify or mitigate the AI credibility gap.) We anticipate topic-driven differences: for data-heavy news (e.g. finance, weather), an AI author might even be seen as more objective, yielding no drop or a slight boost in credibility. In contrast, for political or creative topics, AI authorship may trigger more skepticism (concerns about nuance or bias), yielding a bigger credibility drop. No specific direction is set in stone – H2 is exploratory pending data. • RQ2: Individual Differences – Do audience traits (AI literacy, age, etc.) moderate these effects? For instance, tech-savvy readers might be more critical of AI content (less swayed by the “AI” label), whereas older or less AI-familiar readers might distrust AI-written news more. We will examine if credibility ratings differ by such traits (though these analyses are secondary). Methodology: Two-Wave Experiment Design Figure: Two-wave experimental design. Wave 1 (within-subjects) – each participant rates multiple news headlines, some labeled “AI-Generated” vs others “By Reporter”, to test immediate reactions across topics. Wave 2 (between-subjects) – participants split into two groups, reading full news articles either with an AI author disclosure or without (human author), to observe credibility in a deeper reading context without cross-comparison bias. • Participants: ~600 U.S. adults (approx. N = 200 in Wave1, N = 400 in Wave2) recruited via Prolific. Sample is demographically diverse (age, gender, etc.) to reflect general news consumers. We also measure each person’s AI literacy (familiarity with AI in news) as a potential moderating variable. • Wave 1 (Headline Evaluation): A within-subjects experiment. Each participant sees 15 news headlines (short titles) from diverse topics (politics, business, sports, entertainment, etc.). For each headline, they see a byline cue indicating authorship. Some headlines are randomly labeled “Generated by AI”; others “By Staff Reporter”. Each participant sees both AI and human attributions across different headlines, allowing a direct comparison of credibility ratings within the same person (controls individual differences). We counterbalance which headlines get the AI label vs human label across participants to avoid any one headline being inherently “more credible” irrespective of label. After each headline + label, participants rate perceived credibility (e.g. “How accurate and trustworthy is this headline?”) on a Likert scale. • Wave 2 (Full Article Reading): A between-subjects experiment. Participants are randomly assigned to one of two groups: AI-disclosed vs No AI disclosure. Each group reads the same set of 5 full news articles (≈300–500 words each) selected from Wave 1 results (articles that showed the largest AI-vs-human credibility differences in Wave 1). The experimental group sees these articles with an AI author label (e.g. “Author: News Bot AI”), whereas the control group sees the identical articles but attributed to a human reporter. Participants then rate each article’s credibility using the same questions as Wave 1. Between-subjects design here ensures that each person sticks to one condition (AI or human) for all articles, preventing explicit comparison between AI and human content within a session (which could tip off participants or cause contrast effects). This wave tests if the AI disclosure effect holds during in-depth reading when readers engage more fully with content (and to see if initial headline impressions carry over to full stories). • Key Measures: The primary outcome is Perceived News Credibility – an index combining items on accuracy, believability, and bias, adapted from established scales. We will compare mean credibility scores for AI-labeled vs non-AI content. In Wave 1 we expect to use paired comparisons (since each person saw both conditions), and in Wave 2 independent comparisons between groups. We will also record open-ended comments (qualitative feedback) on why participants trusted or doubted the articles, to add context. Additionally, demographics and AI literacy scores will be recorded to analyze RQ2 (e.g. does familiarity with AI correlate with smaller credibility differences?). Key Concepts • Artificial Intelligence (AI) Authorship Disclosure: An explicit label or note revealing that a piece of news content was written by an AI rather than a human journalist. In this study, the disclosure is conveyed via the author byline (e.g. “Generated by AI”). This source cue is the central independent variable expected to shape credibility perceptions. • Perceived News Credibility: The audience’s judgment of a news item’s believability, accuracy, and trustworthiness. We measure this on a 5-point agreement scale (e.g. rating statements like “I find this news story credible and reliable”). Higher scores mean the content is seen as more credible. Credibility is influenced by content features and source cues – here, the “source” is either AI or human. • News Topics: The subject domain of a news item (politics, technology, sports, etc.). We sample a broad range of topics to test H2. Topic matters because audience trust in AI might depend on the content: e.g. factual topics like financial updates or weather (where data and objectivity are valued) may invoke a positive machine heuristic (AI seen as competent), whereas opinion-laden or narrative topics (political news, cultural commentary) might invoke skepticism about an AI’s understanding or fairness. • Machine Heuristic (MH): A cognitive shortcut (stereotype) where people equate machine output with objectivity, efficiency, and lack of bias. “It’s done by a computer, so it must be impartial and error-free.” This can enhance credibility if the audience activates the positive side of the heuristic. However, if an AI is expected to lack creativity or empathy, a negative machine heuristic may appear (e.g. thinking “an AI can’t grasp nuance, so its news is less trustworthy”). In our study, MH theory predicts the baseline effect (some trust AI’s objectivity, others doubt AI’s ability), and helps explain why results might differ by topic or individual. • Algorithm Appreciation vs. Aversion: Related to MH, this refers to mixed attitudes toward AI decisions. Some people show algorithm appreciation – preferring algorithmic judgments over human ones in certain tasks (due to perceived accuracy). Others show algorithm aversion – if they see an AI make a mistake once, they lose trust in it more than they would for a human error. These attitudes may influence how participants respond to AI-written news. High MH believers might give AI-authored news the benefit of the doubt, whereas those with algorithm aversion may default to distrust. Expected Findings Expected credibility ratings: Illustrative comparison on a 5-point scale (5 = high credibility). We anticipate news labeled with an AI author will score slightly lower on perceived credibility (e.g. ~3.7/5) than the same news labeled with a human author (e.g. ~4.0/5), reflecting a modest credibility gap. • Slight Credibility Gap: In line with H1, we expect AI disclosure to modestly reduce credibility. The difference will likely be small (on the order of a few tenths on a 5-point scale), but noticeable. This aligns with meta-analytic evidence that overall, machine-written news tends to be seen as slightly less credible than human-written news. We interpret this as some readers applying a cautious bias: “AI might miss context or have unknown errors.” • Variation by Topic: Supporting H2, the magnitude of the AI effect should differ by news category. We anticipate little to no drop in credibility for “straight facts” topics (e.g. finance stats, sports scores, weather reports) – some participants might even rate AI-authored factual reports as equally or more credible than human, due to the machine’s perceived data-processing strength. In contrast, for controversial or nuanced topics (e.g. politics, social issues), we expect a bigger negative impact of the AI label, as readers might doubt an AI’s ability to handle nuance or bias (“Can an algorithm understand complex social context?”). There is also a possibility that in polarizing news, an AI author could reduce perceived partisan bias, which might improve trust among some users – an interesting twist to explore if observed. • Role of Individual Traits: We expect to find that audience background moderates these perceptions (RQ2). For example, participants with high AI literacy (familiar with AI) might show a smaller credibility gap, or even trust AI-writers more, because they understand how the AI works (or know that it’s widely used for fact-based reports). Conversely, older adults or those less comfortable with AI tech may exhibit a larger drop in credibility when AI is disclosed, reflecting lower trust in new technology. We will look for such patterns (e.g. an interaction between AI disclosure effect and age or AI familiarity in the data). • Qualitative Insights: From open-ended responses, we expect comments like “AI writers stick to facts, which I like for finance news” versus “I worry an AI can’t capture the human angle in this story.” These anecdotes will help illustrate the reasons behind the numbers, enriching our conclusions about MH in action. Implications / Conclusions • Credibility vs. Transparency Dilemma: If AI disclosure indeed slightly lowers credibility, news organizations face a tricky balance. On one hand, transparency is ethically required – audiences overwhelmingly want to be informed about AI use. On the other hand, disclosure might inadvertently trigger skepticism. Newsrooms should proactively address this by educating audiences that an AI-written piece has been edited or fact-checked by humans, thereby maintaining trust. The goal is to avoid a “black box” effect where readers distrust what they don’t understand. • When to Use AI Writers: Results could guide editors on which topics are “safe” for AI authorship. If we find, for example, that AI-written sports recaps or weather updates retain credibility, newsrooms can confidently use AI in those areas (freeing human reporters for complex stories). However, if political or investigative pieces suffer credibility loss with AI labels, those might remain human-reported domains, or at least AI involvement should be carefully messaged. • Harness Positive MH: The machine heuristic can be leveraged: emphasize the strengths of AI (accuracy, data handling) in content that plays to those strengths. For instance, an AI-generated financial report might note “Compiled from 10,000 data points in seconds by AI” – framing that invites trust in accuracy. By contrast, for stories needing empathy or analysis, pairing AI with human oversight (hybrid authorship) and highlighting that may reassure readers. • Audience Training and Literacy: The findings underscore the need for boosting media and AI literacy. When readers understand what AI journalism is (and isn’t), they can judge content on its merits rather than just the source label. Educating the public that AI can do certain reporting tasks well (and that there are editorial checks in place) could mitigate knee-jerk skepticism. Over time, as AI becomes a common tool, the novelty factor will wane and credibility judgments may normalize. • Theoretical Contribution: This study will extend MH theory into the news context, showing how machine-as-source cues influence trust. It will clarify the conditions under which the “machines are objective” belief holds true versus when it backfires. The two-wave approach also provides a template for future research on initial impressions vs. sustained reading effects in AI-mediated communication. Ultimately, understanding these dynamics can help build human–AI collaborations in journalism that preserve public trust. Selected References (abridged) • Wang & Huang (2024) – Meta-analysis: AI authorship has a small negative effect on perceived news credibility. • Tandoc et al. (2020) – “Man vs. Machine?”: Found minimal differences in credibility between algorithm-written and human-written news. • Waddell (2018) – “A Robot Wrote This?”: Early experiment showing readers rated AI-written news slightly lower in credibility. • Yang & Sundar (2024) – Machine Heuristic scale: Defines the “machine heuristic” and measures belief in machine objectivity. • Fletcher & Nielsen (2024) – Reuters Institute report: Survey of 6 countries – majority of news consumers want AI use disclosed in journalism.

Image Details