The EBPM Seminar Series at the Department of Public Policy brings external academics and practitioners to discuss their ongoing research. This event series prioritizes understanding how data, observations, and ultimately evidence is approached by each researcher.
Abstract
We experimentally investigate whether AI advisors of varying efficacy can help people
distinguish between truth and lies in written text. We design AI advisors with low,
medium, and high efficacy and either reveal or conceal their effectiveness from subjects.
We utilize transcripts of conversations from the TV show “To Tell The Truth” to mitigate
the influence of motivated reasoning and simulate social media exchanges marred
by lies between parties with conflicting agendas on a topic with an objective truth.
We discover that while subjects marginally outperform chance in detecting truth, expectations
regarding AI efficacy greatly impact their reliance on AI advisors. As the
efficacy of low and medium-quality AI advisors falls short of their expectations and
remains undisclosed, subjects’ overreliance on them causes the truth-detection rate to
descend below their intrinsic ability. Upon disclosing AI efficacy, subjects reduce their
reliance, which improves truth detection. The high-quality AI advisor whose efficacy
matches the subjects’ beliefs enhances truth detection, regardless of whether its efficacy
is disclosed. These findings highlight the risks of undisclosed AI efficacy, which can
exacerbate misinformation and underscore the urgent need for transparent AI policies.
Haimanti Bhattacharya,Subhasish Dugar,Sanchaita Hazra,Bodhisattwa Majumder