Is my AI working? Auditing for accuracy and impact
Presentation from the 2025 Button Conference
My proposal was selected among nearly 300 submissions for the Button conference program. Hundreds of content designers from around the world tuned in virtually to sharpen their skills, meet fellow industry folks, and learn new trends.
I was honored to present the following talk about AI audits:
Why do an astonishing three in five people distrust AI systems? One big reason: AI chatbots are often untruthful or unhelpful. The good news is that, as content designers, we’re well-positioned to audit our organization’s AI chatbots to prove how well (or poorly) they’re really performing.
In this session, I covered how my content team took on this challenge. We analyzed and reported on thousands of chatbot conversations in just a few weeks. The results? Tangible, impactful metrics that revealed when the bot resolved the customer’s problem and when it fueled their frustration. From identifying opportunities to save money by reducing human support cases, to discovering content gaps they never saw coming, they proved that content audits — content’s classic tool — are perfect for AI, too.
In this session, attendees learned how to:
- Explain the purpose and value of a content audit for AI products to gain buy-in from stakeholders.
- Discover how to customize metrics and audit processes to fit your organization’s AI or LLM solutions.
- Get practical advice on creating the artifacts you need to run the audit, report the results, and make data-driven improvements.
- See real-world examples of how auditing AI chatbots can lead to better customer support, improved content accuracy, and smarter automation opportunities.
Image from Button Events
