LLM evaluation · Talk, Dec 2026 · 2026
AI vs. human sentiment analysis
An LLM pipeline that codes newspaper coverage the way trained research assistants do, and measures how closely it matches them.
- LLMs
- Classification
- Evaluation
- Python
What they had
BYU researchers are studying how U.S. newspapers covered the LDS Church during the “Mormon Moment,” roughly 2011 to 2013: Mitt Romney’s campaign, The Book of Mormon musical, and more. Research assistants had hand-coded each article in a large workbook: article type, tone, audience, a 1–5 favorability score, and topic tags.
How I got involved
The team wasn’t sure what a software engineer could do for the project. I asked to try, and a week later came back with a set of their articles already scored by AI.
What I built
- A pipeline that replays the same articles through an LLM and returns the same schema the human coders used.
- A comparison that measures how well the LLM reproduces human judgment, so the team can extend the coding to articles nobody reached and know how far to trust it.
Status
The manual-versus-LLM comparison is the newest work, and a talk on it is planned for December 2026.