AI job interviews for nonprofits can get candidates on camera within minutes. But when Staffing Boutique founder and president Katie Warnock tested an AI interview tool for 90 days, some of its highest scores went to people her team wouldn’t shortlist.

Katie takes us inside the trial, from the moment an applicant books a recorded video interview to the moment a recruiter checks the score. The appeal is easy to see. If you’re staffing up for a busy season and HR is one of several hats you wear, screening for location and start dates can eat up an afternoon. Katie’s team used the tool for those early questions and followed up with human interviews.

Then came the surprises. Candidates who completed the interview often gave it top marks on the vendor’s five-point survey. Katie doesn’t recall placing anyone who went through the process. Some candidates skipped the bot and emailed a recruiter directly. And when AI résumé screening saw the word “development,” it could give credit for business development where a nonprofit needed fundraising experience. As Katie puts it, “I was just looking at the content.”

Katie also raises the question of AI hiring bias. In the résumé-screening study discussed in the episode, the models significantly favored White-associated names in about 85% of the cases tested.

Key Takeaways:

– Pilot automated interview screening on a defined set of roles before signing a longer contract.

– Use early screens for basics like location and availability, then verify role-specific experience in a human interview.

– Compare scores with recruiter reviews, candidate drop-off, and actual placements.

– Use recording timestamps to find answers, then review enough of the conversation to hear them in context.

– Check that recruiting chatbots identify themselves as AI from the first message.