How Beta Testing A New Member Experience Led To Better AI Use

A great member experience doesn’t happen on its own. At TruStone Financial Credit Union ($5.4B, Plymouth, MN) it takes a cross-functional team.

The Twin Cities-area cooperative began a digital transformation process two years ago with an eye toward improving member experience in all areas but particularly via self-service channels. In-house testing is key to that, and the process has two parts: beta testing and pilot testing.

The in-house technology team’s beta-testing process focuses on finding major bugs and operational problems. Staffers from different departments serve as pilot testers who focus primarily on the user experience.

“We’ve been on this digital transformation adventure for two years, and we’ve developed a pilot-testing team with so many different resumes — some from the technology side, some from staff development, some from the front- or back-office,” says Lisamarie Meyer, senior vice president and director of Minnesota branches. “Less is best when it comes to the beta testing group, and then we spread it out larger. With too many cooks in the kitchen, it can get kind of convoluted.”

Sneak Peek At Beta Testing Process

Pilot testers’ emphasis on UX means focusing heavily on product functionalities, ease of use, design, and member communicability. The credit union selects a range of different testers for each product, aiming for a mix of ages, roles, tech savvy, and more. Sometimes it also brings on board specific users who are familiar with or have been vocal about a product.

“We almost promote it as a sneak peek,” says Jill Schaffler, senior vice president of enterprise applications. “You’ve been asking for card controls, here’s a sneak peek — help us make it the best it can be.”

If pilot testers report a particular product needs work, they’re brought back into the fold to retest it once developers have made changes. In one recent project, the combination of new procedures and repeated testing helped reduce friction in online account opening, bringing the process from more than 10 minutes to less than three, Meyer says.

Timelines for testing vary by product — TruStone subjected its AI chatbot to five iterations of beta testing before moving it to pilot users — but pilot testing should take only a week or so. If that process uncovers features that derail the rollout to members, then the credit union pauses production. But most of the time, notes Schaffler, beta testers have already uncovered most of the major issues. Even if pilot testers do uncover defects, the credit union still moves the product into production if it expects fewer than 10% of users to run into the issues.

“We want our employees to break stuff and find problems before our members do,” Schaffler says, noting that members haven’t uncovered any new bugs to date. Most of the changes that come from pilot testing, she adds, are cosmetic tweaks, such as changing a font or the location of a button.

The beta- and pilot-testing model has use cases beyond just member-facing digital tools. TruStone is currently working toward implementing a new CRM that will touch the entire organization, so testers will need to encompass a wider variety of departments than normal, including service, sales, and more. Although the credit union hasn’t started that process or set the timeline yet, it expects at least five phases of testing.

Lessons Learned

“We want our employees to break stuff and find problems before our members do.”

One of Meyer’s biggest takeaways from the past two years is the importance and differentiation of digital versus in-person experiences.

“A digital experience is different from a branch experience,” she says. “In the credit union space we want to make it the same, but it’s not.

In other words, a great member experience for someone who primarily uses self-service channels will be wildly different from a great member experience for frequent branch visitors.

“We want to serve the members where they are,” Schaffler adds. “If they want to go into the branch, we should make that the best experience they’ve ever had in a branch. If they want to conduct business online, let’s make that the best experience.”

Two years of digital transformation has also reinforced the importance of partnerships. It’s easy to remain focused on one department, Meyer says, but it’s in members’ best interest for different groups to work together.

“What’s so great about Jill and I — the dynamic duo — working together is that we think of things differently,” Meyer says. “But at the end of the day we have the same common goal: member experience. We want what’s quickest, fastest, and most efficient for our members.”