In 2018, Credit Karma hired me into the Core Product team as a Product UX Writer — a title later adjusted to Staff Content Designer. "Conversation design" as its own job title was still years in the future. So the hiring manager didn't really know what role she was looking for and I didn't know anything about designing for a chatbot. What got me the role was an odd combination of skills that had nothing to do with designing conversations.
While at Adecco's incubator about 5 years earlier, I often wrote content straight to GitHub repos, committing copy changes myself instead of handing text across a wall to an engineer. After that, while at an HR-tech startup, I built questions, answers, and engagements to improve product quality and get direct research from early users with Intercom. I didn't think of them as conversations. If anything, content was getting smaller. Microcopy was more valuable than cute turns of phrase. If you could say it with an icon, even better. But I was designing conversations to effectively get freeform feedback from alpha and beta testers for the product backlog.
Two ordinary jobs at small startups. Together, they matched the problem space Credit Karma was starting to define. They'd recently acquired Penny.io, a financial coaching chatbot, and needed someone to shape its conversations and work directly in the code base. This was a rarity for any designer, but that's how Penny's founder and engineers worked together to build the voice, tone and fundamentals of the conversations we shipped.
The first thing I learned about Credit Karma as a first-year “karmanaut” was this: talking about money is intimate; people are guarded about telling even their closest friends or family members the reality of their personal finances.
Penny created an important safe space where users felt comfortable sharing personal details to get highly personalized insights. These attributes made it especially attractive to younger users and immigrants with less American credit experience. They could learn critical details in a shame-free space that already knew their sensitive financial situation. This became a critical benefit of Penny for Credit Karma's growth strategy over the next few years.
The craft came first: a voice and tone that made it safe to ask an embarrassing question about credit. The harder, less visible part was technical. Penny ran on hand-built regex and NLP years before LLMs existed, so every improvement to how well Penny understood a user had to be built by hand. I parsed freeform inputs, found patterns to detect issues, identified strings to address the issues and expanded the conversation's range. If a user phrased something in a way the parser missed, I could open it and fix it directly to understand the phrasing for future conversations.
We built conversations for each vertical, from credit health and identity protection, home loans, and personal debt consolidation to auto financing and insurance. Each vertical has a handful of conversations, each conversation has dozens of branches and hundreds of interactions, each with its own vocabulary and understanding to address users' concerns and anxieties. I dedicated time each week to ingesting new information and queries and continually improving the content, researching, prioritizing and adding authoritative answers to the emergent questions and changing macroeconomic conditions.
Credit Karma had an internal strategic argument: why did people convert on a financial product? One theory, a PM favorite, was that it worked like the candy aisle at the grocery checkout line, users came to check their credit score, saw an offer they could plausibly get, and bought it on impulse. However our technology couldn't see what happened to the user after we handed them off to a partner. An inquiry to their credit might appear, a new tradeline might eventually show up on their credit report, but we didn't have the technology to make those connections and we didn't know whether that was the offer they came for, or left for. We didn't know. But our email targeting capability was able to tie together these individual actions into targeting criteria.
The head of core product wanted to use Penny to run an experiment. Penny had the capability to manage thousands of conversations at scale, but in the product, the user had to engage Penny to start the conversation. The objective was to target users that had 1) followed a partner prompt out of the product, then 2) got a hard inquiry from that partner to their credit, and then 3) had a tradeline from that partner appear on their credit, all within a specific window of time, and 4) ask that user what they got.
We knew from Penny usage data that users didn't engage in conversations in order to divulge their activity. That kind of conversation didn't meet a need, so why would they? Since email could target the users with these specific conditions, the plan was to send an email that simply asked them about their experience and then would deep-link them into a Penny conversation in-product. We'd never used this pattern to start a conversation with Penny, so I was worried users wouldn't know where they had been transported when they tapped through. I shouldn't have.
The email bombed. The subject line read like a generic feedback request, leading to a dismal 7% open rate and a bounce rate in the high 80s. Getting statistically significant data was going to be a long slog. The response wasn't strong enough, but the few hundred that did make it through the conversation adjusted comfortably to deep-linking into a chat interface with Penny; a great learning. I suggested approaching the subject another way based on what we knew about Credit Karma users.
People pay close attention to their credit right before a big purchase, so when a new hard inquiry was made to their credit, that was something that elicited a strong response from an engaged audience. Using the same targeting already built into the email campaign, our content strategy for the email was to focus on the hard inquiry to engage the user.
Subject: "There's a new hard inquiry on your credit, was this you?"
The body of the email explained there had been a hard inquiry and they should verify if it isn't malicious behavior or if they've been hacked. Users simply had to tap through to give Penny the all-clear or to find out how to dispute the inquiry if it's malicious.
After the jump, Penny divulged what organization had issued the hard inquiry and the user either verified that they had engaged that organization or indicated they may be a victim of identity fraud. In the event there was no fraud (by far the most common scenario) Penny went on to ask about the offer and if they'd been approved. If they had, Penny would usher them to some financial advice about making the most of that product. If they hadn't, Penny would take them to advice on how to improve their score to have a better chance next time. This one email and small conversation changed so many things. The CSAT score was crazy good, people felt looked out for, saw Penny being proactive about their safety and assistive with the followup. The business now had a constant read on user outcomes to build data and bring to our partners when it was time to renegotiate contracts. Leadership had been thinking about partner placements as advertising. This data made the case for a different model: Penny as a curator, surfacing the right opportunity because it understood a member's situation and intent.
Beyond the campaign numbers, users who engaged with Penny after a hard inquiry came back to Credit Karma about three times in rapid succession before purchasing a financial product, codifying the pattern of trust that preceded a sale. The email was a huge win.
The success of that conversation made me realize Penny's superpower. It could have these conversations at scale and remember the outcomes of each conversation, adding them as facts to the user's profile. Later that year, when I was working on the Stories project, a core product redesign, we'd worked with the data science team to establish strong propensity models with ML to personalize user experiences on the new dashboard experience. Early ramps showed us that users quickly lost trust in product recommendations if they seemed generic, and personalizing by inserting their name in the offer wasn't personal enough. So we established a triangulation of value, propensity and intent. Content or offers would surface to users when they fit the propensity model (they could get the offer), they matched the value model (their credit profile indicated they would benefit from the offer), and they had some intent to learn more or pursue the offer. The way we would determine intent was initially just to use data that showed they'd previously engaged with an email or feature or offer that indicated intent, but this method was too passive for product managers that wanted to market their offers aggressively on the front door of the app.
So I created a new “Story type,” The Signal Story. This was a Penny tile that would appear in their feed if they met 2 of the 3 triangulating criteria. This story would simply ask them about the topic area of the offer and give users the opportunity to indicate intent and become eligible to see the offer in their dashboard.
We ended up retrofitting Penny's novel container technology to drive the entire Stories experience so each interaction could contribute to memory and trigger more interactions, more at-bats, and a more robust picture of users so we could serve them better.
Then the pandemic hit and we responded by building the Relief Roadmap. Penny's technology was still the only thing in our codebase that could hold session memory and store facts in a user profile. But facts were changing fast during the pandemic. Businesses were closing, people were working from home, everyone's situation was changing and we needed to be able to rapidly update the facts we kept about our users. What we knew about a user's credit picture, employment status, home and car ownership, immediate needs, and family situation was all changing fast. So we built the Relief Roadmap on the Penny technology, framing the experience around an updatable profile at the top of the roadmap so they could update their facts and instantly see their options adjust accordingly. Read more about that work in the Relief Roadmap case study.
After the pandemic, Relief Roadmap was operating on its own and I moved into the emerging verticals. While working on the Autos team, I built an auto affordability conversational flow that calculated current car value and operating costs, insurance rates, and financing interest rates within the dialogue. At first, we had some indications from feedback widgets that users wanted to better understand the value of their vehicle, how to plan for expenses, get cheaper insurance, get a car that will last them a long time, etc. To validate the hypothesis, we stubbed up a quick conversation about automobile affordability, addressing primarily insurance questions or financing questions about your personal vehicle. Conversation engagement was huge and we got straight to work building out the conversations to cover more surface area within the Autos experience domain and helped us hone targeting criteria for different product opportunities. Penny had gone from "the chatbot we acquired" to core infrastructure for the product, both in terms of learning and serving users better.
Credit Karma more or less ingested Penny's interface technology and stopped developing new conversations. A few years later I was working on the Credit Karma Money product and we were trying to solve a complex issue with stakeholders across different organizations in a high-risk environment: filing a transaction dispute claim. I led design on Credit Karma's first customer-facing generative AI product, a dispute-resolution agent built on Intuit's instance of OpenAI, Intuit Assist. Read the Dispute Automation story. Same discipline, time for a new conversation.