AI Experiment: Hacking HubSpot Chat to Improve the Customer Experience

We integrated AI into our website chat and our KPIs flew off the charts. Here’s exactly how we did it and what we learned along the way.

Key Outcomes:

I like to joke that one of my dumbest moments in early HubSpot history was realizing that potential customers preferred to talk or chat to us directly about buying our product, instead of filling out forms or sending emails.

This seemingly obvious insight was a key turning point, however, as it led to the integration of chat on our HubSpot website.

This simple change ensured that users could instantly connect with our sales and support teams. Since then, chat has been one of our most valuable channels for driving customer satisfaction and acquisitions.

Now, with the expanding capabilities of AI, we’ve been wondering: what would happen if we integrated AI into our chat process? Would AI be able to match the effectiveness of our human team?

In the latest episode of Marketing Against the Grain, Emmy Jonassen, VP of Marketing for Demand Generation at HubSpot, and I walk you through this exact experiment — and the lessons we learned along the way.

The Hypothesis: Using AI for Unstructured Data Analysis

Given AI’s particular effectiveness at interpreting unstructured data, such as free-form text in chat interactions, we hypothesized that integrating AI into our chat system would improve the customer experience by providing faster and more precise responses.

We also believed that this strategy would free up our human agents to focus on more complex inquiries and high-intent customers.

As Emmy explained during the episode, "If we could use AI to truly understand what people are looking for and be able to answer their questions when they come to our website, we anticipate meaningful improvement in user satisfaction."

But we still were unsure about how significant these improvements would be — and whether AI would be able to capture the personable, empathetic tone of our support teams.

The Set-Up: Choosing Our Test Pages and KPIs

To set up the experiment, we first decided to integrate the AI chatbot on webpages with high traffic and lower risk.

"This would allow us to test, gather data, and iterate very quickly, without significantly disrupting the user journey," Emmy explains.

Our knowledge base pages, where customers ask practical, straightforward questions, were the ideal candidates.

We then determined our key performance indicators (KPIs):

With these details in place, we were ready to launch our AI experiment and track its impact on customer interactions.

AI Chatbot Experiment Adjustments and Results

After launching the first AI chatbot, we initially saw a decline in CSAT scores — but this was more or less expected, as the model was new and needed training.

To improve this, a team member (shout out to David G.!), began manually annotating chat transcripts to improve the AI's responses, editing each one to be more accurate and contextually relevant based on the users’ questions.

By the end of the experiment, the results were impressive:

While both the conversion rate and value per chat were exciting, the CSAT score was a game-changer. "Any time you can get automation on par with a human experience, that is a huge win," says Emmy.

Given the positive results of the experiment, we felt confident about integrating the AI chatbot onto other pages with different intents.

AI Chat Test Next Steps and Thinking Ahead

Our next major AI chat test is now happening on the pricing page.

Here, the AI chatbot has been trained to not only handle product-related questions — but also to help prospective customers understand our packages and pricing, even enabling touchless purchases in some cases.

While we’re still testing and analyzing, we’re very excited to see the final results and expect similar, if not better, outcomes.

We’re also hard at work developing an annotation user interface that allows more team members to participate in training the AI model.

"The annotation piece is really one of the most important pieces through all of this," says Emmy. "But it’s also the most time-intensive." So by involving more people in this process, we aim to speed up the AI’s training and improve the accuracy of our chatbot even further.

4 Tips for Using AI to Improve the Customer Experience

While the sheer volume of AI technologies can be intimidating, it’s critical for marketing leaders to stay current with these advancements and begin implementation now.

To learn how to incorporate AI into your workflow, download our AI Guide for Marketers and follow the tips and insights below.

1. Start experimenting now.

"Get your AI experiment to a good enough place, get it out in the wild, and then iterate based on real-world feedback," Emmy says. "That’s where you’ll see the magic start to happen."

While it may be tempting to aim for perfection, this will delay your progress and put you behind the competition. Put those concerns aside and get started now.

2. Aim for dramatic results.

There are still marketers who are optimizing for a 5% improvement. Those days are over. AI gives you the ability to build entirely new systems, programs, and automation that get you 100%, 300%, even 500% gains.

The potential for exponential improvement is there, and the opportunity cost of settling for minor gains is too high.

3. Be transparent with customers.

We made a choice early on at HubSpot to be 100% transparent that customers are speaking with an AI assistant in chat.

AI transparency builds trust and helps manage user expectations, which again improves customer satisfaction. This transparency can also help mitigate any potential concerns about privacy or data usage.

4. Start with chat.

If you’re really not sure where to start, I always say that chat is a great, low-stakes option. AI chatbots are key for curating a seamless user journey by giving users exactly what they need and, as we demonstrated in our experiment, can be tested without drastically interrupting the user experience.

To watch our entire discussion about our AI experiment, check out the full episode of Marketing Against the Grain below:

This blog series is in partnership with Marketing Against the Grain, the video podcast. It digs deeper into ideas shared by marketing leaders Kipp Bodnar (HubSpot’s CMO) and Kieran Flanagan (SVP, Marketing at HubSpot) as they unpack growth strategies and learn from standout founders and peers.