Teaching Claude to be a Customer Support Chatbot with Prompt Engineering

19 min readMay 29, 2023

The growing popularity of Language Model Models (LLMs) has given rise to a new area of expertise known as prompt engineering. This field is centered on creating and refining prompts or inputs to generate the most desired and effective outputs from these models. The value of prompt engineering is particularly evident when we consider the vast potential of these models, not only in terms of their immediate output but also in how they can be adapted to perform specific tasks.

With this in mind, I took on a prompt engineering project to build a customer support bot for Mailchimp (my current company) using Claude-100k, an LLM with a 100k token context window. I chose Claude because of its extensive context window, which makes it capable of processing lengthy articles and instructions. This feature makes it particularly suitable for incorporating existing help articles as context in the prompt. The ability to handle long context also allows the inclusion of brand voice and tone instructions, which is essential in maintaining a consistent customer experience.

Using prompt engineering to create a customer support bot is both valuable and fascinating. Customer support is a crucial aspect of any business, and automating this process without compromising service quality can bring significant benefits. However, this project presents an exciting challenge in prompt engineering. How can we design and optimize prompts to enable the bot to understand and respond accurately to various customer inquiries while maintaining the brand’s unique tone and style?

In the following sections, I will discuss the process and techniques used in this project, including the challenges and breakthroughs encountered along the way. Join me as we explore the captivating world of prompt engineering in the context of creating a customer support bot.

Goals/Measures of Success

In order to know if I’ve built a bot that will be useful, I set some goals that the bot must meet. I will list each one and then explain each in detail

Gives accurate information
In the correct tone/voice
Responds with empathy to human emotion
Understands when to refer customer to a human
Speaks English and Spanish and responds appropriately to queries in these languages

Gives Accurate Information

In order to be helpful to customers, a customer support bot must give accurate information. If the bot gives information that is not true, it will not solve the customers’ problems and it can cause people to become frustrated and lose trust in the brand. However, this objective is not simple to achieve because LLMs are notoriously prone to hallucination. That is, making up fake information. LLMs hallucinate because they are “trying to be helpful” and don’t recognize that making up information is worse than saying “I don’t know.”[1]. Therefore, prompt engineering to minimize hallucinations is particularly important.

Mailchimp Tone/Voice

Mailchimp’s brand is known for its unmistakable voice and tone, which is characterized as straightforward, authentic, and sprinkled with offbeat humor. To ensure that the bot accurately represents Mailchimp and delivers a consistent customer experience, it is essential that it communicates in this distinctive style. This task poses a dual challenge for Claude: firstly, to comprehend and imitate the nuances of Mailchimp’s voice and tone, and secondly, to produce responses that align with this voice.

Responds with Empathy to Human Emotion

A key element of successful customer service is the ability to respond empathetically to a customer’s emotional state. If a customer is frustrated or angry, the bot must be able to recognize this and respond appropriately, providing reassurance and understanding. This requires the careful design of prompts that not only seek to solve the problem at hand, but also acknowledge the customer’s feelings and express empathy.

Refers Customer to Human When Necessary

While LLMs can handle a wide range of inquiries, there will be instances where a human touch is needed. The bot should be able to recognize its own limitations and know when to refer a customer to a human support representative. This requires prompt engineering that takes into account the complexity or nature of the customer’s issue, and the bot’s ability to provide a satisfactory solution.

Bilingual

Finally, the customer support bot must be bilingual, able to communicate effectively in both English and Spanish. This capability is essential as a significant percentage of Mailchimp’s customer base speaks Spanish, and the company has prioritized internationalization.

Prompt Engineering in Action

With these goals in mind, let’s examine the evolution of the prompts I constructed to ultimately create a customer service bot that achieved these goals.

Initial Prompt

Given tone and voice guidelines and customer support help articles, act as a customer support bot. After I give you the tone and voice guidelines and customer support help articles, acknowledge receipt of the materials and let me know you are ready to act as a customer support bot. Answer any further questions as if you are Mailchimp’s customer support bot. Tone and voice guidelines: {tone/voice guidelines}. Help Articles: {help articles}

Link to prompt
Link to chat history

This initial prompt has some important components: role prompting, context, and instructions.

Role prompting is when you give the LLM a particular identity to assume. It has been shown that when given a role, LLM will often produce better output [2]. With role prompting I tell Claude to “act as [Mailchimp’s] customer support bot.” When I tell Claude to act as a customer support bot, I don’t need to give it all the instructions on how to answer questions. It takes the information it has been trained on and what it already knows about how a customer support bot is supposed to act and incorporates that into its responses

I also give it some important context: the tone and voice guidelines and the help articles I want it to draw knowledge from.

Finally I give it instructions:

Acknowledge receipt of materials
Confirm that it is ready to act as a customer support bot
Answer all further questions in that role

So, how did Claude do? This iteration of the prompt was definitely a good start (see full results by clicking the link “Link to chat history” at the top of this section). It achieved a fair amount of the goals I set. It responded on brand, with empathy, and properly handled both English and Spanish queries. However, where Claude struggled was with giving accurate information and referring to human support when necessary.

When I asked Claude about various things that I didn’t give it information about, it confidently produced responses. I gave Claude help articles on how to import contacts and troubleshooting if contacts failed to import. Although I gave Claude limited information, it answered all sorts of questions. Some of its answers were correct (for example, the purpose of Freddie as a “home button,” and even the reason why search was missing from the left navigation). Since Claude was trained on information on the internet up to 2021, that could explain how it knew some of the information. However, some of its information was incorrect, even though it seemed plausible. In particular, Claude gave out a bogus customer support number when I asked to speak to a human. When I asked it where it got the number, Claude admitted it was incorrect. It also made up that you could contact support via a “Messages” tab, which does not exist on Mailchimp.

The ideal response from Claude would be giving accurate information based on the support articles given, and when asked something it was not given information about, to defer to a human employee. Therefore, I updated the prompt to attempt to minimize hallucinations.

Prompt V2: Addressing Hallucinations

Link to prompt
Link to chat history

In this iteration of the prompt I made two changes. I named the customer service chatbot persona and gave Claude an “out” if it did not know the answer. First, to help Claude stay in character, I named the customer service chatbot persona “MailMate” and instructed Claude to act as MailMate. Secondly, and most importantly, to address the hallucinations, I added this instruction: “If a question cannot be answered with the information given, answer politely that you don’t know.” In this instruction I followed two prompt engineering best practices. First, in the Claude documentation on reducing hallucinations Anthropic suggests that you give Claude a “way out” and explicitly tell it how to respond if it does to know the answer [1]. This is because it will hallucinate in an attempt to be helpful. The second prompt engineering best practice I followed was avoiding negative instructions. Instead of telling LLMs not to do something, it is best to tell them what it is that they should do.[3] So, instead of saying “do not give make up information,” I phrased it to give positive instructions instead.

Unfortunately Claude still produced hallucinations. Claude continued to make up a fake customer support number (and a fake url to a customer support page) and give incorrect information on how to contact Mailchimp.

Prompts V3 — V4: Further Addressing Hallucinations

V3 — Link to prompt
V3 — Link to chat history

V4 — Link to prompt
V4 — Link to chat history

In order to further address the hallucinations, I decided to employ another prompt engineering best practice, few shot learning. Few shot learning is when you provide several examples (shots) for what acceptable output is [4]. Giving the LLM examples of acceptable output has been known to improve responses. It differs from fine-tuning in that few shot learning does not actually change the parameters of the model and requires much fewer examples. I initially included two examples, one example in which the LLM had the information to answer the question, and one in which the LLM did not have enough information to answer the question. However, I had to update the examples to make them more comprehensive because initially, though I succeeded in preventing Claude from hallucinating, Claude was also overcorrecting and saying it didn’t know the answer when it did have the information to answer the question. Finally, I also updated the prompt to include a number for customer support (I gave it a fake number for demonstration purposes), so that Claude would stop hallucinating a customer support number.

Claude correctly directs user to human support when it can’t help

V3-Claude initially directs the user to human support even though it was given information on this problem

V4-given more examples, Claude correctly suggests what could be wrong

Prompts V5 — V8: Proper Bilingual Behavior

V5 — Link to prompt
V5 — Link to chat history

V6 — Link to prompt
V6 — Link to chat history

V7 — Link to prompt
V7 — Link to chat history

V8 — Link to prompt
V8 — Link to chat history

Prompt V4 was successful at getting Claude to avoid hallucinating and also answer questions for which it had information, but when I tested it on an inquiry in Spanish that it had the information for, it responded to contact customer support.

Customer: How do I import contacts?
Claude: I’m sorry, I only speak English. To receive help in Spanish from Mailchimp, speak with our support team at 1(888)-825–0101. They would be happy to help you.

Claude’s behavior in Spanish is undesirable, because we know that in V1, Claude did have the ability to answer queries in Spanish. Furthermore, Claude actually responds in Spanish, saying that it only speaks English. The examples that were provided were all in English, so Claude took then extrapolated that it couldn’t answer in Spanish. In V5 of the prompt I addressed this problem by adding a Spanish example. I also added to the role section of the prompt that the persona MailMate could speak English and Spanish. Finally, I added to the instructions section of the prompt for Claude to translate the help articles into Spanish and respond to inquiries in Spanish according to the Spanish version of the articles.

These changes led to some surprising unintended behaviors. Due to the Spanish example being last, Claude thought it was supposed to default to Spanish.

Thanks for sharing the help articles and tone and voice guidelines. I have received the materials and I’m ready to act as MailMate, Mailchimp’s customer support bot. My name is MailMate and I am happy to help by answering your questions based on the information provided in the help articles and always in a friendly and professional tone. If I can’t answer a question with the information given, I’ll kindly indicate that I don’t know the answer and suggest contacting customer support at 1 (888)-825–0101.

In V6, I addressed the problem introduced in V5 by moving the Spanish example to the top of the list. This gave good results for the most part. It spoke in English to me when I spoke in English, and responded in Spanish when I spoke in Spanish. However, when I asked it something that required knowledge of the help articles (in English), it reverted to Spanish.

It responded in Spanish when it needed to quote the help articles

In V7 and V8, I fixed this issue by explicitly stating when Claude should speak in English vs. Spanish. I also removed the instruction that Claude was to translate the help articles to Spanish, as it was getting confused by that. These instructions followed the prompt engineering best practice of avoiding impreciseness [5]. With those changes, I finally got the results I desired and I met all the goals I had set out to achieve.

Lessons Learned

In the end, I learned that role prompting, instruction prompting and including proper context can get you very far in prompt engineering. In addition, being as specific as possible, using few shot learning (including comprehensive examples) can help you get even better results.

Prompt Version History

In the following prompts, when {help articles} are referenced, these are the help articles: https://mailchimp.com/help/import-contacts-mailchimp/ and https://mailchimp.com/help/troubleshooting-contact-imports/. When {tone/voice guidelines} are referenced, this is the article: https://styleguide.mailchimp.com/tldr/

V1 — Chat History