Chatbots: metrics for generating customer insights

The great interest for chatbots in the e-commerce segment has brought hundreds of chatbots for miscellaneous purposes to life within short time. From productivity bots, brainstorming assistance and cooking bots all the way to meditation and entertainment bots – the range is great. Providers of the core sector of bot programming and adjacent services as tracking, visualization and conception come alive.

Although some customers have a cautious attitude towards artificial intelligence, more and more clients count on chatbots and see them as useful tools. Jeremy Pounder, Futures Director bei Mindshare, states:

Chatbots have emerged as a new way for brands to engage with customers […]. Surprisingly consumers are very receptive to AI chat technology, and even trust it over human interaction in certain scenarios.

Chatbots – new customer journey components

Many companies mainly use bots for image purposes in order to represent themselves as an innovative, technological business. Furthermore, bots can be installed as useful tools in all phases of the customer journey. Apart from their main tasks, bots can generate valuable data regarding customer insights.

As a bot replaces immediate human communication in many aspects, there is a possibility to record and analyze almost “normal” conversations with customers in form of texts. These texts themselves and their corresponding interaction index numbers show new and interesting possibilities for marketers.

What kind of chatbot metrics are relevant?

Starting with the basic questions regarding the usage of chatbots up to the qualitative characteristics of the course of the conversation the chatbot index numbers offer a great overview about the user interactions.

Active and engaged rates – do the users interact with a bot?

With the increase of the user’s trust towards chatbots, the question is no longer if the user would build up a first contact to the chatbot. The question now is rather if the conversation can be held up. 

Image: chat record in "Morning Text"
 

When a user reads a message during a session, this session can be obtained as active. Therefore, the active rate equals the number of active sessions divided by the numbers of all sessions of this user.

When a user answers to a message during a session, this session can be obtained as engaged. Therefore, the engaged rate equals the number of engaged sessions divided by the number of all sessions of this user.

The validity of both index numbers (especially of their combination) mainly depends on the chatbot’s aim and on which product is concerned.

Chatbots which for example deliver the user daily updates on a certain topic or motivation quotes do not require an interaction and therefore show a low engaged rate (see Morning Text). For a bot, which serves as sales service this would be disastrous.

However, a high active rate is for all kinds of chatbots relevant as even a one-way communication offers an added value for the user and the company.

Confusion rate and triggers – how often occurs a misunderstanding between bot and user and why?

Depending on the question and the issue users can enter many different input versions, especially those which the chatbot does not understand (see MojiHunt – It exists more than one key emoji!).

On the other side, there are cases that the user does not understand the chatbot due to misleading wording, unfortunate diction or lack of knowledge. 

Image: chat record in "MojiHunt"
 

Both ways of misunderstanding can be estimated with the confusion rate index number. The confusion rate equals the number of sessions with a confusion situation divided by the number of all sessions of this user.

Yet this index number cannot be classified without a qualitative analysis. The evaluation of the dialogue texts offer a very good insight into the confusion triggers (cause of the confusion situations) and give possibilities to integrate new input alternatives.

A qualitative analysis could have a great influence on the decision when the conversation should be handed over to a CRM staff member. Moreover, the confusion triggers could point to customer questions which were not taken into consideration at the stage of conception of the chatbot.

Conversation steps – how much text is needed?

How often have questions and answers to get changed until the user gets to his goal? How can the user focus the dialogue with the chatbot? These questions can be answered with the help of the number of conversation steps.

The conversation step is an exchange of messages within the chat. Example: The chatbot asks a question and the users answers with yes or no.

A great number of conversation steps is not always a sign of a productive conversation with the bot. It even often means, that the bot does not understand the user properly and has to ask again. Or the user states wrong or unsatisfactory information which would necessary for the conversation or the service.

Very short conversations are in reverse a sign for a high termination rate. This can be monitored either in standardized way or in an individual period of non-reactions.

Monitoring the number of conversation steps and the corresponding termination rate can help to cluster the users regarding their goals and their typical course of conversation and it helps to optimize the paths of dialogue.

Average number of conversations/user – how often do users interact with the bot?

Many chatbots get positioned as promotion touchpoints. Therefore, they are only relevant within a certain period of time. Other chatbots serve as a continuing service and last longer.

The average number of conversations/user is an index number which only delivers a descriptive observation if taken into consideration without background information about product or service. For example, these questions could be asked:

• How does the average number of conversations develop during the course of time?
• What is the average number of conversations per day, week or month?
• How often is the service used usually?
• How often gets the user encouraged to continue the chatbot conversation? Does the user have to interact with the chatbot?
• Which answers does the chatbot offer?
   

A decline in number of conversations/user could be an indicator for the lack of interest regarding the offered information and formats or else the user’s questions are not answered properly. If chatbots should for example offer product alternatives for sale, a decline in number of conversation/user could be an indicator that the displayed products do not match the user’s needs. A logical consequence could be to ask more questions from the start.

This index number has a significant power when new features get implemented or tested.

Retention – how often do they return?

The combination of the average number of conversation/user, active rate and engaged rate gives information about the user retention.

A clustering of new features, dialogue threads, formats, promotions etc. after the time of introduction shows a qualitative effect on the user experience. The return rate as basic indicator for the retention should not be ignored.

Sentiment analysis – the top of the icing

Chatbots lead a dialogue with real people with feelings. An empathic chatbot is not only entertaining but can also generate insights into the emotional coloring of different dialogue topics.

In the neurolinguistics programming, words which describe emotions, can get evaluated either by categories or by numerical values.

In this matter, a precise statement regarding the emotional perspective of the user within the conversation can be made without a deeper text analysis. Hence, positive and negative conversations can be identified regardless of content or aims and can be used for optimization and training purposes.

Functions and targets

Chatbots can be used many-sided within the user journey. At the beginning of a chatbot project, it is important to understand, which targets have to be fulfilled. The bot’s aim can be derived from the corporate objectives. Its functions and goals as well as the character of the product which should be promoted (service, combination of service and product, product line etc.) decide over the metrics, which should be tracked and about the customer insights which should be gained.

A chatbot will be integrated into an existing customer journey of a company. That’s why chatbot metrics cannot stand alone apart from other journey stations and equivalent KPIs.

The following example shows schematically the functions and targets of the chatbot in a service-oriented company: 

Apart from hard facts and figures, also qualitative aspects play a role. The metrics provide information about how efficiently processes take place. But they cannot encourage to question single dialogue threads.
When it comes to the evaluation, a phase of learnings should be generated. Just like a person who learns through experience, a chatbot should be developed further.

Sources

Venturebeat.com: 5 metrics every chatbot developer needs to track
Businessinsider.com: Here's how chatbot metrics differ from traditional apps
Chatbotsmagazine.com: The Top 5 Analytics Platforms For Chatbots
Chatbotsmagazine.com: Empathic Chatbots