Understand observability in AI Agents
Last Update: Nov 2024 • Est. Read Time: 3 MINObservability refers to the ability to monitor, track, and understand the performance of your AI Agent teams. The key feature of our observability tool is traces, which provide a detailed view of how the AI Agent processes an inbound message from start to finish. Traces allow you to see the journey a request takes through different parts of the AI system, giving you a granular look at each step in the process.
By analyzing traces, you can gain insight into how effectively an AI Agent Team handles customer interactions, identify issues, and make improvements to ensure optimal performance.
Who can access this feature? | |
User types | Admins can access the AI Agents Teams page. |
Learn more about AI Agents in this article.
In this article
What are traces?
Traces follow a request's lifecycle as it moves through your AI Agent. From the moment an inbound message is received, traces log each interaction, decision point, and operation the AI performs. This includes accessing tools and retrieving or updating data in Kustomer.
Each trace provides a timestamped breakdown of the following:
- Any articles in your public knowledge base that were used to respond to the inquiry.
- Reasoning operations provide insight into how the AI arrived at a decision or why it took a particular action while handling a customer inquiry.
- The message each AI Agent received from the supervisor.
- Checks to ensure that both default and custom guardrails are followed.
View traces for an AI Agent team
You can view traces for both AI Agent teams that are currently deployed and past versions of a team.
- Go to Settings > Kustomer IQ > AI Agent Team.
- Select View Traces for the team whose performance you want to analyze.
- By default, you will see a list of traces for conversations using the latest deployed team. Traces are shown for every inbound message from a customer. You can select any trace to see details of what took place.
How to read a trace
Note: The results you see in your traces may vary based on how you have your AI Agent team set up.
A trace shows all the operations that occurred to generate the final response the AI sent to the customer. You can select each operation to see details, such as which help article was used and how the AI reasoned its response.
In this section, we will walk through the sample trace below to see what information you can gather from this feature.
1. E-Comm Support Supervisor response
You can start to review a trace by selecting the Supervisor agent at the top. Here, you can see that the the inbound message from the customer was "I would like to return the popcorn bowl." and that the AI responded with:
Hi John! I've initiated the return process for the popcorn bowl. Could you please let me know the reason for your return? This will help me assist you better!
2. Knowledge search
Once an inbound message comes in, the AI performs a Knowledge Search operation to see if there is a help article that can answer the inquiry. If help articles are found, you will see the article's name, the content the AI extracted from it, and a direct link to it.
3. Reasoning
The Reasoning operation gives you visibility into the AI’s internal decision-making process and helps you understand why the AI did what it did. Here, you can see that the AI decided to work with theReturnDataSpecialist agent to answer the inquiry.
The Supervisor used one of the tools available to give the the ReturnDataSpecialist the order number for the item the customer wanted to return:
Customer John Smith wants to return the popcorn bowl from order BAJ1UHPC5W.
4. ReturnDataSpecialist response
Along with the Supervisor's final response, you can see every response formed by other agents on the team. In this example, the ReturnDataSpecialist used the order number it received from the Supervisor to form a response to the customer.