Understand observability in AI Agents for Customers
Last Update: Feb 2025 • Est. Read Time: 4 MINObservability refers to the ability to monitor, track, and understand the performance of your AI Agent teams. The key feature of our observability tool is traces, which provide a detailed view of how the AI Agent processes an inbound message from start to finish. Traces allow you to see the journey a request takes through different parts of the AI system, giving you a granular look at each step in the process.
By analyzing traces, you can gain insight into how effectively an AI Agent Team handles customer interactions, identify issues, and make improvements to ensure optimal performance.
| Who can access this feature? | |
| User types | Admins can access the AI Agent Studio page. |
Learn more about AI Agents in this article.
In this article
What are traces?
Traces follow a request's lifecycle as it moves through your AI Agent. From the moment an inbound message is received, traces log each interaction, decision point, and operation the AI performs. This includes accessing tools and retrieving or updating data in Kustomer.
Each trace provides a timestamped breakdown of the following:
- Any articles in your public knowledge base that were used to respond to the inquiry.
- Reasoning operations provide insight into how the AI arrived at a decision or why it took a particular action while handling a customer inquiry.
- The message each AI Agent received from the supervisor.
- Checks to ensure that both default and custom guardrails are followed.
View traces for an AI Agent team
You can view traces for both AI Agent teams that are currently deployed and past versions of a team.
- From the AI Agents Studio page:
- Go to Settings > Kustomer IQ > AI Agent Studio.
- Select View Traces
for the team whose performance you want to analyze. By default, you will see a list of traces for all conversations answered by the latest deployed team.
- From directly within a conversation:
- Go to a conversation answered by an AI Agent and select Conversation options
. - Then, select Observe traces.
- Go to a conversation answered by an AI Agent and select Conversation options
You can select any trace to see details of what took place for every inbound message in the conversation.
How to read a trace
Note: The results you see in your traces may vary based on how you have your AI Agent team set up.
A trace shows all the operations that occurred to generate the final response the AI sent to the customer. You can select each operation to see details, such as which help article was used and how the AI reasoned its response.
In this section, we will walk through the sample trace below to see what information you can gather from this feature.
1. E-Comm Support Supervisor response
You can start to review a trace by selecting the Supervisor agent at the top. Here, you can see that the inbound message from the customer was "I would like to return the popcorn bowl." and that the AI responded with:
Hi John! I've initiated the return process for the popcorn bowl. Could you please let me know the reason for your return? This will help me assist you better!
2. Knowledge search
Once an inbound message comes in, the AI performs a Knowledge Search operation to see if there is a help article that can answer the inquiry. If help articles are found, you will see the article's name, the content the AI extracted from it, and a direct link to it.
3. Reasoning
The Reasoning operation gives you visibility into the AI’s internal decision-making process and helps you understand why the AI did what it did. Here, you can see that the AI decided to work with theReturnDataSpecialist agent to answer the inquiry.
The Supervisor used one of the tools available to give the ReturnDataSpecialist the order number for the item the customer wanted to return:
Customer John Smith wants to return the popcorn bowl from order BAJ1UHPC5W.
4. ReturnDataSpecialist response
Along with the Supervisor's final response, you can see every response formed by other agents on the team. In this example, the ReturnDataSpecialist used the order number it received from the Supervisor to form a response to the customer.
Understand guardrails
By setting guardrails, you can prevent your AI agents from mentioning or discussing competitors in customer interactions. In addition to protecting against mentions of competitors, these guardrails work alongside standard safeguards within the product to ensure AI agents don’t disclose sensitive information. You can see if any guardrails were detected in the trace for each message.
The following guardrails are available for AI Agents.
| Guardrail | Definition |
| Competitor | Detects only explicit mentions of the companies defined in the Competitor Guidance field for the team's settings. It's fine to mention other company names in a neutral or positive way. |
| Toxicity | Detects toxic messages including profanities, obscenities, threats, insults, and identity attacks, including examples of obfuscated profanities using alphanumeric and non-alphanumeric characters. |
| NSFW | Detects content that is not safe for work and non-inclusive, including sexual comments or offensive notes around age, gender, or other categories and protected classes. |
| Secrets | Detects proprietary information used to access confidential systems like passwords, private keys, or other credentials. This guardrail does not flag messages related to verification processes, such as "six-digit passcode", credit card numbers, email addresses, phone numbers, or other personal information. |