Evaluating single-turn agent interactions follows a sample that the majority groups perceive effectively. You present an enter, acquire the output, and decide the end result. Frameworks like Strands Analysis SDK make this course of systematic by evaluators that assess helpfulness, faithfulness, and power utilization. In a earlier weblog publish, we coated tips on how to construct complete analysis suites for AI brokers utilizing these capabilities. Nevertheless, manufacturing conversations not often cease at one flip.
Actual customers have interaction in exchanges that unfold over a number of turns. They ask follow-up questions when solutions are incomplete, change course when new info surfaces, and categorical frustration when their wants go unmet. A journey assistant that handles “Ebook me a flight to Paris” effectively in isolation may wrestle when the identical person follows up with “Truly, can we take a look at trains as an alternative?” or “What about lodges close to the Eiffel Tower?” Testing these dynamic patterns requires greater than static take a look at instances with mounted inputs and anticipated outputs.
The core problem is scale as a result of you may’t manually conduct a whole lot of multi-turn conversations each time your agent adjustments, and writing scripted dialog flows locks you into predetermined paths that miss how actual customers behave. What analysis groups want is a technique to generate lifelike, goal-driven customers programmatically and allow them to converse naturally with an agent throughout a number of turns. On this publish, we discover how ActorSimulator in Strands Evaluations SDK addresses this problem with structured person simulation that integrates into your analysis pipeline.
Why multi-turn analysis is basically tougher
Single-turn analysis has a simple construction. The enter is understood forward of time, the output is self-contained, and the analysis context is restricted to that single trade. Multi-turn conversations break each one in all these assumptions.
In a multi-turn interplay, every message is dependent upon all the pieces that got here earlier than it. The person’s second query is formed by how the agent answered the primary. A partial reply attracts a follow-up about no matter was overlooked, a misunderstanding leads the person to restate their authentic request, and a shocking suggestion can ship the dialog in a brand new course.
These adaptive behaviors create dialog paths that may’t be predicted at test-design time. A static dataset of I/O pairs, irrespective of how massive, can’t seize this dynamic high quality as a result of the “right” subsequent person message is dependent upon what the agent simply stated.
Guide testing covers this hole in idea however fails in apply. Testers can conduct lifelike multi-turn conversations, however doing so for each state of affairs, throughout each persona sort, after each agent change just isn’t sustainable. Because the agent’s capabilities develop, the variety of dialog paths grows combinatorially, effectively past what groups can discover manually.
Some groups flip to immediate engineering as a shortcut, asking a big language mannequin (LLM) to “act like a person” throughout testing. With out structured persona definitions and express aim monitoring, these approaches produce inconsistent outcomes. The simulated person’s habits drifts between runs, making it tough to match evaluations over time or determine real regressions versus random variation. A structured strategy to person simulation can bridge this hole by combining the realism of human dialog with the repeatability and scale of automated testing.
What makes a very good simulated person
Simulation-based testing is effectively established in different engineering disciplines. Flight simulators take a look at pilot responses to eventualities that may be harmful or not possible to breed in the actual world. Sport engines use AI-driven brokers to discover tens of millions of participant habits paths earlier than launch. The identical precept applies to conversational AI. You create a managed setting the place lifelike actors work together together with your system underneath circumstances you outline, then measure the outcomes.
For AI agent analysis, a helpful simulated person begins with a constant persona. One which behaves like a technical skilled in a single flip and a confused novice within the subsequent produces unreliable analysis knowledge. Consistency means to keep up the identical communication fashion, experience degree, and character traits by each trade, simply as an actual particular person would.
Equally vital is goal-driven habits. Actual customers come to an agent with one thing they need to accomplish. They persist till they obtain it, modify their strategy when one thing just isn’t working, and acknowledge when their aim has been met. With out express targets, a simulated person tends to both finish conversations too early or proceed asking questions indefinitely, neither of which displays actual utilization.
The simulated person should additionally reply adaptively to what the agent says, not observe a predetermined script. When the agent asks a clarifying query, the actor ought to reply it in character. If the response is incomplete, the actor follows up on no matter was overlooked slightly than shifting on. If the dialog drifts off matter, the actor steers it again towards the unique aim. These adaptive behaviors make simulated conversations beneficial as analysis knowledge as a result of they train the identical dialog dynamics your agent faces in manufacturing.
Constructing persona consistency, aim monitoring, and adaptive habits right into a simulation framework is what differentiates structured person simulation from ad-hoc prompting. ActorSimulator in Strands Evals is designed round precisely these rules.
How ActorSimulator works
ActorSimulator implements these simulation qualities by a system that wraps a Strands Agent configured to behave as a practical person persona. The method begins with profile era. Given a take a look at case containing an enter question and an non-obligatory process description, ActorSimulator makes use of an LLM to create a whole actor profile. A take a look at case with enter “I need assistance reserving a flight to Paris” and process description “Full flight reserving underneath funds” may produce a budget-conscious traveler with beginner-level expertise and an off-the-cuff communication fashion. Profile era provides every simulated dialog a definite, constant character.
With the profile established, the simulator manages the dialog flip by flip. It maintains the complete dialog historical past and generates every response in context, maintaining the simulated person’s habits aligned with their profile and targets all through. When your agent addresses solely a part of the request, the simulated person naturally follows up on the gaps. A clarifying query out of your agent will get a response that stays in keeping with the persona. The dialog feels natural as a result of each response displays each the actor’s persona and all the pieces stated to this point.
Purpose monitoring runs alongside the dialog. ActorSimulator features a built-in aim completion evaluation software that the simulated person can invoke to judge whether or not their authentic goal has been met. When the aim is glad or the simulated person determines that the agent can’t full their request, the simulator emits a cease sign and the dialog ends. If the utmost flip rely is reached earlier than the aim is met, the dialog additionally stops. This offers you a sign that the agent may not be resolving person wants effectively. This mechanism makes certain conversations have a pure endpoint slightly than operating indefinitely or reducing off arbitrarily.
Every response from the simulated person additionally contains structured reasoning alongside the message textual content. You’ll be able to examine why the simulated person selected to say what they stated, whether or not they have been following up on lacking info, expressing confusion, or redirecting the dialog. This transparency is effective throughout analysis growth as a result of you may see the reasoning behind every flip, making it extra easy to hint the place conversations succeed or go off observe.
Getting began with ActorSimulator
To get began, you’ll need to put in the Strands Analysis SDK utilizing: pip set up strands-agents-evals. For a step-by-step setup, you may confer with our documentation or our earlier weblog for extra particulars. Placing these ideas into apply requires minimal code. You outline a take a look at case with an enter question and a process description that captures the person’s aim. ActorSimulator handles profile era, dialog administration, and aim monitoring robotically.
The next instance evaluates a journey assistant agent by a multi-turn simulated dialog.
from strands import Agent
from strands_evals import ActorSimulator, Case, Experiment
# Outline your take a look at case
case = Case(
enter=”I need to plan a visit to Tokyo with lodge and actions”,
metadata={“task_description”: “Full journey bundle organized”}
)
# Create the agent you need to consider
agent = Agent(
system_prompt=”You’re a useful journey assistant.”,
callback_handler=None
)
# Create person simulator from take a look at case
user_sim = ActorSimulator.from_case_for_user_simulator(
case=case,
max_turns=5
)
# Run the multi-turn dialog
user_message = case.enter
conversation_history = []
whereas user_sim.has_next():
# Agent responds to person
agent_response = agent(user_message)
agent_message = str(agent_response)
conversation_history.append({
“position”: “assistant”,
“content material”: agent_message
})
# Simulator generates subsequent person message
user_result = user_sim.act(agent_message)
user_message = str(user_result.structured_output.message)
conversation_history.append({
“position”: “person”,
“content material”: user_message
})
print(f”Dialog accomplished in {len(conversation_history) // 2} turns”)
The dialog loop continues till has_next() returns False, which occurs when the simulated person’s targets are met or simulated person determines that the agent can’t full the request or the utmost flip restrict is reached. The ensuing conversation_history accommodates the complete multi-turn transcript, prepared for analysis.
Integration with analysis pipelines
A standalone dialog loop is helpful for fast experiments, however manufacturing analysis requires capturing traces and feeding them into your evaluator pipeline. The following instance combines ActorSimulator with OpenTelemetry telemetry assortment and Strands Evals session mapping. The duty operate runs a simulated dialog and collects spans from every flip, then maps them right into a structured session for analysis.
from opentelemetry.sdk.hint.export import BatchSpanProcessor
from opentelemetry.sdk.hint.export.in_memory_span_exporter import InMemorySpanExporter
from strands import Agent
from strands_evals import ActorSimulator, Case, Experiment
from strands_evals.evaluators import HelpfulnessEvaluator
from strands_evals.telemetry import StrandsEvalsTelemetry
from strands_evals.mappers import StrandsInMemorySessionMapper
# Setup telemetry for capturing agent traces
telemetry = StrandsEvalsTelemetry()
memory_exporter = InMemorySpanExporter()
span_processor = BatchSpanProcessor(memory_exporter)
telemetry.tracer_provider.add_span_processor(span_processor)
def evaluation_task(case: Case) -> dict:
# Create simulator
user_sim = ActorSimulator.from_case_for_user_simulator(
case=case,
max_turns=3
)
# Create agent
agent = Agent(
system_prompt=”You’re a useful journey assistant.”,
callback_handler=None
)
# Accumulate spans throughout dialog
all_target_spans = []
user_message = case.enter
whereas user_sim.has_next():
memory_exporter.clear()
agent_response = agent(user_message)
agent_message = str(agent_response)
# Seize telemetry
turn_spans = checklist(memory_exporter.get_finished_spans())
all_target_spans.lengthen(turn_spans)
# Generate subsequent person message
user_result = user_sim.act(agent_message)
user_message = str(user_result.structured_output.message)
# Map to session for analysis
mapper = StrandsInMemorySessionMapper()
session = mapper.map_to_session(
all_target_spans,
session_id=”test-session”
)
return {“output”: agent_message, “trajectory”: session}
# Create analysis dataset
test_cases = [
Case(
name=”booking-simple”,
input=”I need to book a flight to Paris next week”,
metadata={
“category”: “booking”,
“task_description”: “Flight booking confirmed”
}
)
]
evaluator = HelpfulnessEvaluator()
dataset = Experiment(instances=test_cases, evaluator=evaluator)
# Run evaluations
report = Experiment.run_evaluations(evaluation_task)
report.run_display()
This strategy captures full traces of your agent’s habits throughout dialog turns. The spans embrace software calls, mannequin invocations, and timing info for each flip within the simulated dialog. By mapping these spans right into a structured session, you make the complete multi-turn interplay out there to evaluators like GoalSuccessRateEvaluator and HelpfulnessEvaluator, which may then assess the dialog as an entire, slightly than remoted turns.
Customized actor profiles for focused testing
Computerized profile era covers most analysis eventualities effectively, however some testing targets require particular personas. You may need to confirm that your agent handles an impatient skilled person otherwise from a affected person newbie, or that it responds appropriately to a person with domain-specific wants. For these instances, ActorSimulator accepts a completely outlined actor profile that you simply management.
from strands_evals.sorts.simulation import ActorProfile
from strands_evals import ActorSimulator
from strands_evals.simulation.prompt_templates.actor_system_prompt import (
DEFAULT_USER_SIMULATOR_PROMPT_TEMPLATE
)
# Outline a customized actor profile
actor_profile = ActorProfile(
traits={
“character”: “analytical and detail-oriented”,
“communication_style”: “direct and technical”,
“expertise_level”: “skilled”,
“patience_level”: “low”
},
context=”Skilled enterprise traveler with elite standing who values effectivity”,
actor_goal=”Ebook enterprise class flight with particular seat preferences and lounge entry”
)
# Initialize simulator with customized profile
user_sim = ActorSimulator(
actor_profile=actor_profile,
initial_query=”I must e book a enterprise class flight to London subsequent Tuesday”,
system_prompt_template=DEFAULT_USER_SIMULATOR_PROMPT_TEMPLATE,
max_turns=10
)
By defining traits like persistence degree, communication fashion, and experience, you may systematically take a look at how your agent performs throughout completely different person segments. An agent that scores effectively with affected person, non-technical customers however poorly with impatient specialists reveals a particular high quality hole which you can tackle. Operating the identical aim throughout a number of persona configurations turns person simulation right into a software for understanding your agent’s strengths and weaknesses by person sort.
Finest practices for simulation-based analysis
These finest practices assist you to get essentially the most out of simulation-based analysis:
- Set max_turns primarily based on process complexity, utilizing 3-5 for targeted duties and 8-10 for multi-step workflows. If most conversations attain the restrict with out finishing the aim, enhance it.
- Write particular process descriptions that the simulator can consider towards. “Assist the person e book a flight” is simply too obscure to guage completion reliably, whereas “flight reserving confirmed with dates, vacation spot, and value” provides a concrete goal.
- Use auto-generated profiles for broad protection throughout person sorts and customized profiles to breed particular patterns out of your manufacturing logs, similar to an impatient skilled or a first-time person.
- Give attention to patterns throughout your take a look at suite slightly than particular person transcripts. Constant redirects from the simulated person means that the agent is drifting off matter, and declining aim completion charges after an agent change factors to a regression.
- Begin with a small set of take a look at instances protecting your most typical eventualities and broaden to edge instances and extra personas as your analysis apply matures.
Conclusion
We confirmed how ActorSimulator in Strands Evals permits systematic, multi-turn analysis of conversational AI brokers by lifelike person simulation. Reasonably than counting on static take a look at instances that seize solely single exchanges, you may outline targets and personas and let simulated customers work together together with your agent throughout pure, adaptive conversations. The ensuing transcripts feed immediately into the identical analysis pipeline that you simply use for single-turn testing, providing you with helpfulness scores, aim success charges, and detailed traces throughout each dialog flip.
To get began, discover the working examples within the Strands Brokers samples repository. For groups evaluating brokers deployed by Amazon Bedrock AgentCore, the next AgentCore evaluations pattern reveal tips on how to simulate interactions with deployed brokers. Begin with a handful of take a look at instances representing your most typical person eventualities, run them by ActorSimulator, and consider the outcomes. As your analysis apply matures, broaden to cowl extra personas, edge instances, and dialog patterns.
In regards to the authors
Ishan Singh
Ishan is a Sr. Utilized Scientist at Amazon Internet Providers, the place he helps prospects construct modern and accountable generative AI options and merchandise. With a robust background in AI/ML, Ishan focuses on constructing Generative AI options that drive enterprise worth. Outdoors of labor, he enjoys taking part in volleyball, exploring native bike trails, and spending time along with his spouse and canine, Beau.
Jonathan Buck
Jonathan is a Senior Software program Engineer at Amazon Internet Providers. His work focuses on constructing agent environments, analysis, and post-training infrastructure to help the productization of agentic methods.
Vinayak Arannil
Vinayak is a Sr. Utilized Scientist from the Amazon Bedrock AgentCore workforce. With a number of years of expertise, he has labored on numerous domains of AI like laptop imaginative and prescient, pure language processing, suggestion methods and so forth. Presently, Vinayak helps construct new capabilities on the AgentCore and Strands, enabling prospects to judge their Agentic functions with ease, accuracy and effectivity.
Abhishek Kumar
Abhishek is an Utilized Scientist at AWS, working on the intersection of synthetic intelligence and machine studying, with a give attention to agent observability, simulation, and analysis. His main analysis pursuits middle on agentic conversational methods. Previous to his present position, Abhishek spent two years at Alexa, Amazon, the place he contributed to constructing and coaching fashions that powered Alexa’s core capabilities.
