So you are getting into the agentic AI portion of the AI-102 exam and you want to understand what Microsoft actually means by "agentic solutions.. If you searched for Azure AI-102 agentic solutions, you're in the right place" This is the newest domain on the exam, covering only 5 to 10% of questions, but those questions are very specific. If you do not understand agent architecture, you will guess. Let us make sure you do not have to.
What Is an Agentic Solution?
An agentic solution is an AI system that can autonomously perform tasks on behalf of a user. Unlike a simple chatbot that responds to individual messages, an agent can:
- Plan a sequence of steps to achieve a goal
- Use tools to take actions (call APIs, query databases, execute code)
- Ground its responses in enterprise data
- Apply guardrails to stay within safety boundaries
- Evaluate its own outputs and escalate when uncertain
Think of an agent as an AI employee: it receives a goal, figures out how to achieve it, uses available tools, and knows when to ask for help.
Agent Architecture on Microsoft Foundry
Microsoft Foundry (formerly Azure AI Foundry) is the primary platform for building agentic solutions on Azure. Here is how the pieces fit together:
Core Components
Model. The brain of the agent. Foundry provides access to models from OpenAI (GPT-4, GPT-4o), Meta (Llama), Mistral, and others. You deploy a model in Foundry and connect it to your agent.
Tools. Actions the agent can take. Tools include:
- Function calls: Custom Python or REST API functions that the agent invokes
- Azure AI Search: The agent queries an index to retrieve relevant documents
- Built-in tools: Web search, file retrieval, code interpreter
Grounding data. The knowledge base the agent references. This is typically an Azure AI Search index containing vector embeddings of enterprise documents. The agent retrieves relevant context before generating responses.
Guardrails. Safety mechanisms that prevent harmful, off-policy, or hallucinated outputs. Guardrails include content safety filters, system prompt constraints, output validation, and escalation rules.
Orchestration. The logic that determines when to call tools, how to chain multiple steps, and when to return a final answer. Foundry provides prompt flow and agent service for orchestration.
How an Agent Conversation Works
A typical agent interaction follows this flow:
- User sends a message. The message enters the agent service.
- System prompt is applied. The system prompt defines the agent's role, available tools, and behavioral constraints.
- Model plans its response. The model decides whether to answer directly, query a tool, or ask a clarifying question.
- Tool is invoked (if needed). If the model decides it needs information, it calls a tool (for example, Azure AI Search to retrieve documents).
- Tool result is incorporated. The tool output is added to the conversation history.
- Model generates grounded response. Using the retrieved context, the model produces a final answer.
- Content safety filter applies. Output is checked against content safety policies before being returned to the user.
| Agent Component | Purpose | Example on Azure |
|---|---|---|
| Model | Reasoning, planning, generation | GPT-4 via Azure OpenAI in Foundry |
| Tools | Actions the agent can take | Azure AI Search, custom APIs, code interpreter |
| Grounding data | Enterprise knowledge for RAG | Azure AI Search with vector indexes |
| Guardrails | Safety and policy enforcement | Azure AI Content Safety, system prompts |
| Orchestration | Tool chaining and planning | Azure AI Foundry prompt flow, agent service |
| Evaluation | Test quality before and after deployment | Foundry evaluation tools, automated test suites |
Building an Agent: Step by Step
Step 1: Define the Agent's Purpose
Every agent needs a clear scope. A customer support agent has a different purpose than a data analysis agent. Write a detailed system prompt that includes:
- The agent's role and expertise
- Available tools and when to use each one
- Behavioral constraints (do not speculate, do not share personal data, stay on topic)
- Escalation criteria (when to transfer to a human)
Step 2: Set Up Grounding Data
Create an Azure AI Search index with vector search enabled. Ingest and chunk your enterprise documents, generate embeddings using Azure OpenAI's text-embedding model, and store them in vector fields. The agent will query this index to ground its responses.
Use semantic ranking to improve retrieval quality. Configure the semantic configuration to understand domain-specific language and rank results by meaning, not just keyword overlap.
Step 3: Define Tools
Register tools in the agent configuration. Each tool needs a name, description, parameter schema, and implementation. For example:
- search_knowledge_base: Queries Azure AI Search for relevant documents. Parameters: query string, top_k results.
- lookup_order: Queries an order management API for order details. Parameters: order_id string.
- escalate_to_human: Transfers the conversation to a human agent. Parameters: reason string, context summary.
Step 4: Configure Guardrails
Implement guardrails at multiple levels:
- Input filtering: Use Azure AI Content Safety to screen user inputs before processing
- System prompt constraints: Instruct the agent to refuse certain types of requests
- Output validation: Check agent responses for PII, harmful content, or off-topic drift
- Escalation rules: Transfer to a human agent when confidence is below a threshold or the request is outside the agent's scope
Step 5: Test and Evaluate
Before deploying, test the agent comprehensively:
- Task completion: Can the agent successfully complete representative user requests?
- Tool use accuracy: Does the agent choose the right tool for each situation?
- Grounding quality: Are responses factually accurate and properly cited?
- Safety compliance: Does the agent refuse inappropriate requests and escalate correctly?
- Edge cases: How does the agent handle ambiguous, multi-step, or out-of-scope requests?
Use Azure AI Foundry's evaluation tools to run automated test suites. Measure precision (are the retrieved documents relevant?), recall (did the agent find all relevant information?), and safety (were any harmful outputs produced?).
Exam-Focused Agent Scenarios
The AI-102 tests agentic solutions in predictable ways:
Scenario 1: Agent needs enterprise knowledge Answer: Use Azure AI Search with vector and semantic search as the grounding layer. The agent retrieves relevant documents at query time.
Scenario 2: Agent needs to take action via an API Answer: Define a tool that calls the REST API. The model decides when to invoke it based on the conversation context.
Scenario 3: Agent must not produce harmful content Answer: Apply Azure AI Content Safety filters on both input and output. Include safety instructions in the system prompt.
Scenario 4: Agent must escalate to humans Answer: Define an escalation tool with clear criteria. The agent invokes it when it cannot resolve the request or when the request is outside its scope.
Scenario 5: Testing an agent before deployment Answer: Evaluate task completion, tool use accuracy, grounding quality, and safety compliance. Bounding box metrics are for object detection, not agent workflows.
Common Agent Anti-Patterns
The exam loves to describe anti-patterns and ask you to identify the problem:
No grounding data. An agent that answers from model knowledge alone will hallucinate enterprise-specific information. Always ground with Azure AI Search or similar retrieval.
No guardrails. Without content safety and system prompt constraints, the agent may produce harmful, biased, or off-policy responses.
No escalation path. An agent that never escalates will either give wrong answers or frustrate users with "I cannot help with that" responses for everything.
Over-reliance on a single tool. Good agents use multiple tools flexibly. An agent that only searches and never takes actions is limited.
Ignoring tool results. Sometimes the model generates a response without properly incorporating tool output. Test for this specifically.
FAQ
What is the difference between an agent and a chatbot?
A chatbot responds to individual messages in a linear conversation. An agent can plan multi-step actions, use tools autonomously, ground responses in enterprise data, and escalate decisions. Agents are more flexible and capable than simple chatbots.
What Azure service is used for agent grounding?
Azure AI Search is the primary grounding service for agents. It provides vector search, semantic ranking, and hybrid retrieval over enterprise document collections.
How do you test an AI agent?
Test task completion, tool use accuracy, grounding quality, and safety compliance. Use Azure AI Foundry's evaluation tools to run automated test suites. Do not use computer vision metrics (like bounding boxes) for agent testing.
What are agent guardrails?
Guardrails are safety mechanisms that prevent harmful, off-policy, or inaccurate outputs. They include content safety filters, system prompt constraints, output validation rules, and escalation paths to human agents.
What is the difference between prompt flow and agent service?
Prompt flow is a visual tool for designing, testing, and deploying prompt-based workflows. The agent service provides autonomous tool-use capabilities. Agents use prompt flow-like orchestration but add autonomous decision-making.
How do agents handle tools they were not explicitly trained on?
Agents use the tool descriptions and parameter schemas provided in their configuration. The model reads these descriptions at runtime and decides which tool to invoke based on the current conversation context.
Master agentic solutions and you will own 5 to 10% of the AI-102 that most candidates get wrong. Practice at cert-pass.com/exams/azure-ai-102-azure-ai-engineer-associate/take. Full prep with 1000+ questions starts at EUR 29.
