4.21 FM evaluation

Technical evaluation methods

• Benchmark datasets
• Human-in-the-loop evaluation
• Amazon Bedrock Model Evaluation
• Automated text metrics
• LLM-as-a-judge
• RAG retrieval-quality evaluation
• Agent workflow evaluation
• Safety, robustness, and adversarial testing

Text-generation metrics

Application-level evaluation

A good FM application is more than the base model.

Evaluate:

• Task completion rate
• User satisfaction
• Cost per interaction
• Response latency
• Groundedness
• Retrieval relevance
• Hallucination rate
• Correct tool usage
• Agent success and failure paths
• Escalation to humans
• User engagement
• Productivity improvement
• ROI

Exam trap

A high offline quality metric does not prove business success. Pair technical evaluation with business-alignment metrics.

Domain 4: Guidelines for Responsible AI

4.22 Responsible AI characteristics

Responsible AI includes:

• Fairness
• Inclusivity
• Robustness
• Safety
• Veracity and truthfulness
• Transparency
• Explainability
• Human-centered design
• Privacy
• Sustainability
• Accountability

4.23 Bias and fairness

Bias can arise from:

• Historical inequalities
• Unrepresentative data
• Poor labels
• Sampling problems
• Proxy variables
• Inadequate subgroup testing
• Drift after deployment

Responsible evaluation flow

1. Review the use case and potential harm.
2. Curate representative data.
3. Test overall performance.
4. Analyze subgroup performance.
5. Review labels and edge cases.
6. Add human review where needed.
7. Monitor after deployment.
8. Reassess when data, users, or business context change.

Exam trap

Overall accuracy can hide poor results for a demographic group. If the scenario mentions unequal group performance, choose subgroup analysis, fairness review, and mitigation.

4.24 Bias, variance, overfitting, and underfitting

4.25 Responsible-AI tools

4.26 Transparency and explainability

Use explainable and transparent approaches when:

• Decisions affect people materially.
• Regulators or auditors require justification.
• Users need a clear reason and appeal path.
• The organization must document intended use and limitations.

Human-centered explainability

Good explanations should be:

• Relevant to the affected user
• Understandable without unnecessary jargon
• Clear about the role of AI
• Honest about uncertainty and limitations
• Connected to feedback, appeal, or human-review mechanisms

4.27 Legal and trust risks

Before publishing GenAI content, consider:

• Intellectual-property claims
• Licensing restrictions
• Privacy leakage
• Biased output
• Hallucinations
• Customer trust
• Reputational harm
• End-user safety
• Inadequate disclosure
• Lack of human review for high-impact content

Sustainability rule

When two models satisfy the requirement, prefer the smaller or more efficient suitable model. Larger is not automatically better.

Domain 5: Security, Compliance, and Governance for AI Solutions

4.28 Shared responsibility

AWS secures the underlying cloud infrastructure. The customer remains responsible for choices such as:

• Data classification
• IAM policies
• Application configuration
• Access control
• Prompt and response handling
• Secure integrations
• Retention
• Residency
• Logging
• Governance processes

Managed services reduce operational burden. They do not eliminate customer responsibility.

4.29 Core security services

4.30 Secure data engineering

Good AI security begins before model inference:

• Assess data quality.
• Classify sensitive data.
• Apply access controls.
• Encrypt data at rest and in transit.
• Preserve data integrity.
• Document data origins.
• Minimize sensitive data sent to models.
• Apply retention and deletion policies.
• Validate retrieved content.
• Monitor data flows.

4.31 Data lineage and citations

Document:

• Where training and grounding data came from
• Who approved it
• How it was transformed
• Which model version used it
• Which source supported a generated answer
• When the source was updated
• Whether users are authorized to access it

Lineage improves trust, investigation, compliance review, and answer validation.

4.32 Securing GenAI applications

A layered pattern:

1. Authenticate the user.
2. Authorize access with least privilege.
3. Filter or validate input.
4. Protect system instructions.
5. Retrieve only authorized knowledge.
6. Treat retrieved text as data, not trusted instructions.
7. Restrict agent tool permissions.
8. Apply safeguards and output filtering.
9. Validate important outputs.
10. Log interactions and tool activity.
11. Monitor failures, toxicity, and anomalies.
12. Escalate high-risk cases to humans.

4.33 Hallucination reduction

Use several controls together:

• RAG grounding
• Current and approved knowledge sources
• Source citations
• Output validation
• Confidence scoring
• Clear prompt constraints
• Human review for high-impact decisions
• Monitoring and feedback loops

Exam trap

Grounding reduces risk but does not guarantee correctness.

4.34 Prompt injection and agent security

Agents can amplify risk because they may act on external systems.

Controls include:

• Narrow IAM permissions
• Tool allowlists
• Input validation
• Output validation
• Isolation of untrusted content
• Guardrails
• Logging
• Human approval for high-impact actions
• AgentCore Identity
• Policy in AgentCore
• Agent observability and evaluation

AgentCore memory rule

Memory improves contextual interactions but must be controlled. Store only appropriate information, separate users correctly, and avoid leaking data between sessions or tenants.

4.35 Governance and compliance services

4.36 Data governance

A governance policy should explicitly define:

• Data owners
• Approved sources
• Retention periods
• Deletion rules
• Data residency
• Logging
• Monitoring and observability
• Access reviews
• Review cadence
• Model and prompt versioning
• Human escalation
• Transparency expectations
• Training requirements for teams
• Incident response
• Change approval

Exam trap

“Retain everything forever” is not a sound default. Retention must reflect business, legal, privacy, and regulatory needs.

5. Service Selection Guide

5.1 Managed AI service quick map

5.2 Security and governance service quick map

5.3 Commonly confused pairs

6. Architecture Patterns

6.1 Pattern: Simple managed AI capability

Use when one managed service directly solves the problem.

Input → Purpose-built AWS AI service → Result

Examples:

• Audio file → Amazon Transcribe → transcript
• Written text → Amazon Polly → spoken audio
• Invoice scan → Amazon Textract → extracted tables and fields
• Customer review → Amazon Comprehend → sentiment and entities

Exam rule: do not propose an FM, custom model, or complex pipeline when a purpose-built managed service directly meets the need.

6.2 Pattern: Traditional ML prediction

Historical labeled data
        ↓
Prepare and evaluate data
        ↓
Train classification or regression model
        ↓
Deploy suitable inference endpoint
        ↓
Monitor performance and drift
        ↓
Retrain when needed

Use for:

• Fraud-risk prediction
• Numeric forecasting
• Discrete approval categories
• Narrow regulated predictions

6.3 Pattern: RAG knowledge assistant

Approved enterprise documents
        ↓
Chunking
        ↓
Embeddings
        ↓
Vector-capable data store
        ↓
User query → query embedding → retrieve relevant chunks
        ↓
Add grounded context to FM prompt
        ↓
Generate answer with citations
        ↓
Validate, monitor, and collect feedback

Use for:

• Internal policy assistant
• Product-documentation chatbot
• Research assistant
• Customer support knowledge base

Security additions:

• Retrieve only content the user may access.
• Keep system instructions separate from retrieved documents.
• Treat retrieved text as untrusted input.
• Log source retrieval and response generation.
• Add human escalation for uncertain answers.

6.4 Pattern: Prompt-managed production application

Versioned prompt template
        ↓
Representative evaluation set
        ↓
Controlled release
        ↓
Application traffic
        ↓
Quality, safety, latency, and cost monitoring
        ↓
Rollback or improve prompt version

Use Amazon Bedrock Prompt Management when prompt changes need repeatability and governance.

6.5 Pattern: Agentic workflow

Authenticated user request
        ↓
Agent receives goal and context
        ↓
Plan and tool selection
        ↓
Authorized tool invocation
        ↓
Inspect tool output
        ↓
Continue, stop, or request human approval
        ↓
Return result with logs and traceability

Add:

• Least-privilege permissions
• AgentCore Identity and Policy where appropriate
• Tool allowlists
• Memory boundaries
• Observability
• Evaluation
• Human approvals for high-impact actions

6.6 Pattern: Responsible AI review

Use-case risk assessment
        ↓
Representative and curated data
        ↓
Technical evaluation
        ↓
Subgroup analysis and bias review
        ↓
Safeguards and human-review path
        ↓
Transparent documentation
        ↓
Production monitoring
        ↓
Periodic review and remediation

6.7 Pattern: Governance and auditability

Policies and ownership
        ↓
IAM, encryption, private connectivity, secrets
        ↓
Logging and configuration tracking
        ↓
Audit evidence and compliance documentation
        ↓
Retention, residency, monitoring
        ↓
Periodic governance review

Relevant services:

• IAM
• AWS KMS
• AWS Secrets Manager
• AWS PrivateLink
• AWS CloudTrail
• AWS Config
• AWS Audit Manager
• AWS Artifact
• Amazon CloudWatch

7. Exam Traps and Elimination Rules

7.1 Wrong-service traps

Eliminate an option when it solves a different data modality or outcome:

• Amazon Polly cannot transcribe calls.
• Amazon Transcribe cannot generate speech.
• Amazon Rekognition is not the specialist service for extracting invoice tables.
• Amazon Textract is not the general-purpose object-recognition service.
• Amazon Comprehend does not translate text.
• Amazon Translate does not analyze sentiment.
• AWS Artifact does not collect your internal audit evidence.
• AWS Audit Manager does not provide private network connectivity.
• AWS KMS does not discover sensitive content in S3.
• Amazon Macie does not create encryption keys.

7.2 Overengineering traps

Prefer a lightweight solution unless the requirement justifies complexity.

7.3 Metrics traps

• Accuracy can be misleading with class imbalance.
• Precision matters when false positives are expensive.
• Recall matters when missing true positives is expensive.
• F1 is useful when balancing precision and recall.
• ROUGE is commonly associated with summaries.
• BLEU is commonly associated with translation.
• BERTScore reflects semantic similarity.
• Technical quality does not replace business metrics such as task completion, satisfaction, ROI, or cost per interaction.

7.4 GenAI traps

• Higher temperature increases diversity, not factuality.
• More context can hurt if it is irrelevant.
• RAG reduces hallucination risk but does not eliminate it.
• Fine-tuning changes model behavior; it is not the right tool for weekly knowledge updates.
• Agents are not automatically needed for every chatbot.
• MCP connects agents to systems; it does not remove the need for authorization.
• Memory improves context but creates privacy and isolation responsibilities.
• A bigger model is not automatically more cost-effective or responsible.

7.5 Security traps

Reject answers that:

• Grant administrator access without justification.
• Make a bucket public.
• Store secrets in prompts or source code.
• Disable logging.
• Trust retrieved text as instructions.
• Retain data forever by default.
• Ignore encryption in transit.
• Skip authorization for agent tools.
• Assume AWS owns all customer security responsibilities.

7.6 Responsible-AI traps

Reject answers that:

• Use only overall accuracy despite subgroup harm.
• Publish high-impact output with no human review.
• Assume generated content has no intellectual-property risk.
• Skip documentation.
• Treat initial evaluation as permanent.
• Select the largest model regardless of sustainability or need.
• Remove user feedback and appeal paths.

7.7 Multiple-response strategy

For multiple-response questions:

1. Evaluate each option independently.
2. Select only answers that directly satisfy the scenario.
3. Avoid choosing a technically true but irrelevant statement.
4. Watch for one correct control plus one missing-control distractor.
5. Prefer layered security when the scenario involves public GenAI or agents.

7.8 Ordering strategy

Common correct order:

Define objective → choose data and approach → evaluate → deploy with controls → monitor and improve

Common incorrect orders:

• Deploy before evaluation.
• Grant broad permissions first.
• Skip data review.
• Remove monitoring.
• Add governance only after a failure.

8. Quick Memory Rules

8.1 One-line concept rules

• AI is the umbrella; ML is a subset of AI.
• Deep learning uses multi-layer neural networks.
• GenAI creates content.
• Agentic AI plans, uses tools, and performs steps.
• Category output → classification.
• Numeric output → regression.
• Unlabeled groups → clustering.
• Labels → supervised learning.
• No labels → unsupervised learning.
• Rewards and penalties → reinforcement learning.
• Scheduled predictions → batch inference.
• Immediate prediction → real-time inference.
• Long-running request with later result → asynchronous inference.
• Sporadic traffic with low endpoint-management needs → serverless inference.
• Exact deterministic rule → conventional code or rules engine.
• Frequently changing private knowledge → RAG.
• Stable style or behavior change → fine-tuning.
• Lower cost and latency with acceptable retained behavior → distillation.
• Build broad capability from huge datasets → pre-training.
• Low temperature → consistent.
• High temperature → creative.
• Repeated stable prompt prefix → consider prompt caching.
• Tools and external systems for agents → MCP is relevant.
• Production agents → identity, policy, least privilege, memory boundaries, observability, and evaluation.
• Hallucination reduction → grounding, validation, citations, confidence scoring, and human review.
• False positives expensive → precision.
• Missed positives expensive → recall.
• Need balance → F1.

8.2 One-line AWS service rules

• Speech to text → Amazon Transcribe.
• Text to speech → Amazon Polly.
• Translation → Amazon Translate.
• Sentiment and entities → Amazon Comprehend.
• Intent and slots chatbot → Amazon Lex.
• Image and video objects → Amazon Rekognition.
• Document text, forms, and tables → Amazon Textract.
• Recommendations → Amazon Personalize.
• Managed FMs and GenAI building blocks → Amazon Bedrock.
• Broad ML platform → Amazon SageMaker AI.
• Pre-trained model templates → SageMaker JumpStart.
• AWS FM family → Amazon Nova.
• Ground Bedrock with enterprise data → Knowledge Bases for Amazon Bedrock.
• Vector search → OpenSearch, Aurora, RDS for PostgreSQL, or Neptune depending on requirements.
• Open-source agent SDK → Strands Agents.
• Production agent infrastructure → Amazon Bedrock AgentCore.
• Agentic coding with specs, code, tests, and docs → Kiro.
• AI assistance for AWS or developer work → Amazon Q.
• AI-powered work, research, dashboards, BI, and automation → Amazon Quick.
• Agentic migration and modernization → AWS Transform.
• Permissions → IAM.
• Encryption keys → AWS KMS.
• Secrets → AWS Secrets Manager.
• Sensitive-data discovery in S3 → Amazon Macie.
• Private service connectivity → AWS PrivateLink.
• AWS API history → AWS CloudTrail.
• Configuration compliance → AWS Config.
• Audit evidence collection → AWS Audit Manager.
• AWS compliance reports → AWS Artifact.
• Vulnerability findings → Amazon Inspector.
• AWS best-practice recommendations → AWS Trusted Advisor.
• Bias and explainability analysis → SageMaker Clarify.
• Human review → Amazon A2I.
• Model documentation → SageMaker Model Cards.
• GenAI safeguards → Amazon Bedrock Guardrails.

9. Final Revision Notes

9.1 Highest-value comparison table

9.2 April 2026 revision focus

Ensure you can recognize these additions and updates:

• Agentic AI as a foundational concept
• Asynchronous inference
• Serverless inference
• Traditional ML vs FM selection
• Token-based pricing
• Context engineering
• Multi-agent concepts
• MCP
• Tool usage, memory, and workflow orchestration
• Amazon Quick
• Kiro
• Strands Agents
• Amazon Bedrock AgentCore
• Prompt caching
• Model distillation
• Amazon Bedrock Prompt Management
• LLM-as-a-judge
• Business-alignment metrics
• Agent and workflow evaluation
• AgentCore Identity
• Policy in AgentCore
• Data leakage prevention
• Output filtering and validation
• AI interaction audit trails
• Toxicity handling
• Hallucination detection
• RAG grounding
• Confidence scoring
• AWS Transform
• Amazon Aurora as an in-scope service

9.3 Final ten-minute recall

Before ending revision, recite:

1. Classification, regression, clustering.
2. Batch, real-time, asynchronous, serverless inference.
3. Transcribe, Polly, Translate, Comprehend, Lex, Rekognition, Textract, Personalize.
4. Bedrock vs SageMaker AI.
5. Tokens, chunks, embeddings, vectors.
6. RAG vs fine-tuning vs pre-training vs distillation.
7. Temperature and maximum output length.
8. Prompt injection, jailbreaking, poisoning, MCP, memory, agent controls.
9. Clarify, A2I, Model Cards, Guardrails.
10. IAM, KMS, Macie, PrivateLink, CloudTrail, Config, Audit Manager, Artifact.

10. Exam-Day Checklist

Before starting

• Read every scenario for the true business requirement.
• Pay attention to words such as MOST appropriate, best fit, least operational overhead, lowest cost, regulated, real-time, and human review.
• Remember that several answers may be technically possible. Select the answer that directly meets the stated need with the fewest unnecessary components.

During the exam

• Answer every question; there is no penalty for guessing.
• Eliminate answers that solve the wrong modality or business problem.
• Eliminate unnecessary complexity.
• Prefer AWS managed capabilities when the scenario values simplicity or lower operational overhead.
• Check whether the workload needs predictions, generated content, current knowledge, or tool-using automation.
• For public GenAI or agents, look for layered safety and security controls.
• For regulated or high-impact scenarios, look for explainability, human review, documentation, and monitoring.
• For multiple-response questions, judge each option separately.
• For ordering questions, place evaluation before deployment and monitoring after deployment.
• For matching questions, use the service families to eliminate swaps.

Before submitting

• Return to unanswered questions.
• Review questions where two options remained plausible.
• Re-check directionality:
  • Transcribe vs Polly
  • Rekognition vs Textract
  • Artifact vs Audit Manager
  • CloudTrail vs Config
  • Precision vs recall
  • RAG vs fine-tuning
• Confirm that you did not choose a larger or more complex service without a requirement.

11. Official References

Use these AWS sources when reviewing the live exam scope:

• AWS Certified AI Practitioner (AIF-C01) exam guide:
  https://docs.aws.amazon.com/aws-certification/latest/ai-practitioner-01/ai-practitioner-01.html
• Revisions:
  https://docs.aws.amazon.com/aws-certification/latest/ai-practitioner-01/aif-01-revisions.html
• Domain 1 — Fundamentals of AI and ML:
  https://docs.aws.amazon.com/aws-certification/latest/ai-practitioner-01/ai-practitioner-01-domain1.html
• Domain 2 — Fundamentals of GenAI:
  https://docs.aws.amazon.com/aws-certification/latest/ai-practitioner-01/ai-practitioner-01-domain2.html
• Domain 3 — Applications of Foundation Models:
  https://docs.aws.amazon.com/aws-certification/latest/ai-practitioner-01/ai-practitioner-01-domain3.html
• Domain 4 — Guidelines for Responsible AI:
  https://docs.aws.amazon.com/aws-certification/latest/ai-practitioner-01/ai-practitioner-01-domain4.html
• Domain 5 — Security, Compliance, and Governance for AI Solutions:
  https://docs.aws.amazon.com/aws-certification/latest/ai-practitioner-01/ai-practitioner-01-domain5.html
• In-scope AWS services:
  https://docs.aws.amazon.com/aws-certification/latest/ai-practitioner-01/aif-01-in-scope-services.html

Closing study rule

Choose the simplest suitable approach, ground GenAI answers when knowledge matters, evaluate both model quality and business value, and never treat security or responsible AI as an afterthought.