Domain 1: Data Ingestion and Transformation : 34%
This is the largest domain. Expect scenarios that combine source type, latency, replay, processing choice, scheduling, failure recovery, and deployment.
Streaming ingestion
| Service or pattern |
Choose it when |
Do not choose it when |
| Amazon Kinesis Data Streams |
Multiple consumers need a retained stream, replay, ordered records within a shard, shard-based throughput, or enhanced fan-out |
The only requirement is managed delivery to S3 with low operational effort |
| Amazon Kinesis Data Firehose |
You need managed near-real-time delivery to destinations such as S3 with buffering and optional lightweight transformation |
Consumers must replay records from a durable stream or use custom stream-processing semantics |
| Amazon MSK |
Existing Kafka applications, partitions, consumer groups, Kafka client compatibility, or Kafka ecosystem tools must be preserved |
A simple managed delivery stream is enough |
| DynamoDB Streams |
You need item-level change events from a DynamoDB table |
You need general-purpose ingestion from arbitrary producers |
| AWS DMS with CDC |
You need ongoing inserts, updates, and deletes replicated from a database with low source impact |
You only need one static full extract or a SaaS connector |
Replayability rule
Replay requirement → think Kinesis Data Streams or Kafka/MSK, not Firehose.
Firehose is excellent for managed destination delivery. It is not the default answer when the scenario explicitly says consumers need to re-read the last several hours after fixing processing logic.
Fan-in and fan-out
- Fan-in: many producers write into a shared ingestion path.
- Fan-out: multiple consumers read the same stream for different purposes.
- Kinesis enhanced fan-out is relevant when consumers need dedicated read throughput and low-latency delivery without competing for shared stream read throughput.
- An SQS queue distributes messages among competing consumers. It does not automatically give every independent analytics application its own copy of every record.
Throttling and rate limits
Recognize these signals:
- HTTP 429 responses from APIs
- Kinesis
ProvisionedThroughputExceeded
- DynamoDB hot partitions or throttled requests
- RDS connection exhaustion
- downstream systems overloaded by Lambda concurrency
Use:
- exponential backoff with jitter;
- bounded retries;
- appropriate stream capacity or shard scaling;
- well-distributed partition keys;
- controlled Lambda concurrency;
- buffering with SQS when bursts must be absorbed.
Do not retry immediately in a tight loop. That amplifies the failure.
Batch and file ingestion
| Service |
Best fit |
| Amazon S3 |
Durable landing zone for files and raw lake data |
| Amazon AppFlow |
Low-code transfer between supported SaaS sources and AWS destinations, such as Salesforce to S3 |
| AWS DataSync |
Managed movement between supported file systems and object-storage locations |
| AWS Transfer Family |
Managed SFTP, FTPS, and FTP-style transfer access into AWS storage |
| AWS Snow Family |
Large offline or edge transfer when network transfer is impractical |
| AWS DMS |
Database migration, full load, CDC, or both |
Events versus schedules
| Requirement |
Best tool |
| Run when an object arrives in S3 |
S3 Event Notifications or EventBridge event routing |
| Run every day at 02:00 even if no object arrives |
EventBridge Scheduler or a time-based scheduler |
| Run crawlers or jobs according to a DAG schedule |
MWAA or a workflow-specific scheduler |
Trap: An S3 event cannot guarantee a daily run when no object arrives. A time schedule cannot provide immediate object-by-object responsiveness without polling.
APIs and connectivity
Know these concepts:
- consuming external data APIs;
- building data APIs for downstream consumers;
- Amazon API Gateway as an API front door;
- JDBC and ODBC connectivity to data sources;
- IP allowlists where a source requires them;
- retries, rate limits, authentication, and secret storage.
Select the processing engine
| Requirement |
Best starting choice |
Why |
| Small, short, event-driven transformation |
AWS Lambda |
Low operational overhead for bounded per-event work |
| Serverless Spark ETL over S3 data |
AWS Glue ETL |
Managed Spark, catalog integration, job scheduling and serverless operations |
| Highly customized distributed Spark or Hadoop workload |
Amazon EMR |
More control over clusters, frameworks, bootstrap actions and packages |
| Stateful streaming windows, sessions, rolling aggregates |
Amazon Managed Service for Apache Flink |
Continuous stateful stream processing |
| SQL transformations on warehouse staging tables |
Amazon Redshift SQL |
Efficient set-based transformations where the data already lives |
| Containerized custom workload |
Amazon ECS or Amazon EKS |
Useful when packaging, runtime control, dependencies, or Kubernetes requirements matter |
| Large parallel batch jobs |
AWS Batch |
Batch scheduling and compute provisioning for containerized jobs |
Glue versus EMR versus Lambda
Use this elimination rule:
- Choose Lambda for a small event handler.
- Choose Glue for managed serverless ETL, especially Spark over S3 and catalog-integrated workflows.
- Choose EMR when you need distributed open-source frameworks with deeper cluster-level control.
- Choose Flink when the computation is continuously stateful over streams.
Trap: Do not choose EMR merely because the word “data” appears. Do not choose Lambda for a massive distributed join. Do not choose Glue for a requirement that explicitly demands detailed cluster bootstrap control.
File formats and partitioning
CSV is readable but expensive for repeated analytics scans. Apache Parquet is columnar and typically reduces scanned bytes when queries read only a subset of columns.
For Athena and data-lake analytics:
- convert large analytical datasets to Parquet;
- compress appropriately;
- partition by common selective filters such as date, region, or source;
- avoid excessive tiny files;
- avoid partitioning by extremely high-cardinality fields when it creates too many small partitions.
Volume, velocity, and variety
| Characteristic |
Question to ask |
| Volume |
How much data must be stored or processed? |
| Velocity |
How quickly does data arrive and how quickly must it be processed? |
| Variety |
Is the data structured, semi-structured, unstructured, tabular, text, logs, images, or events? |
These dimensions help decide batch versus streaming, storage layout, processing engine, and cost model.
LLM-assisted processing
The revised guide includes integrating LLMs for data processing. The expected reasoning is practical:
- use Amazon Bedrock models in a controlled workflow for tasks such as summarization, categorization, enrichment, or extraction from unstructured text;
- validate generated output before downstream use;
- store prompts, outputs, metadata, and quality signals where traceability matters;
- do not assume a custom model must be trained from scratch.
Task 1.3 : Orchestrate data pipelines
| Service |
Choose it when |
Closest wrong answer |
| AWS Step Functions |
Serverless sequence, branching, retries, wait states, error handling, and service integrations |
EventBridge can trigger the workflow but is not the detailed state machine |
| Amazon MWAA |
Existing or complex Apache Airflow DAGs, backfills, dependencies, scheduling |
Step Functions is excellent for AWS-native state machines but is not an Airflow migration target |
| AWS Glue workflows and triggers |
Glue-native crawler and ETL-job dependency chain |
S3 Lifecycle changes storage classes; it does not orchestrate jobs |
| Amazon EventBridge and Scheduler |
Event routing or time-based triggering |
SQS buffers work but is not the time scheduler |
| Amazon SQS |
Buffer bursts, decouple producer and consumer, retry messages, use DLQ |
SNS is pub/sub notification fan-out, not durable queue buffering for workers |
| Amazon SNS |
Fan out alerts or notifications to subscribers |
SQS is better when workers must consume queued tasks |
Dead-letter queue and quarantine pattern
Use a DLQ or quarantine store when records repeatedly fail processing. Preserve:
- original payload or reference;
- failure reason;
- timestamp;
- pipeline stage;
- retry count;
- correlation identifier.
This allows investigation and controlled replay without blocking valid records.
Task 1.4 : Apply programming concepts
Know the following:
- use AWS SDKs and APIs instead of scraping the console;
- use AWS SAM for serverless application packaging;
- use AWS CloudFormation or AWS CDK for repeatable IaC deployment;
- use CI/CD services such as CodeBuild, CodeDeploy, and CodePipeline;
- use Lambda layers for shared dependencies when appropriate;
- mount EFS from Lambda when shared file-system access is required and properly configured;
- do not attach EBS volumes directly to Lambda;
- tune Lambda memory, timeout, concurrency, event-source batch size, and retry behavior;
- use logs and metrics during debugging;
- consider EC2 when unmanaged compute control is required;
- consider ECS/EKS for containerized processing.
Lambda concurrency rule
If a downstream database has limited connections, unlimited Lambda concurrency can create a failure storm. Configure a reasonable concurrency limit and tune the event-source behavior. Optimize the entire path, not only Lambda throughput.
