Data-Engineer-Associate Questions and Answers

 This solution meets the requirements of implementing real-time analytics capabilities with the least operational overhead. By creating an external schema in Amazon Redshift, you can access the data from Kinesis Data Streams using SQL queries without having to load the data into the cluster. By creating a materialized view on top of the stream, you can store the results of the query in the cluster and make them available for analysis. By setting the materialized view to auto refresh, you can ensure that the view is updated with the latest data from the stream at regular intervals. This way, you can derive near real-time insights by using existing BI and analytics tools. References:

Amazon Redshift streaming ingestion

Creating an external schema for Amazon Kinesis Data Streams

Creating a materialized view for Amazon Kinesis Data Streams

AWS Glue Data Quality transforms allow you to define built-in rules like IsComplete for null validation and ReferentialIntegrity for relationship validation—all with minimal code and operational overhead.

“Use AWS Glue Data Quality rules such as IsComplete and ReferentialIntegrity within ETL jobs to automatically validate incoming data.”

The requirement is to detect violations of data access policies and receive notifications with the username of the violator. AWS CloudTrail can provide object-level tracking for S3 to capture detailed API actions on specific S3 objects, including the user who performed the action.

AWS CloudTrail:

CloudTrail can monitor API calls made to an S3 bucket, including object-level API actions such as GetObject, PutObject, and DeleteObject. This will help detect access violations based on the API calls made by different users.

CloudTrail logs include details such as the user identity, which is essential for meeting the requirement of including the username in notifications.

The CloudTrail logs can be forwarded to Amazon CloudWatch to trigger alarms based on certain access patterns (e.g., violations of specific policies).

When processing many small files, the overhead of managing partitions can degrade performance. Rather than scaling workers manually, enabling AWS Glue auto scaling dynamically adjusts worker count based on resource requirements, which is cost-effective and addresses the performance concern.

“AWS Glue auto scaling automatically adds or removes workers based on the workload, which is efficient for handling performance bottlenecks when working with many small files.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

Manually scaling up workers or increasing their size (A, B) can be more expensive and less adaptive.

The performance issue in Amazon Redshift Spectrum queries arises due to the nature of CSV files, which are row-based storage formats. Spectrum is more optimized for columnar formats, which significantly improve performance by reducing the amount of data scanned. Also, partitioning data based on relevant columns like order date can further reduce the amount of data scanned, as queries can focus only on the necessary partitions.

A. Configure the third-party application to create the files in a columnar format:

Columnar formats (like Parquet or ORC) store data in a way that is optimized for analytical queries because they allow queries to scan only the columns required, rather than scanning all columns in a row-based format like CSV.

Amazon Redshift Spectrum works much more efficiently with columnar formats, reducing the amount of data that needs to be scanned, which improves query performance.

Option A is correct because AWS Glue is a serverless ETL service built to run PySpark workloads at scale with minimal infrastructure management. AWS documentation states that you can install additional Python modules and libraries for use with AWS Glue ETL jobs, including by using the --additional-python-modules parameter and Amazon S3 paths for wheel artifacts or other supported package delivery methods. That directly addresses the requirement for custom internal libraries. Since the data is already in Amazon S3 and the result must be loaded into Amazon Redshift, Glue is a natural low-overhead fit for this daily transformation pipeline.

Option B and C require managing compute infrastructure or cluster lifecycle, which increases operational overhead. Option D is not the best fit because SageMaker Processing is designed primarily for ML-oriented data preparation, not as the standard AWS service for large-scale ETL into Redshift. The question explicitly asks for least operational overhead with scalable PySpark and custom libraries, and AWS Glue provides exactly that managed capability.

To achieve the highest throughput and efficiently use cluster resources while loading data into an Amazon Redshift cluster, the optimal approach is to use a single COPY command that ingests data in parallel.

Option D: Use a single COPY command to load the data into the Redshift cluster.The COPY command is designed to load data from multiple files in parallel into a Redshift table, using all the cluster nodes to optimize the load process. Redshift is optimized for parallel processing, and a single COPY command can load multiple files at once, maximizing throughput.

Options A, B, and C either involve unnecessary complexity or inefficient approaches, such as using multiple COPY commands or INSERT statements, which are not optimized for bulk loading.

Option C is the correct answer because the Map state is designed to process a collection of data in parallel by applying the same transformation to each element. The Map state can invoke a nested workflow for each element, which can be another state machine or a Lambda function. The Map state will wait until all the parallel executions are completed before moving to the next state.

Option A is incorrect because the Parallel state is used to execute multiple branches of logic concurrently, not to process a collection of data. The Parallel state can have different branches with different logic and states, whereas the Map state has only one branch that is applied to each element of the collection.

Option B is incorrect because the Choice state is used to make decisions based on a comparison of a value to a set of rules. The Choice state does not process any data or invoke any nested workflows.

Option D is incorrect because the Wait state is used to delay the state machine from continuing for a specified time. The Wait state does not process any data or invoke any nested workflows.

AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide, Chapter 5: Data Orchestration, Section 5.3: AWS Step Functions, Pages 131-132

Building Batch Data Analytics Solutions on AWS, Module 5: Data Orchestration, Lesson 5.2: AWS Step Functions, Pages 9-10

AWS Documentation Overview, AWS Step Functions Developer Guide, Step Functions Concepts, State Types, Map State, Pages 1-3

Option C is correct because the fastest design combines a columnar storage format with partitioning on the most common query predicate. AWS Athena guidance says that Parquet and ORC are optimized columnar formats and that columns frequently used as filters are good candidates for partitioning. AWS also states that when a query filters on a partition key, the engine reads only matching partitions, which significantly reduces data scanned. Since the company’s main analytical need is insights specific to sensor categories, partitioning by sensor category provides the strongest pruning. Sorting by date then helps organize time-based data within each category partition.

Options B and D use CSV, which is row-based and much slower for analytical scans than Parquet or ORC. Option A uses a good format, but partitioning by date is less optimal than partitioning by sensor category when category is the main filter in the company’s queries. The study guide also identifies Parquet as the preferred analytic storage format for efficient columnar queries.

Problem Analysis:

The company uses AWS Glue for ETL pipelines and requires automatic data quality checks during pipeline execution.

The solution must integrate with existing AWS Glue pipelines and evaluate data quality rules based on predefined thresholds.

Key Considerations:

Ensure minimal implementation effort by leveraging built-in AWS Glue features.

Use a standardized approach for defining and evaluating data quality rules.

Avoid custom libraries or external frameworks unless absolutely necessary.

Solution Analysis:

Option A: SQL Transform

Adding SQL transforms to define and evaluate data quality rules is possible but requires writing complex queries for each rule.

Increases operational overhead and deviates from Glue ' s declarative approach.

Option B: Evaluate Data Quality Transform with DQDL

AWS Glue provides a built-in Evaluate Data Quality transform.

Allows defining rules in Data Quality Definition Language (DQDL), a concise and declarative way to define quality checks.

Fully integrated with Glue Studio, making it the least effort solution.

Option C: Custom Transform with PyDeequ

PyDeequ is a powerful library for data quality checks but requires custom code and integration.

Increases implementation effort compared to Glue ' s native capabilities.

Option D: Custom Transform with Great Expectations

Great Expectations is another powerful library for data quality but adds complexity and external dependencies.

Final Recommendation:

Use Evaluate Data Quality transform in AWS Glue.

Define rules in DQDL for checking thresholds, null values, or other quality criteria.

This approach minimizes development effort and ensures seamless integration with AWS Glue.

AWS Glue Data Quality Overview

DQDL Syntax and Examples

Glue Studio Transformations

Problem Analysis:

The company uses DynamoDB for gaming data storage and needs to ingest data into Amazon OpenSearch Service in near real time.

Data updates must propagate quickly to OpenSearch for analytics or search purposes.

Key Considerations:

DynamoDB Streams provide near-real-time capture of table changes (inserts, updates, and deletes).

Integration with AWS Lambda allows seamless processing of these changes.

OpenSearch offers APIs for indexing and updating documents, which Lambda can invoke.

Solution Analysis:

Option A: Step Functions with Periodic Export

Not suitable for near-real-time updates; introduces significant latency.

Operationally complex to manage periodic exports and S3 data ingestion.

Option B: AWS Glue Job

AWS Glue is designed for ETL workloads but lacks real-time processing capabilities.

Option C: DynamoDB Streams + Lambda

DynamoDB Streams capture changes in near real time.

Lambda can process these streams and use the OpenSearch API to update the index.

This approach provides low latency and seamless integration with minimal operational overhead.

Option D: Custom OpenSearch Plugin

Writing a custom plugin adds complexity and is unnecessary with existing AWS integrations.

Implementation Steps:

Enable DynamoDB Streams for the relevant DynamoDB tables.

Create a Lambda function to process stream records:

Parse insert, update, and delete events.

Use OpenSearch APIs to index or update documents based on the event type.

Set up a trigger to invoke the Lambda function whenever there are changes in the DynamoDB Stream.

Monitor and log errors for debugging and operational health.

Amazon DynamoDB Streams Documentation

AWS Lambda and DynamoDB Integration

Amazon OpenSearch Service APIs

Option B is correct because access to specific S3 folders for AWS Glue jobs should be restricted by scoping permissions to the correct S3 object ARNs in the IAM policy Resource element, such as bucket paths with wildcard patterns for the allowed prefixes. Time-based access is then enforced by adding a Condition that uses the global IAM condition key aws:CurrentTime with date operators such as DateGreaterThan and DateLessThan. AWS IAM documentation explicitly states that date condition operators are used with aws:CurrentTime or aws:EpochTime in the Condition element.

Option C is not the best answer because s3:prefix is generally used with bucket listing operations rather than as the main mechanism to control object-level access for Glue reads and writes. For object access, the cleanest least-privilege design is to scope the allowed object ARNs directly. Option A is incorrect because $util.time is not an IAM policy condition key. Option D uses the wrong S3 condition key for the stated requirement. The uploaded study guide also emphasizes granting AWS Glue access through an IAM service role with the right scoped permissions, which fits this pattern.

Amazon RDS controls network-level access through security groups, which act as virtual firewalls. To allow an EC2 instance to connect to an RDS database using JDBC, the RDS security group must explicitly allow inbound traffic from the EC2 instance or its associated security group on the appropriate database port.

IAM roles do not control network connectivity to RDS databases. The ec2_authorized_hosts parameter does not exist for RDS engines. The Amazon RDS Data API is not supported for standard JDBC connections and is only available for Aurora Serverless v1 and v2.

Updating the RDS security group is the correct and simplest approach. By adding an inbound rule that allows traffic from the EC2 instance’s security group, the application can securely connect to the database without additional configuration or restarts.

Therefore, Option C is the correct solution.

 A data mesh is an architectural framework that organizes data into domains and treats data as products that are owned and offered for consumption by different teams1. A data mesh requires a centralized layer for data governance and access control, as well as a distributed layer for data storage and analysis. AWS Glue can provide data catalogs and ETL operations for the data mesh, but it cannot provide data governance and access control by itself2. Therefore, the company needs to use another AWS service for this purpose. AWS Lake Formation is a service that allows you to create, secure, and manage data lakes on AWS3. It integrates with AWS Glue and other AWS services to provide centralized data governance and access control for the data mesh. Therefore, option E is correct.

For data storage and analysis, the company can choose from different AWS services depending on their needs and preferences. However, one of the benefits of a data mesh is that it enables data to be stored and processed in a decoupled and scalable way1. Therefore, using serverless or managed services that can handle large volumes and varieties of data is preferable. Amazon S3 is a highly scalable, durable, and secure object storage service that can store any type of data. Amazon Athena is a serverless interactive query service that can analyze data in Amazon S3 using standard SQL. Therefore, option B is a good choice for data storage and analysis in a data mesh. Option A, C, and D are not optimal because they either use relational databases that are not suitable for storing diverse and unstructured data, or they require more management and provisioning than serverless services. References:

1: What is a Data Mesh? - Data Mesh Architecture Explained - AWS

2: AWS Glue - Developer Guide

3: AWS Lake Formation - Features

[4]: Design a data mesh architecture using AWS Lake Formation and AWS Glue

[5]: Amazon S3 - Features

[6]: Amazon Athena - Features

Apache Parquet is a columnar storage format that is much more space-efficient than row-based formats like CSV, especially for analytics workloads. Transforming data from CSV to Parquet significantly reduces storage costs and improves query performance. According to the study guide:

“Parquet is a columnar storage file format that is optimized for use with analytics workloads, providing efficient storage and fast query performance.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

By switching to Parquet, the company can reduce both storage size and retrieval times, making it the optimal choice for cost-effective data analysis.

Option C is correct because Amazon OpenSearch Service is designed for search and analytics in near real time with low-latency queries. AWS documentation states that Amazon OpenSearch Service makes it easy to deploy, secure, operate, and scale OpenSearch to search, analyze, and visualize data in real time, and that it provides real-time analytics for use cases such as log analytics, full-text search, application monitoring, and clickstream analytics. That aligns directly with the requirement for handling many requests per second with low query latency.

This option also best satisfies the requirement that individual data teams can own and configure their own environments for cost and performance optimization. By assigning each team a separate OpenSearch cluster, each team can independently tune indexing, storage, shard layout, retention, and scaling policies. Option A is less suitable because Amazon Athena is serverless SQL over data in S3 and is not the best fit for low-latency, high-request-rate interactive search workloads. Option B focuses on streaming data processing, not primary low-latency indexed search. Option D uses a relational database, which is not the native AWS choice for large-scale search and analytics. Therefore, OpenSearch Service is the most appropriate and scalable solution.

Option C is correct because the requirement is to verify that at least 90% of values in the Email column are valid email addresses. In AWS Glue Data Quality, rules are written in DQDL, and ColumnValues is the rule component used to validate whether column values satisfy a condition or pattern. AWS Glue Data Quality documentation explains that DQDL is the language used to define rules, and it specifically notes that ColumnValues rules can evaluate column content and that NULL values do not pass during comparisons. That is important here because the problem explicitly says the column contains null entries and invalid email values. A threshold of > 0.9 means the rule passes only when more than 90% of rows satisfy the email-validation condition.

Option A is incorrect because Uniqueness checks the percentage of values that are unique, not whether values are valid email addresses. AWS documents Uniqueness as measuring how many values occur exactly once, which is unrelated to email-format validity. Option D has the same issue because UniqueValueRatio addresses uniqueness characteristics, not format validation. Option B uses the right rule family but the wrong threshold because > 0.1 would require only 10% valid values. Therefore, ColumnValues with a threshold greater than 0.9 is the correct choice.

Amazon Athena Federated Query is a feature that allows you to query data from multiple sources using standard SQL. You can use Athena Federated Query to join data from Amazon DynamoDB, Amazon RDS, Amazon Redshift, and Amazon S3, as well as other data sources such as MongoDB, Apache HBase, and Apache Kafka1. Athena Federated Query is a serverless and interactive service, meaning you do not need to provision or manage any infrastructure, and you only pay for the amount of data scanned by your queries. Athena Federated Query is the most cost-effective solution for performing a one-time analysis job on data from multiple sources, as it eliminates the need to copy or move data, and allows you to query data directly from the source.

The other options are not as cost-effective as Athena Federated Query, as they involve additional steps or costs. Option A requires you to provision and pay for an Amazon EMR cluster, which can be expensive and time-consuming for a one-time job. Option B requires you to copy or move data from DynamoDB, RDS, and Redshift to S3, which can incur additional costs for data transfer and storage, and also introduce latency and complexity. Option D requires you to have an existing Redshift cluster, which can be costly and may not be necessary for a one-time job. Option D also does not support querying data from RDS directly, so you would need to use Redshift Federated Query to access RDS data, which adds another layer of complexity2. References:

Amazon Athena Federated Query

Redshift Spectrum vs Federated Query

Option D is the best solution to meet the requirements with the least operational overhead because AWS Lake Formation is a fully managed service that simplifies the process of building, securing, and managing data lakes. AWS Lake Formation allows you to define granular data access policies at the row and column level for different users and groups. AWS Lake Formation also integrates with Amazon Athena, Amazon Redshift Spectrum, and Apache Hive on Amazon EMR, enabling these services to access the data in the data lake through AWS Lake Formation.

Option A is not a good solution because S3 access policies cannot restrict data access by rows and columns. S3 access policies are based on the identity and permissions of the requester, the bucket and object ownership, and the object prefix and tags. S3 access policies cannot enforce fine-grained data access control at the row and column level.

Option B is not a good solution because it involves using Apache Ranger and Apache Pig, which are not fully managed services and require additional configuration and maintenance. Apache Ranger is a framework that provides centralized security administration for data stored in Hadoop clusters, such as Amazon EMR. Apache Ranger can enforce row-level and column-level access policies for Apache Hive tables. However, Apache Ranger is not a native AWS service and requires manual installation and configuration on Amazon EMR clusters. Apache Pig is a platform that allows you to analyze large data sets using a high-level scripting language called Pig Latin. Apache Pig can access data stored in Amazon S3 and process it using Apache Hive. However, Apache Pig is not a native AWS service and requires manual installation and configuration on Amazon EMR clusters.

Option C is not a good solution because Amazon Redshift is not a suitable service for data lake storage. Amazon Redshift is a fully managed data warehouse service that allows you to run complex analytical queries using standard SQL. Amazon Redshift can enforce row-level and column-level access policies for different users and groups. However, Amazon Redshift is not designed to store and process large volumes of unstructured or semi-structured data, which are typical characteristics of data lakes. Amazon Redshift is also more expensive and less scalable than Amazon S3 for data lake storage.

AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide

What Is AWS Lake Formation? - AWS Lake Formation

Using AWS Lake Formation with Amazon Athena - AWS Lake Formation

Using AWS Lake Formation with Amazon Redshift Spectrum - AWS Lake Formation

Using AWS Lake Formation with Apache Hive on Amazon EMR - AWS Lake Formation

Using Bucket Policies and User Policies - Amazon Simple Storage Service

Apache Ranger

Apache Pig

What Is Amazon Redshift? - Amazon Redshift

The best practice to restrict Amazon S3 access to specific VPCs is to use a bucket policy with a StringEquals or StringLike condition on aws:SourceVpc. This ensures only requests from a specified VPC are allowed.

IAM policies (option C) control who can access the resource but are not suitable alone to restrict by VPC.

Security Groups and NACLs (options B and D) do not apply to Amazon S3 because it is a global service and not VPC-bound.

“You can restrict access to your S3 bucket so that only requests coming from a specific VPC endpoint are allowed.”

Source: AWS Documentation – Amazon S3 Bucket Policies for VPC Endpoints

To query the Sales table in Amazon Redshift for city names that start with " San " or " El, " the appropriate query uses a regular expression (regex) pattern to match city names that begin with those prefixes.

Option B: Select * from Sales where city_name ~ ' ^(San|El) ' ;In Amazon Redshift, the ~ operator is used to perform pattern matching using regular expressions. The ^(San|El) pattern matches city names that start with " San " or " El. " This is the correct SQL syntax for this use case.

Other options (A, C, D) contain incorrect syntax or incorrect use of special characters, making them invalid queries.

 QuickSight does not have access to the S3 bucket and QuickSight does not have access to decrypt S3 data are two possible factors that could cause the permissions-related errors. Amazon QuickSight is a business intelligence service that allows you to create and share interactive dashboards based on various data sources, including Amazon Athena. Amazon Athena is a serverless query service that allows you to analyze data stored in Amazon S3 using standard SQL. To use an Amazon QuickSight dashboard that is based on Amazon Athena queries on data that is stored in an Amazon S3 bucket, you need to grant QuickSight access to both Athena and S3, as well as any encryption keys that are used to encrypt the S3 data. If QuickSight does not have access to the S3 bucket or the encryption keys, it will not be able to read the data from Athena and display it on the dashboard, resulting in an error message that indicates insufficient permissions.

The other options are not factors that could cause the permissions-related errors. Option A, there is no connection between QuickSight and Athena, is not a factor, as QuickSight supports Athena as a native data source, and you can easily create a connection between them using the QuickSight console or the API. Option B, the Athena tables are not cataloged, is not a factor, as QuickSight can automatically discover the Athena tables that are cataloged in the AWS Glue Data Catalog, and you can also manually specify the Athena tables that are not cataloged. Option E, there is no IAM role assigned to QuickSight, is not a factor, as QuickSight requires an IAM role to access any AWS data sources, including Athena and S3, and you can create and assign an IAM role to QuickSight using the QuickSight console or the API. References:

Using Amazon Athena as a Data Source

Granting Amazon QuickSight Access to AWS Resources

Encrypting Data at Rest in Amazon S3

AWS Database Migration Service (AWS DMS) is a cloud service that makes it possible to migrate relational databases, data warehouses, NoSQL databases, and other types of data stores to AWS quickly, securely, and with minimal downtime and zero data loss1. AWS DMS supports migration between 20-plus database and analytics engines, such as Microsoft SQL Server to Amazon RDS for SQL Server2. AWS DMS takes over many of the difficult or tedious tasks involved in a migration project, such as capacity analysis, hardware and software procurement, installation and administration, testing and debugging, and ongoing replication and monitoring1. AWS DMS is a cost-effective solution, as you only pay for the compute resources and additional log storage used during the migration process2. AWS DMS is the best solution for the company to migrate the financial transaction data from the on-premises Microsoft SQL Server database to AWS, as it meets the requirements of minimal downtime, zero data loss, and low cost.

Option A is not the best solution, as AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers, but it does not provide any built-in features for database migration. You would have to write your own code to extract, transform, and load the data from the source to the target, which would increase the operational overhead and complexity.

Option C is not the best solution, as AWS Direct Connect is a service that establishes a dedicated network connection from your premises to AWS, but it does not provide any built-in features for database migration. You would still need to use another service or tool to perform the actual data transfer, which would increase the cost and complexity.

Option D is not the best solution, as AWS DataSync is a service that makes it easy to transfer data between on-premises storage systems and AWS storage services, such as Amazon S3, Amazon EFS, and Amazon FSx for Windows File Server, but it does not support Amazon RDS for SQL Server as a target. You would have to use another service or tool to migrate the data from Amazon S3 to Amazon RDS for SQL Server, which would increase the latency and complexity. References:

Database Migration - AWS Database Migration Service - AWS

What is AWS Database Migration Service?

AWS Database Migration Service Documentation

AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide

Option C is correct because the required access pattern is: find all completed rides for a specific driver. In DynamoDB, when you need to query data efficiently by attributes that are not part of the base table primary key, you typically create a global secondary index (GSI). AWS documentation states that a GSI can have a key schema that is different from the base table and can use top-level attributes such as DriverID and RideStatus as its partition and sort keys. That makes it possible to query directly for a given driver and then narrow the results to completed rides, without scanning the full table.

Option A is incorrect because a local secondary index (LSI) must use the same partition key as the base table. Since the table partition key is RideID, an LSI cannot be created with DriverID as the partitioning access path.

Option B uses RiderID, which does not satisfy the requirement to retrieve rides by driver. Option D is also wrong because filter expressions are applied after items are read and therefore do not avoid scanning large amounts of data. The study guide emphasizes choosing the correct data model and access pattern for the workload, which places this question in the Data Storage and Management domain.

Option D is correct because the problem is lock contention between ETL writes and analytical reads on the same Redshift tables. AWS documents that table locks can block both reads and writes, and that Redshift data sharing lets you share live data across Redshift clusters and workgroups without manually moving or copying the data. Because the shared data is live, consumer users see the most up-to-date information as soon as it is updated. Separating ETL and analytics onto different Redshift clusters while using data sharing is therefore the cleanest way to isolate workloads and still provide consistent data views.

Option A reduces contention only by changing schedules and does not fully solve concurrent peak-hour conflicts. Option B is not correct because Redshift federated queries are read-only against external sources and do not provide a Redshift “read replica” design for this case. Option C does not address the locking problem on the dimension tables that ETL is updating. The official Redshift guidance and service capabilities point to workload isolation plus data sharing as the best fix. This also matches the exam guide’s emphasis on choosing architectures that maintain performance and consistency without unnecessary copying.

Option C is correct because Amazon Redshift data sharing is built for exactly this use case: providing live, immediate, read-only access to data across Redshift clusters and AWS accounts without copying or moving the data. AWS states that data sharing lets you share live data “without having to create a copy or move it,” and that users can access the most up-to-date and consistent information as it is updated in the producer cluster. AWS also documents that data sharing provides instant, granular, and high-performance access across Amazon Redshift clusters and AWS accounts, which directly matches the requirement to preserve current Redshift performance. Since the downstream clusters use RA3 nodes, they satisfy the node-family requirement commonly associated with Redshift data sharing scenarios.

Option A would move the data to S3 and query it through Spectrum, which changes the performance profile and does not maintain native Redshift performance characteristics as well as direct Redshift data sharing. Option B introduces delay and duplicated storage through ETL copies. Option D is not an appropriate method for synchronizing Redshift datasets between clusters. Therefore, Redshift data sharing is the most direct, immediate, and performance-preserving solution.

The best solution for managing SSL/TLS certificates on EC2 instances with minimal operational overhead is to use AWS Certificate Manager (ACM). ACM simplifies certificate management by automating the provisioning, renewal, and deployment of certificates.

AWS Certificate Manager (ACM):

ACM manages SSL/TLS certificates for EC2 and other AWS resources, including automatic certificate renewal. This reduces the need for manual management and avoids operational complexity.

ACM also integrates with other AWS services to simplify secure connections between AWS infrastructure and customer-managed environments.

AWS Glue is a fully managed and serverless ETL platform that supports scripts in PySpark or Python shell jobs.

Workflows in Glue orchestrate multiple job stages without infrastructure management.

“Use AWS Glue Workflows to build and orchestrate complex multi-stage ETL pipelines using serverless AWS Glue jobs.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

This meets the “serverless only” and “runs scripts” requirements precisely.

To meet both encryption at rest and in transit, a single Amazon EMR security configuration can be created specifying the AWS KMS key for encryption at rest and the PEM file for in-transit encryption. The study guide clearly states:

“AWS Key Management Service (KMS) provides encryption for data at rest, and SSL/TLS ensures encryption for data in transit, providing end-to-end encryption within an AWS environment.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

A single security configuration is sufficient and the cleanest way to apply these security features during EMR cluster setup.

For low-latency, high-throughput workloads with API-based access and predictable reads/writes, Amazon DynamoDB is the optimal choice. It scales automatically, supports thousands of read/write operations per second, and offers fully managed API-driven access.

“Amazon DynamoDB provides consistent, single-digit millisecond latency for high-traffic applications and scales seamlessly to handle thousands of concurrent reads and writes.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

AWS Database Migration Service (AWS DMS) is purpose-built to migrate and continuously replicate data from both AWS-hosted and on-premises databases. It supports full-load (existing data) and change data capture (CDC) for ongoing changes, making it the most cost-effective and operationally simple solution in this scenario.

“DMS supports both full-load and continuous replication via CDC. This enables replicating existing and future data from various sources to a data lake in Amazon S3.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

AWS Glue is not suitable for real-time CDC replication across hybrid environments.

Amazon Redshift is a fully managed, petabyte-scale data warehouse service that enables fast and cost-effective analysis of large volumes of data. Amazon Redshift uses columnar storage, compression, and zone maps to optimize the storage and performance of data. However, over time, as data is inserted, updated, or deleted, the physical storage of data can become fragmented, resulting in wasted disk space and degraded query performance. To address this issue, Amazon Redshift provides the VACUUM command, which reclaims disk space and resorts rows in either a specified table or all tables in the current schema1.

The VACUUM command has four options: FULL, DELETE ONLY, SORT ONLY, and REINDEX. The option that best meets the requirements of the question is VACUUM REINDEX, which re-sorts the rows in a table that has an interleaved sort key and rewrites the table to a new location on disk. An interleaved sort key is a type of sort key that gives equal weight to each column in the sort key, and stores the rows in a way that optimizes the performance of queries that filter by multiple columns in the sort key. However, as data is added or changed, the interleaved sort order can become skewed, resulting in suboptimal query performance. The VACUUM REINDEX option restores the optimal interleaved sort order and reclaims disk space by removing deleted rows. This option also analyzes the sort key column and updates the table statistics, which are used by the query optimizer to generate the most efficient query execution plan23.

The other options are not optimal for the following reasons:

A. VACUUM FULL Orders. This option reclaims disk space by removing deleted rows and resorts the entire table. However, this option is not suitable for tables that have an interleaved sort key, as it does not restore the optimal interleaved sort order. Moreover, this option is the most resource-intensive and time-consuming, as it rewrites the entire table to a new location on disk.

B. VACUUM DELETE ONLY Orders. This option reclaims disk space by removing deleted rows, but does not resort the table. This option is not suitable for tables that have any sort key, as it does not improve the query performance by restoring the sort order. Moreover, this option does not analyze the sort key column and update the table statistics.

D. VACUUM SORT ONLY Orders. This option resorts the entire table, but does not reclaim disk space by removing deleted rows. This option is not suitable for tables that have an interleaved sort key, as it does not restore the optimal interleaved sort order. Moreover, this option does not analyze the sort key column and update the table statistics.

1: Amazon Redshift VACUUM

2: Amazon Redshift Interleaved Sorting

3: Amazon Redshift ANALYZE

The requirement is to manage the size of an Amazon DynamoDB table by automatically deleting data older than 1 month without disrupting ongoing read or write operations. The simplest and most maintenance-free solution is to use DynamoDB Time-to-Live (TTL).

Option A: Use the DynamoDB TTL feature to automatically expire data based on timestamps.DynamoDB TTL allows you to specify an attribute (e.g., a timestamp) that defines when items in the table should expire. After the expiration time, DynamoDB automatically deletes the items, freeing up storage space and keeping the table size under control without manual intervention or disruptions to ongoing operations.

Other options involve higher maintenance and manual scheduling or scanning operations, which increase complexity unnecessarily compared to the native TTL feature.

Option B best meets a highly resilient DR requirement with an RTO under 1 hour because a Multi-AZ deployment is designed to provide rapid recovery through redundancy across Availability Zones, rather than requiring a restore-and-rebuild process after a failure. A snapshot-based strategy (Option C) can be valuable for backup and regional recovery, but restoring a new cluster from snapshots is a heavier operation and is not the most reliable path to consistently achieving sub-hour RTO during real incidents. The study material highlights snapshot/restore as an operational mechanism for creating a new environment from a Redshift cluster’s saved state, which inherently implies a restore process and associated time to bring resources online.

Option D (cluster relocation) is primarily an operational move mechanism and is not the same as maintaining continuously resilient capacity for unplanned AZ-impacting events. Option A is not the best choice because Multi-AZ resiliency is aligned with modern Redshift architectures, and RA3 is the recommended node family for contemporary Redshift deployments where performance and managed storage characteristics are key for enterprise analytics. The study material reinforces Amazon Redshift as the platform for high-performance enterprise analytics, so a resilience-first configuration is appropriate.

AWS Step Functions allows you to orchestrate ETL jobs and workflows by integrating with AWS Glue and EventBridge. It supports both scheduled and event-driven executions with minimal operational overhead, making it ideal for hybrid triggering.

AWS Glue workflows are limited in orchestration logic.

MWAA (option B) is powerful but requires more infrastructure management.

EMR Serverless + Lambda (option D) is possible but involves more components.

“Use Step Functions to coordinate multiple services like Glue and EventBridge for event-driven or scheduled workflows.”

The company needs to use machine learning models for real-time inventory recommendations and future inventory predictions while leveraging both Amazon Redshift and Amazon SageMaker.

Option A: Use Amazon Redshift ML to generate inventory recommendations.Amazon Redshift ML allows you to build, train, and deploy machine learning models directly from Redshift using SQL statements. It integrates with SageMaker to train models and run inference. This feature is useful for generating inventory recommendations directly from the data stored in Redshift.

Option B: Use SQL to invoke a remote SageMaker endpoint for prediction.You can use SQL in Redshift to call a SageMaker endpoint for real-time inference. By invoking a SageMaker endpoint from within Redshift, the company can get real-time predictions on inventory, allowing for integration between the data warehouse and the machine learning model hosted in SageMaker.

Option C (offline model training) and Option D (creating dashboards with SageMaker Autopilot) are not relevant to the real-time prediction and recommendation requirements.

Option E (archiving inventory reports in Redshift) is not related to making predictions or recommendations.

Problem Analysis:

The company runs a daily AWS Glue ETL pipeline to clean and transform files received in an S3 bucket.

If a file is incomplete or empty, the previous day’s file should be retained.

Need a solution to validate files before overwriting the existing file.

Key Considerations:

Automate data validation with minimal human intervention.

Use built-in AWS Glue capabilities for ease of integration.

Ensure robust validation for missing or incomplete data.

Solution Analysis:

Option A: Lambda Function for Validation

Lambda can validate files, but it would require custom code.

Does not leverage AWS Glue’s built-in features, adding operational complexity.

Option B: AWS Glue Data Quality Rules

AWS Glue Data Quality allows defining Data Quality Definition Language (DQDL) rules.

Rules can validate if required fields are missing or if the file is empty.

Automatically integrates into the existing ETL pipeline.

If validation fails, retain the previous day’s file.

Option C: AWS Glue Studio with Filling Missing Values

Modifying ETL code to fill missing values with most common values risks introducing inaccuracies.

Does not handle empty files effectively.

Option D: Athena Query for Validation

Athena can drop rows with missing values, but this is a post-hoc solution.

Requires manual intervention to copy the corrected file to S3, increasing complexity.

Final Recommendation:

Use AWS Glue Data Quality to define validation rules in DQDL for identifying missing or incomplete data.

This solution integrates seamlessly with the ETL pipeline and minimizes manual effort.

Implementation Steps:

Enable AWS Glue Data Quality in the existing ETL pipeline.

Define DQDL Rules, such as:

Check if a file is empty.

Verify required fields are present and non-null.

Configure the pipeline to proceed with overwriting only if the file passes validation.

In case of failure, retain the previous day’s file.

AWS Glue Data Quality Overview

Defining DQDL Rules

AWS Glue Studio Documentation

Amazon RDS is a fully managed service that provides relational databases in the cloud. Amazon RDS for MySQL is one of the supported database engines that you can use to run your applications. Amazon RDS provides various features and tools to monitor and optimize the performance of your DB instances, such as Performance Insights, Enhanced Monitoring, CloudWatch metrics and alarms, etc.

Using the Performance Insights feature of Amazon RDS to identify queries that have high CPU utilization and optimizing the problematic queries will help reduce the CPU utilization of the DB instance. Performance Insights is a feature that allows you to analyze the load on your DB instance and determine what is causing performance issues. Performance Insights collects, analyzes, and displays database performance data using an interactive dashboard. You can use Performance Insights to identify the top SQL statements, hosts, users, or processes that are consuming the most CPU resources. You can also drill down into the details of each query and see the execution plan, wait events, locks, etc. By using Performance Insights, you can pinpoint the root cause of the high CPU utilization and optimize the queries accordingly. For example, you can rewrite the queries to make them more efficient, add or remove indexes, use prepared statements, etc.

Implementing caching to reduce the database query load will also help reduce the CPU utilization of the DB instance. Caching is a technique that allows you to store frequently accessed data in a fast and scalable storage layer, such as Amazon ElastiCache. By using caching, you can reduce the number of requests that hit your database, which in turn reduces the CPU load on your DB instance. Caching also improves the performance and availability of your application, as it reduces the latency and increases the throughput of your data access. You can use caching for various scenarios, such as storing session data, user preferences, application configuration, etc. You can also use caching for read-heavy workloads, such as displaying product details, recommendations, reviews, etc.

The other options are not as effective as using Performance Insights and caching. Modifying the database schema to include additional tables and indexes may or may not improve the CPU utilization, depending on the nature of the workload and the queries. Adding more tables and indexes may increase the complexity and overhead of the database, which may negatively affect the performance. Rebooting the RDS DB instance once each week will not reduce the CPU utilization, as it will not address the underlying cause of the high CPU load. Rebooting may also cause downtime and disruption to your application. Upgrading to a larger instance size may reduce the CPU utilization, but it will also increase the cost and complexity of your solution. Upgrading may also not be necessary if you can optimize the queries and reduce the database load by using caching. References:

Amazon RDS

Performance Insights

Amazon ElastiCache

[AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide], Chapter 3: Data Storage and Management, Section 3.1: Amazon RDS

AWS Secrets Manager is a service that allows you to securely store and manage secrets, such as database credentials, API keys, passwords, etc. You can use Secrets Manager to encrypt, rotate, and audit your secrets, as well as to control access to them using fine-grained policies. AWS Glue is a fully managed service that provides a serverless data integration platform for data preparation, data cataloging, and data loading. AWS Glue jobs allow you to transform and load data from various sources into various targets, using either a graphical interface (AWS Glue Studio) or a code-based interface (AWS Glue console or AWS Glue API).

Storing the credentials in AWS Secrets Manager and granting the AWS Glue job 1AM role access to the stored credentials will meet the requirements, as it will remediate the security vulnerability in the AWS Glue job and securely store the credentials. By using AWS Secrets Manager, you can avoid hard coding the credentials in the job script, which is a bad practice that exposes the credentials to unauthorized access or leakage. Instead, you can store the credentials as a secret in Secrets Manager and reference the secret name or ARN in the job script. You can also use Secrets Manager to encrypt the credentials using AWS Key Management Service (AWS KMS), rotate the credentials automatically or on demand, and monitor the access to the credentials using AWS CloudTrail. By granting the AWS Glue job 1AM role access to the stored credentials, you can use the principle of least privilege to ensure that only the AWS Glue job can retrieve the credentials from Secrets Manager. You can also use resource-based or tag-based policies to further restrict the access to the credentials.

The other options are not as secure as storing the credentials in AWS Secrets Manager and granting the AWS Glue job 1AM role access to the stored credentials. Storing the credentials in the AWS Glue job parameters will not remediate the security vulnerability, as the job parameters are still visible in the AWS Glue console and API. Storing the credentials in a configuration file that is in an Amazon S3 bucket and accessing the credentials from the configuration file by using the AWS Glue job will not be as secure as using Secrets Manager, as the configuration file may not be encrypted or rotated, and the access to the file may not be audited or controlled. References:

AWS Secrets Manager

AWS Glue

AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide, Chapter 6: Data Integration and Transformation, Section 6.1: AWS Glue

Option A is the most cost-effective because it enables seamless integration by allowing analysts to access and join external relational data directly from within Amazon Redshift for reporting, without building and operating a separate ingestion pipeline for a one-time (annual) workload. The study material frames Amazon Redshift as the centralized analytics store for complex queries and reporting, making it the natural place to perform the final transformations and analysis.

The alternatives introduce extra operational steps and recurring costs. Option B requires exporting large historical datasets to S3 and then loading them into Redshift, which increases data movement and operational complexity. Option C adds ETL job development, scheduling, retries, and monitoring overhead that is unnecessary if the primary goal is integrated querying for an annual report. Option D (DMS) is best when you need ongoing migration or continuous replication; it is typically more operationally heavy than needed for “query across sources” use cases and still requires additional setup and maintenance.

Because all databases are already in the same VPC and the analytics platform is Redshift, federated querying provides the most direct, lowest-operations path to integrate and analyze the data where it already needs to be consumed.

AWS security best practices require avoiding long-term credentials and enforcing the principle of least privilege. Using IAM roles with role assumption is the most secure approach for granting temporary, scoped access to AWS resources.

In this solution, an IAM role is created with permissions to access the Amazon RDS databases. A second IAM role is assigned to the AWS Lambda functions and AWS Glue jobs with permission to assume the RDS-access role. This approach eliminates hard-coded credentials, enables automatic credential rotation through AWS Security Token Service (AWS STS), and provides clear separation of duties.

Using the root user or long-term access keys is explicitly discouraged by AWS. Creating IAM users for services violates best practices, as AWS services should use IAM roles, not users. Embedding credentials in JDBC connection strings exposes sensitive information and increases the risk of credential leakage.

Therefore, using IAM role assumption provides the strongest security posture, auditability, and compliance with AWS Well-Architected Framework guidance.

 This solution meets the requirement of gaining access to SageMaker Studio to use AWS Glue interactive sessions. AWS Glue interactive sessions are a way to use AWS Glue DataBrew and AWS Glue Data Catalog from within SageMaker Studio. To use AWS Glue interactive sessions, the data engineer’s IAM user needs to have permissions to assume the AWS Glue service role and the SageMaker execution role. By adding a policy to the data engineer’s IAM user that includes the sts:AssumeRole action for the AWS Glue and SageMaker service principals in the trust policy, the data engineer can grant these permissions and avoid the access denied error. The other options are not sufficient or necessary to resolve the error. References:

Get started with data integration from Amazon S3 to Amazon Redshift using AWS Glue interactive sessions

Troubleshoot Errors - Amazon SageMaker

AccessDeniedException on sagemaker:CreateDomain in AWS SageMaker Studio, despite having SageMakerFullAccess

Option B is the only solution that supports near real-time updates from a continuously changing source to Amazon Redshift. The requirement says the on-premises PostgreSQL database is “continuously updated” and the target must be updated “as quickly as possible.” Nightly full backups or nightly full loads (Options A and D) inherently introduce at least a daily lag, which violates the freshness requirement. Similarly, exporting with pg_dump and reloading with COPY (Option C) is a batch approach and does not provide continuous change propagation.

The study material explicitly positions AWS Database Migration Service (DMS) for database migrations and highlights that it supports both full-load and change data capture (CDC), and that CDC enables continuous replication so ongoing changes can be applied after the initial load.

Therefore, a DMS task configured for full load + CDC provides the fastest ongoing synchronization pattern: it performs the initial migration and then continuously captures and applies changes so Redshift stays current with minimal delay compared to periodic batch reloads.

Option B is the most operationally efficient because it checks the business requirement directly: whether the target table contains the most recent data, not merely whether a job failed. Monitoring only failures (Options C and D) can produce false positives (a job failure might not impact freshness) and false negatives (a job can succeed but still load stale or incomplete data). The study material emphasizes implementing data quality validation as part of the ETL process so data can be verified before or as it is stored, rather than relying only on pipeline execution status.

Using a data quality rule focused on freshness (for example, validating that a “max event timestamp” or “latest partition date” meets today’s expected value) lets the pipeline detect stale loads even when the workflow runs. Then, an EventBridge rule can route failures of that data quality check to SNS for immediate notification, keeping operations serverless and centralized. Macie (Option A) is designed for sensitive-data discovery/classification, not operational “freshness” checks on Redshift tables, so it adds unnecessary services and effort compared to a Glue-native data quality validation approach.

Lake Formation requires that S3 locations be registered before they can be accessed by AWS Glue crawlers, jobs, or queries. Registering the location associates it with the Lake Formation permissions framework.

“Before AWS Glue crawlers can access an S3 path managed by Lake Formation, the S3 location must be registered in Lake Formation.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

AWS Glue ETL is designed for scalable and serverless data processing, and it supports integrated quality enforcement using AWS Glue Data Quality, which makes it the most cost-effective and integrated option when combined with Amazon RDS for MySQL as the relational database.

“AWS Glue can perform data validation as part of the ETL process, ensuring data quality before storing the data in the target data store.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

Using AWS Glue Data Quality directly in the ETL workflow is simpler and more cost-effective than separating transformation (Glue) and validation (DataBrew) into different services.

Option A is correct because this is a sequential, dependent workflow where each validation stage depends on the previous one, and the overall execution can last up to 24 hours. AWS states that Standard Workflows in Step Functions are ideal for long-running, durable, auditable workflows and can run for up to one year. AWS also documents the Wait for a Callback with Task Token pattern, which is specifically designed for workflows that must pause until an external process reports completion. That maps directly to multi-level validations that may complete asynchronously but still need orchestration within a bounded window.

Option B is incorrect because Express Workflows can run for only up to five minutes, so they cannot support a transaction workflow that may take many hours. Option C adds more custom code and operational management than Step Functions and does not naturally provide the same stateful orchestration, dependency tracking, and callback waiting pattern. Option D is also more operationally heavy and is better suited to data pipeline orchestration than per-transaction validation workflows. Therefore, Standard Step Functions provides the lowest operational cost while fully meeting the timing and dependency requirements.

The correct answer is B because AWS Glue streaming ETL jobs are built to continuously consume data from streaming sources such as Amazon Kinesis Data Streams, transform the data, and load the results into Amazon S3 data lakes. The requirement is not just delivery to S3; the telemetry JSON must be flattened before storage, and this needs to happen with low latency. Amazon Data Firehose can deliver streaming data, but the option incorrectly assumes automatic flattening without a transformation layer. SQS plus scheduled batch Glue introduces batch delay and does not meet the lowest-latency requirement. Athena is a query engine, not a real-time transformation and ingestion service for writing flattened streaming output to S3. AWS Glue streaming ETL is the correct serverless streaming transformation layer.

Problem Analysis:

The company requires a centralized data warehouse for consolidating data from various sources.

They use Amazon QuickSight in direct query mode, necessitating fast response times for analytical queries.

Users query the data intermittently, with unpredictable spikes during the day.

Operational overhead should be minimal.

Key Considerations:

The solution must support fast, SQL-based analytics.

It must handle unpredictable spikes efficiently.

Must integrate seamlessly with QuickSight for direct querying.

Minimize operational complexity and scaling concerns.

Solution Analysis:

Option A: Amazon Redshift Serverless

Redshift Serverless eliminates the need for provisioning and managing clusters.

Automatically scales compute capacity up or down based on query demand.

Reduces operational overhead by handling performance optimization.

Fully integrates with Amazon QuickSight, ensuring low-latency analytics.

Reduces costs as it charges only for usage, making it ideal for workloads with intermittent spikes.

Option B: Amazon Athena with S3 (Apache Parquet)

Athena supports querying data directly from S3 in Parquet format.

While it’s cost-effective, performance depends on the size and complexity of the data.

It is not optimized for high-speed analytics needed by QuickSight in direct query mode.

Option C: Amazon Redshift Provisioned Clusters

Requires manual cluster provisioning, scaling, and maintenance.

Higher operational overhead compared to Redshift Serverless.

Option D: Amazon Aurora PostgreSQL

Aurora is optimized for transactional databases, not data warehousing or analytics.

Does not meet the requirement for fast analytics queries.

Final Recommendation:

Amazon Redshift Serverless is the best choice for this use case because it provides fast analytics, integrates natively with QuickSight, and minimizes operational complexity while efficiently handling unpredictable spikes.

Amazon Redshift Serverless Overview

Amazon QuickSight and Redshift Integration

Athena vs. Redshift

 The best combination of resources to meet the requirements of high reliability, cost-optimization, and performance for running Apache Spark jobs on Amazon EMR is to use Amazon S3 as a persistent data store and Graviton instances for core nodes and task nodes.

Amazon S3 is a highly durable, scalable, and secure object storage service that can store any amount of data for a variety of use cases, including big data analytics1. Amazon S3 is a better choice than HDFS as a persistent data store for Amazon EMR, as it decouples the storage from the compute layer, allowing for more flexibility and cost-efficiency. Amazon S3 also supports data encryption, versioning, lifecycle management, and cross-region replication1. Amazon EMR integrates seamlessly with Amazon S3, using EMR File System (EMRFS) to access data stored in Amazon S3 buckets2. EMRFS also supports consistent view, which enables Amazon EMR to provide read-after-write consistency for Amazon S3 objects that are accessed through EMRFS2.

Graviton instances are powered by Arm-based AWS Graviton2 processors that deliver up to 40% better price performance over comparable current generation x86-based instances3. Graviton instances are ideal for running workloads that are CPU-bound, memory-bound, or network-bound, such as big data analytics, web servers, and open-source databases3. Graviton instances are compatible with Amazon EMR, and can be used for both core nodes and task nodes. Core nodes are responsible for running the data processing frameworks, such as Apache Spark, and storing data in HDFS or the local file system. Task nodes are optional nodes that can be added to a cluster to increase the processing power and throughput. By using Graviton instances for both core nodes and task nodes, you can achieve higher performance and lower cost than using x86-based instances.

Using Spot Instances for all primary nodes is not a good option, as it can compromise the reliability and availability of the cluster. Spot Instances are spare EC2 instances that are available at up to 90% discount compared to On-Demand prices, but they can be interrupted by EC2 with a two-minute notice when EC2 needs the capacity back. Primary nodes are the nodes that run the cluster software, such as Hadoop, Spark, Hive, and Hue, and are essential for the cluster operation. If a primary node is interrupted by EC2, the cluster will fail or become unstable. Therefore, it is recommended to use On-Demand Instances or Reserved Instances for primary nodes, and use Spot Instances only for task nodes that can tolerate interruptions. References:

Amazon S3 - Cloud Object Storage

EMR File System (EMRFS)

AWS Graviton2 Processor-Powered Amazon EC2 Instances

[Plan and Configure EC2 Instances]

[Amazon EC2 Spot Instances]

[Best Practices for Amazon EMR]

 The Amazon Redshift Data API is a built-in feature that allows you to run SQL queries on Amazon Redshift data with web services-based applications, such as AWS Lambda, Amazon SageMaker notebooks, and AWS Cloud9. The Data API does not require a persistent connection to your database, and it provides a secure HTTP endpoint and integration with AWS SDKs. You can use the endpoint to run SQL statements without managing connections. The Data API also supports both Amazon Redshift provisioned clusters and Redshift Serverless workgroups. The Data API is the best solution for running real-time queries on the financial data from within the trading application, as it has the least operational overhead compared to the other options.

Option A is not the best solution, as establishing WebSocket connections to Amazon Redshift would require more configuration and maintenance than using the Data API. WebSocket connections are also not supported by Amazon Redshift clusters or serverless workgroups.

Option C is not the best solution, as setting up JDBC connections to Amazon Redshift would also require more configuration and maintenance than using the Data API. JDBC connections are also not supported by Redshift Serverless workgroups.

Option D is not the best solution, as storing frequently accessed data in Amazon S3 and using Amazon S3 Select to run the queries would introduce additional latency and complexity than using the Data API. Amazon S3 Select is also not optimized for real-time queries, as it scans the entire object before returning the results. References:

Using the Amazon Redshift Data API

Calling the Data API

Amazon Redshift Data API Reference

AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide

The correct answer is A because Athena can query new partitions only after the partitions exist in the AWS Glue Data Catalog. Waiting for a nightly crawler creates unnecessary delay. The direct fix is to update the Data Catalog as part of the ETL process by calling the AWS Glue CreatePartition API immediately after each new S3 partition is written. AWS Glue documentation states that CreatePartition creates a new partition and requires the database name, table name, and partition input structure. Copying data to another bucket does not register partitions. DynamoDB Streams is unrelated unless the partition lifecycle is already stored in DynamoDB. StartImportLabelsTaskRun is not the API for partition registration. The correct operational pattern is event/write partition, then immediately register metadata.

This solution will meet the requirements with the least operational overhead because it uses the AWS Glue Data Catalog as the central metadata repository for data sources that run in the AWS Cloud. The AWS Glue Data Catalog is a fully managed service that provides a unified view of your data assets across AWS and on-premises data sources. It stores the metadata of your data in tables, partitions, and columns, and enables you to access and query your data using various AWS services, such as Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum. You can use AWS Glue crawlers to connect to multiple data stores, such as Amazon RDS, Amazon Redshift, and Amazon S3, and to update the Data Catalog with metadata changes. AWS Glue crawlers can automatically discover the schema and partition structure of your data, and create or update the corresponding tables in the Data Catalog. You can schedule the crawlers to run periodically to update the metadata catalog, and configure them to detect changes to the source metadata, such as new columns, tables, or partitions12.

The other options are not optimal for the following reasons:

A. Use Amazon Aurora as the data catalog. Create AWS Lambda functions that will connect to the data catalog. Configure the Lambda functions to gather the metadata information from multiple sources and to update the Aurora data catalog. Schedule the Lambda functions to run periodically. This option is not recommended, as it would require more operational overhead to create and manage an Amazon Aurora database as the data catalog, and to write and maintain AWS Lambda functions to gather and update the metadata information from multiple sources. Moreover, this option would not leverage the benefits of the AWS Glue Data Catalog, such as data cataloging, data transformation, and data governance.

C. Use Amazon DynamoDB as the data catalog. Create AWS Lambda functions that will connect to the data catalog. Configure the Lambda functions to gather the metadata information from multiple sources and to update the DynamoDB data catalog. Schedule the Lambda functions to run periodically. This option is also not recommended, as it would require more operational overhead to create and manage an Amazon DynamoDB table as the data catalog, and to write and maintain AWS Lambda functions to gather and update the metadata information from multiple sources. Moreover, this option would not leverage the benefits of the AWS Glue Data Catalog, such as data cataloging, data transformation, and data governance.

D. Use the AWS Glue Data Catalog as the central metadata repository. Extract the schema for Amazon RDS and Amazon Redshift sources, and build the Data Catalog. Use AWS Glue crawlers for data that is in Amazon S3 to infer the schema and to automatically update the Data Catalog. This option is not optimal, as it would require more manual effort to extract the schema for Amazon RDS and Amazon Redshift sources, and to build the Data Catalog. This option would not take advantage of the AWS Glue crawlers’ ability to automatically discover the schema and partition structure of your data from various data sources, and to create or update the corresponding tables in the Data Catalog.

1: AWS Glue Data Catalog

2: AWS Glue Crawlers

Amazon Aurora

AWS Lambda

Amazon DynamoDB

The correct answer is C because SNS Publish and PublishBatch operations are data-plane operations, not ordinary management events. AWS CloudTrail documentation specifically lists Amazon SNS Publish and PublishBatch API operations on topics as supported data events. Because the company already has a CloudTrail Lake event data store, the lowest-operational-overhead approach is to enable the relevant SNS data events and query them directly in CloudTrail Lake. Creating Glue Data Catalog tables and querying with Athena adds unnecessary cataloging and query infrastructure. Management events would not satisfy the requirement to audit the requested SNS publishing actions. Therefore, the precise configuration is SNS data events plus CloudTrail Lake querying.

Option B is correct because Amazon MSK Replicator is the AWS-managed feature specifically designed to reliably replicate data across Amazon MSK clusters in different AWS Regions. AWS states that MSK Replicator can be used to build regionally resilient streaming applications for availability and business continuity, which directly supports the disaster recovery requirement. It is also the choice with the least operational effort because it avoids deploying and managing self-hosted replication tools such as Kafka Connect or MirrorMaker 2. AWS documentation for cross-Region setup also shows support for secure connectivity and notes that MSK environments support IAM, TLS, and SASL/SCRAM authentication schemes for private connectivity scenarios.

Option A and C would require the company to deploy, scale, patch, and monitor custom replication infrastructure in every Region, which increases operational overhead. Option D is not appropriate because Kinesis Data Firehose is not the native service for replicating data between MSK clusters. For ordering and consistency, Kafka-based replication preserves partitioned stream semantics better than mixing in a different streaming service. Therefore, a mesh of MSK Replicator relationships between regional MSK clusters is the best AWS-native answer for near real-time, encrypted, cross-Region replication with minimal administration.

Creating an EventBridge rule that triggers a Lambda function on AWS Glue job failure events and then sends notifications via Amazon SNS is the most direct and operationally efficient method:

“Practice Quiz 10: A data engineer must monitor the data pipeline... Which solution will meet these requirements?

A. Inspect the job run monitoring section of the AWS Glue console.

Correct Answer: A.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

Although this reference directly supports using AWS Glue’s monitoring features via EventBridge, it implies that solutions like A—which directly use EventBridge failure events for automation—are more optimal and less complex than constructing custom logs and metrics.

 This solution meets the requirements of optimizing the performance of table SQL queries without increasing the size of the cluster. By using the ALL distribution style for rarely updated small tables, you can ensure that the entire table is copied to every node in the cluster, which eliminates the need for data redistribution during joins. This can improve query performance significantly, especially for frequently joined dimension tables. However, using the ALL distribution style also increases the storage space and the load time, so it is only suitable for small tables that are not updated frequently or extensively. By specifying primary and foreign keys for all tables, you can help the query optimizer to generate better query plans and avoid unnecessary scans or joins. You can also use the AUTO distribution style to let Amazon Redshift choose the optimal distribution style based on the table size and the query patterns. References:

Choose the best distribution style

Distribution styles

Working with data distribution styles

This solution will meet the requirements with the lowest latency because it uses Amazon Managed Service for Apache Flink to process the sensor data in real time and write it to Amazon Timestream, a fast, scalable, and serverless time series database. Amazon Timestream is optimized for storing and analyzing time series data, such as sensor data, and can handle trillions of events per day with millisecond latency. By using Amazon Timestream as a source, you can create an Amazon QuickSight dashboard that displays a real-time view of operational efficiency on a large screen in the manufacturing facility. Amazon QuickSight is a fully managed business intelligence service that can connect to various data sources, including Amazon Timestream, and provide interactive visualizations and insights123.

The other options are not optimal for the following reasons:

A. Use Amazon Managed Service for Apache Flink (previously known as Amazon Kinesis Data Analytics) to process the sensor data. Use a connector for Apache Flink to write data to an Amazon Timestream database. Use the Timestream database as a source to create a Grafana dashboard. This option is similar to option C, but it uses Grafana instead of Amazon QuickSight to create the dashboard. Grafana is an open source visualization tool that can also connect to Amazon Timestream, but it requires additional steps to set up and configure, such as deploying a Grafana server on Amazon EC2, installing the Amazon Timestream plugin, and creating an IAM role for Grafana to access Timestream. These steps can increase the latency and complexity of the solution.

B. Configure the S3 bucket to send a notification to an AWS Lambda function when any new object is created. Use the Lambda function to publish the data to Amazon Aurora. Use Aurora as a source to create an Amazon QuickSight dashboard. This option is not suitable for displaying a real-time view of operational efficiency, as it introduces unnecessary delays and costs in the data pipeline. First, the sensor data is written to an S3 bucket by Amazon Kinesis Data Firehose, which can have a buffering interval of up to 900 seconds. Then, the S3 bucket sends a notification to a Lambda function, which can incur additional invocation and execution time. Finally, the Lambda function publishes the data to Amazon Aurora, a relational database that is not optimized for time series data and can have higher storage and performance costs than Amazon Timestream .

D. Use AWS Glue bookmarks to read sensor data from the S3 bucket in real time. Publish the data to an Amazon Timestream database. Use the Timestream database as a source to create a Grafana dashboard. This option is also not suitable for displaying a real-time view of operational efficiency, as it uses AWS Glue bookmarks to read sensor data from the S3 bucket. AWS Glue bookmarks are a feature that helps AWS Glue jobs and crawlers keep track of the data that has already been processed, so that they can resume from where they left off. However, AWS Glue jobs and crawlers are not designed for real-time data processing, as they can have a minimum frequency of 5 minutes and a variable start-up time. Moreover, this option also uses Grafana instead of Amazon QuickSight to create the dashboard, which can increase the latency and complexity of the solution .

1: Amazon Managed Streaming for Apache Flink

2: Amazon Timestream

3: Amazon QuickSight

Analyze data in Amazon Timestream using Grafana

Amazon Kinesis Data Firehose

Amazon Aurora

AWS Glue Bookmarks

AWS Glue Job and Crawler Scheduling

Option B is correct because IAM policies are the standard AWS mechanism to control access to Amazon S3, and SSE-KMS encrypts S3 data with AWS KMS keys. AWS S3 documentation for SSE-KMS explains that it uses AWS Key Management Service keys for server-side encryption, and Secrets Manager documentation states that you can store secrets securely and configure automatic rotation. AWS further explains that rotation updates the credentials in both the secret and the target database or service, and that Secrets Manager uses a Lambda rotation function for supported rotation patterns.

Option A is weaker because Lambda environment variables are not the right service for managed secret storage and automatic rotation. Option C is not the best answer because S3 ACLs are not the preferred modern access-control model compared with IAM and bucket policies, and Parameter Store is not the main AWS service for built-in managed database credential rotation. Option D uses SSE-S3, not KMS-based encryption, and relies on a custom rotation approach instead of the native managed secret-rotation service. The study guide also highlights AWS KMS for encryption-key management and Secrets Manager for rotating credentials.

Scheduling an AWS Glue crawler to periodically update the Data Catalog automates the process of detecting new partitions and updating the catalog, which minimizes manual maintenance and operational overhead.

For exactly-once delivery and processing in Amazon Kinesis Data Streams, the best approach is to design the application so that it handles idempotency. By embedding a unique ID in each record, the application can identify and remove duplicate records during processing.

Exactly-Once Processing:

Kinesis Data Streams does not natively support exactly-once processing. Therefore, idempotency should be designed into the application, ensuring that each record has a unique identifier so that the same event is processed only once, even if it is ingested multiple times.

This pattern is widely used for achieving exactly-once semantics in distributed systems.

AWS Glue streaming jobs are purpose-built for low-latency, near real-time ETL pipelines.

Glue Studio allows visual job authoring and supports complex transformations, including joins with Amazon RDS databases, through JDBC connectors.

Data can be written directly to S3 in Apache Parquet format using built-in AWS Glue support.

Options B and C are more suitable for batch or development/testing. Option D is for custom batch jobs, not real-time processing.

“You can use AWS Glue streaming jobs with Glue Studio to process streaming data, perform transformations, join with RDS data sources, and write results in formats like Parquet to S3.”

Option B is the best answer because Amazon EventBridge is designed to route events based on matching rules, and each rule can use a specific event pattern to match only the relevant error type and then send that event to the correct target. AWS states that EventBridge event buses receive events and deliver them to targets, and that rules can match on event details and route matching events accordingly. This makes it ideal for real-time error classification with very little custom code. An SNS topic per team then provides immediate push-based notification delivery.

Option A adds unnecessary operational overhead because Lambda would be doing routing logic that EventBridge rules already handle natively. Option C is weaker because SQS is pull-based; teams must poll queues, which is not the best fit for “without any delay” notifications. AWS documentation explicitly describes SQS consumption through polling. Option D is also not the best fit because CloudWatch alarms are threshold-based metric monitors, not event routers for per-message error classification.

This aligns with the study guide’s emphasis on using managed AWS services to automate and orchestrate pipeline actions with the least overhead.

For processing the incoming car listings in a cost-effective, scalable, and automated way, the ideal approach involves using AWS Glue for data processing, AWS Lambda with S3 Event Notifications for orchestration, Amazon Athena for one-time queries and analytical reporting, and Amazon QuickSight for visualization on the dashboard. Let’s break this down:

AWS Glue: This is a fully managed ETL (Extract, Transform, Load) service that automatically processes the incoming data files. Glue is serverless and supports diverse data sources, including Amazon S3 and Redshift.

AWS Lambda and S3 Event Notifications: Using Lambda and S3 Event Notifications allows near real-time triggering of processing workflows as soon as new data is uploaded into S3. This approach is event-driven, ensuring that the listings are processed as soon as they are uploaded, reducing the latency for data processing.

Amazon Athena: A serverless, pay-per-query service that allows interactive queries directly against data in S3 using standard SQL. It is ideal for the requirement of one-time queries and analytical reporting without the need for provisioning or managing servers.

Amazon QuickSight: A business intelligence tool that integrates with a wide range of AWS data sources, including Athena, and is used for creating interactive dashboards. It scales well and provides real-time insights for the car listings.

This solution (Option D) is the most cost-effective, because both Glue and Athena are serverless and priced based on usage, reducing costs when compared to provisioning EMR clusters in the other options. Moreover, using Lambda for orchestration is more cost-effective than AWS Step Functions due to its lightweight nature.

AWS Lake Formation is specifically designed to simplify fine-grained access control for data lakes stored in Amazon S3. By using Lake Formation tag-based access control (LF-TBAC), administrators can define access policies once and apply them dynamically based on tags rather than managing individual IAM policies.

LF-Tags can be assigned to databases, tables, and columns in the AWS Glue Data Catalog. Departments are granted permissions based on tags, ensuring that each department can access only the data it is authorized to view. This approach scales efficiently as datasets and departments grow, which is critical for data lakes ingesting millions of records daily.

Managing IAM policies per department is operationally complex and error-prone. Redshift-based access centralization limits flexibility and introduces unnecessary infrastructure. Syncing data into an RDS database adds cost, latency, and maintenance overhead.

Lake Formation provides centralized governance, auditing, and least-privilege enforcement with minimal administrative effort, making it the optimal solution.

Option B is correct because Amazon Managed Service for Apache Flink is the AWS managed service built for real-time stream processing, including stateful computations over streaming data. AWS documentation describes Apache Flink as a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. AWS also states that Managed Service for Apache Flink can read from a Kinesis data stream, apply transformations such as filtering, aggregation, or enrichment, and write the transformed output to a sink. This maps exactly to the requirement for complex, stateful, near-real-time transformations on data coming from Amazon Kinesis Data Streams before storing it in S3.

Option A and D are less suitable because scheduled Glue or EMR Spark jobs are not the least-operational or best-fit solution for continuous near-real-time stateful streaming transformations. Option C can work for lightweight event processing, but Lambda is not the best fit for more complex and stateful streaming logic compared with Flink’s native state management and streaming semantics. Therefore, Managed Service for Apache Flink is the correct managed AWS answer.

To migrate data from an Amazon RDS for PostgreSQL DB instance in the eu-east-1 Region (Account_A) to an Amazon Redshift cluster in the eu-west-1 Region (Account_B), AWS DMS needs a replication instance located in the target region (in this case, eu-west-1) to facilitate the data transfer between regions.

Option A: Set up an AWS DMS replication instance in Account_B in eu-west-1.Placing the DMS replication instance in the target account and region (Account_B in eu-west-1) is the most efficient solution. The replication instance can connect to the source RDS PostgreSQL in eu-east-1 and migrate the data to the Redshift cluster in eu-west-1. This setup ensures data is replicated across AWS accounts and regions.

Options B, C, and D place the replication instance in either the wrong account or region, which increases complexity without adding any benefit.

 AWS Glue workflows are a cost-effective way to orchestrate complex ETL jobs that involve multiple crawlers, jobs, and triggers. AWS Glue workflows allow you to visually monitor the progress and dependencies of your ETL tasks, and automatically handle errors and retries. AWS Glue workflows also integrate with other AWS services, such as Amazon S3, Amazon Redshift, and AWS Lambda, among others, enabling you to leverage these services for your data processing workflows. AWS Glue workflows are serverless, meaning you only pay for the resources you use, and you don’t have to manage any infrastructure.

AWS Step Functions, AWS Glue Studio, and Amazon MWAA are also possible options for orchestrating ETL pipelines, but they have some drawbacks compared to AWS Glue workflows. AWS Step Functions is a serverless function orchestrator that can handle different types of data processing, such as real-time, batch, and stream processing. However, AWS Step Functions requires you to write code to define your state machines, which can be complex and error-prone. AWS Step Functions also charges you for every state transition, which can add up quickly for large-scale ETL pipelines.

AWS Glue Studio is a graphical interface that allows you to create and run AWS Glue ETL jobs without writing code. AWS Glue Studio simplifies the process of building, debugging, and monitoring your ETL jobs, and provides a range of pre-built transformations and connectors. However, AWS Glue Studio does not support workflows, meaning you cannot orchestrate multiple ETL jobs or crawlers with dependencies and triggers. AWS Glue Studio also does not support streaming data sources or targets, which limits its use cases for real-time data processing.

Amazon MWAA is a fully managed service that makes it easy to run open-source versions of Apache Airflow on AWS and build workflows to run your ETL jobs and data pipelines. Amazon MWAA provides a familiar and flexible environment for data engineers who are familiar with Apache Airflow, and integrates with a range of AWS services such as Amazon EMR, AWS Glue, and AWS Step Functions. However, Amazon MWAA is not serverless, meaning you have to provision and pay for the resources you need, regardless of your usage. Amazon MWAA also requires you to write code to define your DAGs, which can be challenging and time-consuming for complex ETL pipelines. References:

AWS Glue Workflows

AWS Step Functions

AWS Glue Studio

Amazon MWAA

AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide

The Amazon Redshift Data API enables you to interact with your Amazon Redshift data warehouse in an easy and secure way. You can use the Data API to run SQL commands, such as loading data into tables, without requiring a persistent connection to the cluster. The Data API also integrates with Amazon EventBridge, which allows you to monitor the execution status of your SQL commands and trigger actions based on events. By using the Data API to publish an event to EventBridge, the data engineer can invoke the Lambda function that writes the load statuses to the DynamoDB table. This solution is scalable, reliable, and cost-effective. The other options are either not possible or not optimal. You cannot use a second Lambda function to invoke the first Lambda function based on CloudWatch or CloudTrail events, as these services do not capture the load status of Redshift tables. You can use the Data API to publish a message to an SQS queue, but this would require additional configuration and polling logic to invoke the Lambda function from the queue. This would also introduce additional latency and cost. References:

Using the Amazon Redshift Data API

Using Amazon EventBridge with Amazon Redshift

AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide, Chapter 2: Data Store Management, Section 2.2: Amazon Redshift

 The best solutions to resolve the performance bottleneck and reduce Athena query planning time are to create an AWS Glue partition index and enable partition filtering, and to use Athena partition projection based on the S3 bucket prefix.

AWS Glue partition indexes are a feature that allows you to speed up query processing of highly partitioned tables cataloged in AWS Glue Data Catalog. Partition indexes are available for queries in Amazon EMR, Amazon Redshift Spectrum, and AWS Glue ETL jobs. Partition indexes are sublists of partition keys defined in the table. When you create a partition index, you specify a list of partition keys that already exist on a given table. AWS Glue then creates an index for the specified keys and stores it in the Data Catalog. When you run a query that filters on the partition keys, AWS Glue uses the partition index to quickly identify the relevant partitions without scanning the entire table metadata. This reduces the query planning time and improves the query performance1.

Athena partition projection is a feature that allows you to speed up query processing of highly partitioned tables and automate partition management. In partition projection, Athena calculates partition values and locations using the table properties that you configure directly on your table in AWS Glue. The table properties allow Athena to ‘project’, or determine, the necessary partition information instead of having to do a more time-consuming metadata lookup in the AWS Glue Data Catalog. Because in-memory operations are often faster than remote operations, partition projection can reduce the runtime of queries against highly partitioned tables. Partition projection also automates partition management because it removes the need to manually create partitions in Athena, AWS Glue, or your external Hive metastore2.

Option B is not the best solution, as bucketing the data based on a column that the data have in common in a WHERE clause of the user query would not reduce the query planning time. Bucketing is a technique that divides data into buckets based on a hash function applied to a column. Bucketing can improve the performance of join queries by reducing the amount of data that needs to be shuffled between nodes. However, bucketing does not affect the partition metadata retrieval, which is the main cause of the performance bottleneck in this scenario3.

Option D is not the best solution, as transforming the data that is in the S3 bucket to Apache Parquet format would not reduce the query planning time. Apache Parquet is a columnar storage format that can improve the performance of analytical queries by reducing the amount of data that needs to be scanned and providing efficient compression and encoding schemes. However, Parquet does not affect the partition metadata retrieval, which is the main cause of the performance bottleneck in this scenario4.

Option E is not the best solution, as using the Amazon EMR S3DistCP utility to combine smaller objects in the S3 bucket into larger objects would not reduce the query planning time. S3DistCP is a tool that can copy large amounts of data between Amazon S3 buckets or from HDFS to Amazon S3. S3DistCP can also aggregate smaller files into larger files to improve the performance of sequential access. However, S3DistCP does not affect the partition metadata retrieval, which is the main cause of the performance bottleneck in this scenario5. References:

Improve query performance using AWS Glue partition indexes

Partition projection with Amazon Athena

Bucketing vs Partitioning

Columnar Storage Formats

S3DistCp

AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide

Amazon Kinesis Data Firehose is a fully managed data ingestion service that enables reliable and scalable delivery of streaming and batch data into Amazon S3 with minimal operational overhead. This directly satisfies the requirement for a fully managed AWS ingestion service while avoiding the need to provision, scale, or manage infrastructure.

By using the Amazon Kinesis Agent on the on-premises servers, the company can automatically forward daily data files to Kinesis Data Firehose. Firehose handles buffering, retry logic, scaling, and delivery without requiring administrative effort. Delivering the data to Amazon S3 allows seamless integration with Amazon Athena, which natively queries data stored in S3 without requiring data movement or transformation.

Enabling Amazon S3 versioning protects files from accidental deletion by preserving previous versions of objects. This aligns with AWS best practices for data durability and governance, especially for analytics workloads and compliance requirements.

Other options introduce unnecessary operational complexity. AWS DataSync with Amazon EFS is not optimized for Athena-based analytics. AWS Glue jobs and Amazon RDS are unsuitable for file-based analytical access. A self-managed Apache Kafka solution with Amazon MSK significantly increases operational overhead.

Therefore, option B is the most efficient, scalable, and operationally optimal solution according to AWS Certified Data Engineer – Associate best practices.

For query workloads on S3 data that depend on date-based filters, partitioning by order date optimizes performance and cost because Athena reads only the relevant partitions.

Athena scales automatically and doesn’t degrade with increasing data size when partitions are managed efficiently.

“Partitioning data in Amazon S3 based on query predicates such as order date improves Athena query performance and reduces scanned data volume.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

This is the most cost-effective and scalable option for date-based queries.

Option B is correct because Amazon Bedrock Knowledge Bases is the managed AWS service for retrieval-augmented generation using an organization’s own content. AWS states that the RetrieveAndGenerate flow queries a knowledge base, generates responses based on retrieved results, and that the response includes citations only for relevant sources. AWS user guide pages also state that Knowledge Bases can return natural-language responses based on retrieved chunks from source documents. This directly fits the requirement for feedback with direct references to textbook sections.

This is also the least operational overhead choice because Amazon Bedrock manages ingestion, chunking, embedding, retrieval, and generation workflow components. Option A would require building and operating a custom vector solution. Option C adds multiple services and custom glue logic. Option D requires custom model hosting and fine-tuning, which is far more operationally heavy. For multilingual processing, Amazon Bedrock supports multilingual-capable models and multilingual embedding options, making it suitable for responses in the student’s selected language when paired with an appropriate model.

 The error message indicates that the AWS Glue job cannot access the Amazon S3 bucket through the VPC endpoint. This could be because the VPC’s route table does not have the necessary routes to direct the traffic to the endpoint. To fix this, the data engineer must verify that the route table has an entry for the Amazon S3 service prefix (com.amazonaws.region.s3) with the target as the VPC endpoint ID. This will allow the AWS Glue job to use the VPC endpoint to access the S3 bucket without going through the internet or a NAT gateway. For more information, see Gateway endpoints. References:

Troubleshoot the AWS Glue error “VPC S3 endpoint validation failed”

Amazon VPC endpoints for Amazon S3

[AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide]

AWS Step Functions is a fully managed, serverless orchestration service designed to coordinate multiple AWS services into reliable workflows. Step Functions natively provide automatic retries, error handling, state-level visibility, and execution history, which directly satisfy the requirements for retry logic and monitoring individual pipeline stages.

In this solution, AWS Lambda functions are used for batch data ingestion, AWS Glue jobs handle transformation, and Amazon S3 is used for storage. Step Functions orchestrate these stages while maintaining clear separation of responsibilities. Each step can be configured with retry and catch policies, allowing the workflow to automatically recover from transient failures without manual intervention.

Chaining AWS Glue jobs alone does not provide detailed stage-level visibility or flexible retry control, and setting MaxRetries to 0 contradicts the retry requirement. EventBridge-based pipelines lack native workflow state tracking. Amazon MWAA introduces unnecessary operational overhead, including environment management and Airflow maintenance.

Therefore, using AWS Step Functions to orchestrate Lambda and Glue provides the most scalable, observable, and low-maintenance solution.

The correct answer is B because Apache Iceberg is designed for analytical data lake tables that need schema evolution, transactional consistency, and historical snapshot access. AWS Glue supports Apache Iceberg for data lakes and can read and write Iceberg tables in Amazon S3 through the AWS Glue Data Catalog. The AWS Glue documentation also shows Iceberg tables being created with TBLPROPERTIES ( " format-version " = " 2 " ), which is the correct table format direction for modern Iceberg capabilities. CSV with scheduled conversion does not provide ACID transactions or point-in-time table snapshots. S3 Versioning tracks object versions but does not provide table-level ACID semantics. DynamoDB is not the best fit for an S3-based analytical data lake requirement.