Question # 1

54 of 55. What is the benefit of Adaptive Query Execution (AQE)? 

A. It allows Spark to optimize the query plan before execution but does not adapt during runtime. 
B. It automatically distributes tasks across nodes in the clusters and does not perform runtime adjustments to the query plan. 
C. It optimizes query execution by parallelizing tasks and does not adjust strategies based on runtime metrics like data skew. 
D. It enables the adjustment of the query plan during runtime, handling skewed data, optimizing join strategies, and improving overall query performance. 

Show Answer

Question # 2

54 of 55. What is the benefit of Adaptive Query Execution (AQE)? 

A. It allows Spark to optimize the query plan before execution but does not adapt during runtime. 
B. It automatically distributes tasks across nodes in the clusters and does not perform runtime adjustments to the query plan. 
C. It optimizes query execution by parallelizing tasks and does not adjust strategies based on runtime metrics like data skew. 
D. It enables the adjustment of the query plan during runtime, handling skewed data, optimizing join strategies, and improving overall query performance. 

Show Answer

Question # 3

49 of 55. In the code block below, aggDF contains aggregations on a streaming DataFrame: aggDF.writeStream \ .format("console") \ .outputMode("???") \ .start() Which output mode at line 3 ensures that the entire result table is written to the console during each trigger execution? 

A. AGGREGATE 
B. COMPLETE  
C. REPLACE 
D. APPEND

Show Answer

Question # 4

48 of 55. A data engineer needs to join multiple DataFrames and has written the following code: from pyspark.sql.functions import broadcast data1 = [(1, "A"), (2, "B")] data2 = [(1, "X"), (2, "Y")] data3 = [(1, "M"), (2, "N")] df1 = spark.createDataFrame(data1, ["id", "val1"]) df2 = spark.createDataFrame(data2, ["id", "val2"]) df3 = spark.createDataFrame(data3, ["id", "val3"]) df_joined = df1.join(broadcast(df2), "id", "inner") \ .join(broadcast(df3), "id", "inner") What will be the output of this code? 

A. The code will work correctly and perform two broadcast joins simultaneously to join df1 with df2, and then the result with df3.
B. The code will fail because only one broadcast join can be performed at a time. 
C. The code will fail because the second join condition (df2.id == df3.id) is incorrect. 
D. The code will result in an error because broadcast() must be called before the joins, not inline. 

Show Answer

Question # 5

47 of 55. A data engineer has written the following code to join two DataFrames df1 and df2: df1 = spark.read.csv("sales_data.csv") df2 = spark.read.csv("product_data.csv") df_joined = df1.join(df2, df1.product_id == df2.product_id) The DataFrame df1 contains ~10 GB of sales data, and df2 contains ~8 MB of product data. Which join strategy will Spark use?

A. Shuffle join, as the size difference between df1 and df2 is too large for a broadcast join to work efficiently.
B. Shuffle join, because AQE is not enabled, and Spark uses a static query plan. 
C. Shuffle join because no broadcast hints were provided. 
D. Broadcast join, as df2 is smaller than the default broadcast threshold. 

Show Answer

Question # 6

46 of 55. A data engineer is implementing a streaming pipeline with watermarking to handle late-arriving records. The engineer has written the following code: inputStream \ .withWatermark("event_time", "10 minutes") \ .groupBy(window("event_time", "15 minutes")) What happens to data that arrives after the watermark threshold?

A. Any data arriving more than 10 minutes after the watermark threshold will be ignored and not included in the aggregation. 
B. Records that arrive later than the watermark threshold (10 minutes) will automatically be included in the aggregation if they fall within the 15-minute window. 
C. Data arriving more than 10 minutes after the latest watermark will still be included in the aggregation but will be placed into the next window.
D. The watermark ensures that late data arriving within 10 minutes of the latest event time will be processed and included in the windowed aggregation.

Show Answer

Question # 7

45 of 55. Which feature of Spark Connect should be considered when designing an application that plans to enable remote interaction with a Spark cluster? 

A. It is primarily used for data ingestion into Spark from external sources. 
B. It provides a way to run Spark applications remotely in any programming language. 
C. It can be used to interact with any remote cluster using the REST API. 
D. It allows for remote execution of Spark jobs. 

Show Answer

Question # 8

44 of 55. A data engineer is working on a real-time analytics pipeline using Spark Structured Streaming. They want the system to process incoming data in micro-batches at a fixed interval of 5 seconds. Which code snippet fulfills this requirement? A. query = df.writeStream \ .outputMode("append") \ .trigger(processingTime="5 seconds") \ .start() B. query = df.writeStream \ .outputMode("append") \ .trigger(continuous="5 seconds") \ .start() C. query = df.writeStream \ .outputMode("append") \ .trigger(once=True) \ .start() D. query = df.writeStream \ .outputMode("append") \ .start() 

A. Option A 
B. Option B 
C. Option C 
D. Option D 

Show Answer

Question # 9

43 of 55. An organization has been running a Spark application in production and is considering disabling the Spark History Server to reduce resource usage. What will be the impact of disabling the Spark History Server in production?

A. Prevention of driver log accumulation during long-running jobs 
B. Improved job execution speed due to reduced logging overhead 
C. Loss of access to past job logs and reduced debugging capability for completed jobs 
D. Enhanced executor performance due to reduced log size 

Show Answer

Question # 10

42 of 55. A developer needs to write the output of a complex chain of Spark transformations to a Parquet table called events.liveLatest. Consumers of this table query it frequently with filters on both year and month of the event_ts column (a timestamp). The current code: from pyspark.sql import functions as F final = df.withColumn("event_year", F.year("event_ts")) \ .withColumn("event_month", F.month("event_ts")) \ .bucketBy(42, ["event_year", "event_month"]) \ .saveAsTable("events.liveLatest") However, consumers report poor query performance. Which change will enable efficient querying by year and month? 

A. Replace .bucketBy() with .partitionBy("event_year", "event_month") 
B. Change the bucket count (42) to a lower number 
C. Add .sortBy() after .bucketBy() 
D. Replace .bucketBy() with .partitionBy("event_year") only 

Show Answer