Databricks-Certified-Associate-Developer-for-Apache-Spark-3.0 Dumps with Practice Exam Questions Answers

Explanation

itemsDf.withColumn("itemId", col("itemId").cast("string"))

Correct. You can convert the data type of a column using the cast method of the Column class. Also note that you will have to use the withColumn method on itemsDf for replacing the existing itemId

column with the new version that contains strings.

itemsDf.withColumn("itemId", col("itemId").convert("string"))

Incorrect. The Column object that col("itemId") returns does not have a convert method.

itemsDf.withColumn("itemId", convert("itemId", "string"))

Wrong. Spark's spark.sql.functions module does not have a convert method. The QUESTION NO: is trying to mislead you by using the word "converted". Type conversion is also called "type

casting". This

may help you remember to look for a cast method instead of a convert method (see correct answer).

itemsDf.select(astype("itemId", "string"))

False. While astype is a method of Column (and an alias of Column.cast), it is not a method of pyspark.sql.functions (what the code block implies). In addition, the QUESTION NO: asks to return a

full

DataFrame that matches the multi-column DataFrame itemsDf. Selecting just one column from itemsDf as in the code block would just return a single-column DataFrame.

spark.cast(itemsDf, "itemId", "string")

No, the Spark session (called by spark) does not have a cast method. You can find a list of all methods available for the Spark session linked in the documentation below.

More info:

- pyspark.sql.Column.cast — PySpark 3.1.2 documentation

- pyspark.sql.Column.astype — PySpark 3.1.2 documentation

- pyspark.sql.SparkSession — PySpark 3.1.2 documentation

Static notebook | Dynamic notebook: See test 3, QUESTION NO: 42 (Databricks import instructions)

Explanation

DataFrame.collect() sends all data in a DataFrame from executors to the driver, so this generally causes a great amount of network traffic in comparison to the other options listed.

DataFrame.coalesce() just reduces the number of partitions and generally aims to reduce network traffic in comparison to a full shuffle.

DataFrame.select() is evaluated lazily and, unless followed by an action, does not cause significant network traffic.

DataFrame.rdd.map() is evaluated lazily, it does therefore not cause great amounts of network traffic.

DataFrame.count() is an action. While it does cause some network traffic, for the same DataFrame, collecting all data in the driver would generally be considered to cause a greater amount of

network traffic.

Explanation

Correct code block:

spark.conf.set("spark.sql.shuffle.partitions", 100)

The conf interface is part of the SparkSession, so you need to call it through spark and not pyspark. To configure spark, you need to use the set method, not the get method. get reads a property,

but does not write it. The correct property to achieve what is outlined in the QUESTION NO: is spark.sql.aggregate.partitions, which needs to be passed to set as a string. Properties

spark.shuffle.partitions and spark.sql.aggregate.partitions do not exist in Spark.

Static notebook | Dynamic notebook: See test 2, QUESTION NO: 45 (Databricks import instructions)

Explanation

The challenge in this QUESTION NO: comes from there being an array variable in the schema. In addition, you should know how to pass a schema to the DataFrameReader that is invoked by

spark.read.

The correct way to define an array of strings in a schema is through ArrayType(StringType()). A schema can be passed to the DataFrameReader by simply appending schema(structType) to the

read() operator. Alternatively, you can also define a schema as a string. For example, for the schema of itemsDf, the following string would make sense: itemId integer, attributes array,

supplier string.

A thing to keep in mind is that in schema definitions, you always need to instantiate the types, like so: StringType(). Just using StringType does not work in pySpark and will fail.

Another concern with schemas is whether columns should be nullable, so allowed to have null values. In the case at hand, this is not a concern however, since the QUESTION NO: just asks for a

"valid"

schema. Both non-nullable and nullable column schemas would be valid here, since no null value appears in the DataFrame sample.

More info: Learning Spark, 2nd Edition, Chapter 3

Static notebook | Dynamic notebook: See test 3, QUESTION NO: 19 (Databricks import instructions)