Databricks-Machine-Learning-Professional Questions and Answers

To obtain run-level results for an experiment ID in a Spark DataFrame, you can use the mlflow.search_runs(exp_id) function. This function takes a filter string, which acts as a filter to the query, and returns a pandas.DataFrame of runs, where each metric, parameter, and tag are expanded into their own columns named metrics., params., and tags.* respectively1. You can then convert the pandas.DataFrame to a Spark DataFrame using the spark.createDataFrame() method2. The other options are incorrect because:

• Option A: client.list_run_infos(exp_id) will return a list of RunInfo objects, not a DataFrame3.
• Option B: spark.read.format(“delta”).load(exp_id) will not work for MLflow experiments, as it expects a Delta table path, not an experiment ID4.
• Option C: There is a way to programmatically return row-level results from an MLflow experiment using the mlflow.search_runs(exp_id) function1.
• Option E: spark.read.format(“mlflow-experiment”).load(exp_id) is not a valid format for reading MLflow experiments, as there is no such format registered in Spark5. References: Search Runs, Create a DataFrame from an RDD, a list or a pandas.DataFrame, MlflowClient.list_run_infos, Delta Lake Quickstart, spark.read.format

To move a model version from the Staging stage to the Production stage using the MLflow Client, you can use the client.transition_model_version_stage method. This method takes the name of the registered model, the version of the model, and the target stage as arguments. It transitions the model version to the specified stage and returns the updated model version object. The syntax of the method is as follows:

Python

client.transition_model_version_stage (name, version, stage)

AI-generated code. Review and use carefully. More info on FAQ.

The other options are incorrect because:

• Option A: client.transition_model_version_stage with the stage argument set to “Staging” will not move the model version from the Staging stage to the Production stage, but rather keep it in the same stage.
• Option B: client.transition_model_stage is not a valid method name, as it is missing an underscore. The correct method name is client.transition_model_version_stage.
• Option D: client.transition_model_stage is not a valid method name, as it is missing an underscore. The correct method name is client.transition_model_version_stage. Also, the method does not take from and to arguments, but only a stage argument.
• Option E: client.transition_model_version_stage does not take from and to arguments, but only a stage argument. The from argument is redundant, as the method will automatically transition the model version from its current stage tothe target stage. References: MLflow Model Registry, MlflowClient.transition_model_version_stage

Data type tests and checks are particularly important for streaming deployments because of the following reasons:

• Because the streaming deployment is always on, all types of data must be handled without producing an error. Streaming data may come from various sources and have different formats, schemas, or quality. If the streaming deployment cannot handle the data properly, it may result in errors, failures, or inaccurate outputs. Therefore, data type tests and checks can help validate the data and ensure the streaming deployment can process it correctly.
• Because the streaming deployment is always on, there is no practitioner to debug poor model performance. Streaming data may also change over time, affecting the model performance and accuracy. If the streaming deployment does not have a mechanism to monitor and evaluate the model performance, it may not be able to detect or correct any issues. Therefore, data type tests and checks can help identify any data drift or anomalies that may impact the model performance and trigger alerts or actions.
• Because the streaming deployment is always on, there is a need to confirm that the deployment can autoscale. Streaming data may have varying volumes and velocities, requiring the streaming deployment to scale up or down accordingly. If the streaming deployment does not have a way to adjust the resources and throughput based on the data demand, it may result in bottlenecks, delays, or inefficiencies. Therefore, data type tests and checks can help measure the data load and ensure the streaming deployment can autoscale effectively. References: Streaming Data, Data Validation, Model Monitoring, Autoscaling

Categorical feature drift is a change in the distribution of the input data over time for categorical features, which can affect the performance and accuracy of the model. Monitoring categorical feature drift is important to ensure that the model is still valid and reliable for the current data. One simple statistic to monitor for categorical feature drift is the combination of mode, number of unique values, and percentage of missing values for each categorical feature. These statistics can provide a quick overview of the changes in the data distribution, such as the most frequent category, the diversity of categories, and the quality of data. If these statistics deviate significantly from the baseline values, it may indicate a categorical feature drift. However, these statistics may not capture all the nuances of the data distribution, such as the relative frequencies of different categories, the similarity of categories, etc. Therefore, other methods, such as statistical tests or information-theoretic measures, may be needed to complement the simple statistics and provide a more comprehensive analysis of the categorical feature drift123 References:

• Monitoring Feature Drift - Databricks
• Drift Metrics: How to Select the Right Metric to Analyze Drift
• Detect data drift on datasets (preview) - Azure Machine Learning

The client.delete_registered_model method of the MlflowClient class can be used to delete a model from the MLflow Model Registry. The method takes the following parameter:

• name: The name of the registered model to delete.

The method deletes the registered model and all its versions from the Model Registry. The method does not return any value1

The other options are incorrect because:

• A. The client.transition_model_version_stage method can be used to transition a model version to a different stage, such as None, Staging, Production, or Archived. The method does not delete the model or the model version from the Model Registry2
• B. The client.delete_model_version method can be used to delete a specific model version from the Model Registry. The method does not delete the registered model or all its versions from the Model Registry3
• C. The client.update_registered_model method can be used to update the metadata of a registered model, such as its name or description. The method does not delete the model or its versions from the Model Registry4
• D. There is no method called client.delete_model. To delete a model from the Model Registry, the client.delete_registered_model method should be used.

References:

• mlflow.tracking.client.MlflowClient.delete_registered_model — MLflow 2.9.1 documentation
• mlflow.tracking.client.MlflowClient.transition_model_version_stage — MLflow 2.9.1 documentation
• mlflow.tracking.client.MlflowClient.delete_model_version — MLflow 2.9.1 documentation
• mlflow.tracking.client.MlflowClient.update_registered_model — MLflow 2.9.1 documentation

A two-way Chi-squared test is a statistical test that can be used to compare the frequencies of categorical variables across two or more groups. In this case, the engineer can use a two-way Chi-squared test to compare the frequencies of the categorical input variable values across different time periods (e.g., months, quarters, etc.) and see if there is a significant difference in the proportion of missing values. A significant difference would indicate that the missing values are not randomly distributed and may affect the performance of the machine learning model. References:

• Databricks Machine Learning Professional Exam Guide, Section 4: Solution and Data Monitoring, p. 5
• Databricks Machine Learning Documentation, Monitoring ML Models, Data Drift Detection, p. 3

Concept drift is a change in the relationship between input and output data over time, which can affect the performance and accuracy of the model. To monitor concept drift, one needs to compare the predicted values from the model with the observed (actual) label values from the data source. A common way to do this is to compute an evaluation metric, such as accuracy, precision, recall, F1-score, etc., using the observed and predicted values. This metric can indicate how well the model is performing on the current data, and how it differs from the baseline performance on the training data. If the metric shows a significant decline over time, it may suggest that concept drift has occurred and the model needs to be updated123 References:

• Monitoring Concept Drift - Databricks
• Data drift, concept drift, and how to monitor for them - Mona Labs
• 8 Concept Drift Detection Methods To Use With ML Models - Aporia

To replace all data in a feature table with newly computed data, you can use the fs.write_table method with the mode argument set to “overwrite”. This will overwrite the existing data in the feature table with the data in the provided DataFrame. The other options are incorrect because:

• Option A: fs.create_table will create a new feature table, not replace the data in an existing one. If the feature table already exists, it will raise an exception.
• Option B: fs.write_table without specifying the mode argument will use the default mode of “merge”, which will merge the new data with the existing data, not replace it.
• Option C: fs.write table is not a valid method name, as it is missing an underscore. Also, the mode argument of “merge” will not replace the data, but merge it with the existing data.
• Option E: fs.create_table with the mode argument of “merge” is not a valid combination, as the mode argument is only applicable for fs.write_table. Also, the mode argument of “merge” will not replace the data, but merge it with the existing data. References: Databricks FeatureStoreClient, Work with features in Workspace Feature Store

Batch serving is a type of model deployment where predictions are computed in advance and stored for later use. This can be useful when the feature values are known ahead of time and the predictions need to be delivered quickly. Querying stored predictions can be faster than computing predictions in real-time, especially if the model is complex or the data volume is high. Real-time serving is a type of model deployment where predictions are computed on-demand at query time. This can be useful when the feature values are not known in advance or the predictions need to be up-to-date. However, computing predictions in real-time can be slower than querying stored predictions, especially if the model is simple or the data volume is low. References: [Batch Serving], [Real-time Serving] Message has links.

The correct way to log a scikit-learn model using MLflow is to use the mlflow.sklearn.log_model function, which logs the model as an MLflow artifact for the current run1. This function takes the scikit-learn model object and an artifact path as required arguments, and optionally accepts other arguments such as conda environment, code paths, serialization format, etc2. The other options are incorrect because they either use the wrong flavor (spark instead of sklearn), the wrong function (load_model or track_model instead of log_model), or do not exist (track_model is not a valid MLflow function). References: MLflow Models, mlflow.sklearn

The MLflow Model Registry allows users to manage the full lifecycle of MLflow Models. It provides model lineage (which MLflow Experiment and Run produced the model), model versioning, stage transitions (for example from staging to production), and annotations. Users can use the MLflow Client API to interact with the Model Registry programmatically.

To move a model version from one stage to another, users can use the transition_model_version_stage method of the MLflow Client. This method takes the following parameters:

• name: The name of the registered model.
• version: The version number of the model.
• stage: The name of the stage to transition the model version to.
• archive_existing_versions: (Optional) If True, archive all other model versions in the same stage.

Therefore, to move the model version model_version for the model model from the Staging stage to the Production stage, and archive any existing model versions in the Production stage, the correct code block is:

PythonAI-generated code. Review and use carefully. More info on FAQ.

client.transition_model_version_stage(

name=model,

version=model_version,

stage="Production",

archive_existing_versions=True

)

References:

• [MLflow Model Registry - Databricks]
• [MLflow Client API - Databricks]
• [MLflow Model Registry API - Databricks] Message has links.

Feature drift, also known as data drift or covariate shift, occurs when the distribution of the input data changes over time, but the relationship between the input and the output remains the same. In this scenario, the expected temperature is an input feature that has changed its distribution, but the output (ice cream sales) still depends on the input in the same way. Feature drift can affect the performance of the model if the new data falls outside the range of the training data, as the model may not have learned how to handle such cases. Feature drift can be detected by monitoring the statistics of the input data over time and comparing them with the baseline statistics from the training data. References:

• Databricks Machine Learning Documentation, Monitoring ML Models, Data Drift Detection, p. 2
• A Gentle Introduction to Concept Drift in Machine Learning, Types of Concept Drift, p. 3
• Understanding Data Drift and Model Drift: Drift Detection in Python, Types of Drift, p. 2

 In a continuous integration, continuous deployment (CI/CD) process for machine learning pipelines, the arrival of a new model version in the MLflow Model Registry is a common event that triggers the execution of automated testing. This is because the MLflow Model Registry is a centralized repository for managing the full lifecycle of MLflow models. It provides model lineage, model versioning, stage transitions, and annotations. When a new model version is registered, it can be automatically tested for quality, performance, and compliance using various tools and frameworks, such as MLflow Projects, MLflow Models, MLflow Tracking, and MLflow Model Serving. The automated testing can help ensure that the new model version meets the desired criteria and standards before it is deployed to production or other stages. The automated testing can also help detect and prevent any errors, bugs, or drifts that may affect the model functionality or accuracy. References:

• MLflow Model Registry Documentation, p. 1-2
• MLOps: Continuous delivery and automation pipelines in machine learning, Continuous integration and delivery (CI/CD) for ML, p. 5-6
• Machine Learning Engineering with MLflow, Testing and Validating ML Models, p. 1-2
• CI/CD for Machine Learning: What it is & Benefits in 2023 - AIMultiple, CI/CD for Machine Learning, p. 1-2