Databricks-Certified-Professional-Data-Scientist Questions and Answers

SVMs can be used to solve various real world problems:

• SVMs are helpful in text and hypertext categorization as their application can significantly reduce the need for labeled training instances in both the standard inductive and transductive settings.

• Classification of images can also be performed using SVMs. Experimental results show that SVMs achieve significantly higher search accuracy than traditional query refinement schemes after just three to four rounds of relevance feedback.

• SVMs are also useful in medical science to classify proteins with up to 90% of the compounds classified correctly.

• Hand-written characters can be recognized using SVM

In Phase 3, the data science team identifies candidate models to apply to the data for clustering, classifying, or finding relationships in the data depending on the goal of the project, It is during this phase that the team refers to the hypotheses developed in Phase 1, when they first became acquainted with the data and understanding the business problems or domain area. These hypotheses help the team frame the analytics to execute in Phase 4 and select the right methods to achieve its objectives.

Some of the activities to consider in this phase include the following: Assess the structure of the datasets. The structure of the datasets is one factor that dictates the tools and analytical techniques for the next phase. Depending on whether the team plans to analyze textual data or transactional data, for example, different tools and approaches are required.

Ensure that the analytical techniques enable the team to meet the business objectives and accept or reject the working hypotheses. Determine if the situation warrants a single model or a series of techniques as part of a larger analytic workflow. A few example models include association rules and logistic regression Other tools, such as Alpine Miner, enable users to set up a series of steps and analyses and can serve as a front-end user interface (Ul) for manipulating Big Data sources in PostgreSQL.

Logistic regression

Logistic regression is a model used for prediction of the probability of occurrence of an event. It makes use of several predictor variables that may be either numerical or categories.

Support Vector Machines

As with naive Bayes, Support Vector Machines (or SVMs) can be used to solve the task of assigning objects to classes. But the way this task is solved is completely different to the setting in naive Bayes.

Neural Network

Neural Networks are a means for classifying multidimensional objects.

Hidden Markov Models

Hidden Markov Models are used in multiple areas of machine learning, such as speech recognition, handwritten letter recognition, or natural language processing.

An advantage of naive Bayes is that it only requires a small amount of training data to estimate the parameters (means and variances of the variables) necessary for classification. Because independent variables are assumed, only the variances of the variables for each class need to be determined and not the entire covariance matrix.

From the definition, P(A,B|C) P(B|C) =P(A,B.C)/P(C) P(B.C)/P(C) =P(A,B.C)

P(B,C) =P(A|BC)

This follows from the definition of conditional probability, applied twice: P(A,B)=(PA|B)P(B)

Regularization is a very important technique in machine learning to prevent overfitting. Mathematically speaking, it adds a regularization term in order to prevent the coefficients to fit so perfectly to overfit. The difference between the L1 and L2 is just that L2 is the sum of the square of the weights, while L1 is just the sum of the weights. As follows: L1 regularization on least squares:

A picture containing text Description automatically generated

In unsupervised learning we don't have a target variable as we did in

classification and regression.

Instead of telling the machine Predict Y for our data X, we're asking What can you

tell me about X?

Things we ask the machine to tell us about

X may be What are the six best groups we can make out of X? or What three

features occur together most frequently in X?

To see how this is done, let W represent the event that the conversation was held with a woman, and L denote the event that the conversation was held with a long­haired person. It can be assumed that women constitute half the population for this example. So, not knowing anything else, the probability that W occurs is P(W) = 0.5. Suppose it is also known that 75% of women have long hair which we denote as P(L |W) = 0.75 (read: the probability of event L given event W is 0.75, meaning that the probability of a person having long hair (event "L"): given that we already know that the person is a woman ("event W") is 75%). Likewise, suppose it is known that 15% of men have long hair, or P(L |M) = 0.15; where M is the complementary event of W: i.e.; the event that the conversation was held with a man (assuming that every human is either a man or a woman).

Our goal is to calculate the probability that the conversation was held with a woman, given the fact that the person had long hair, or, in our notation, P(W |L). Using the formula for Bayes' theorem, we have:

Text Description automatically generated with low confidence

where we have used the law of total probability to expand

P(L),

The numeric answer can be obtained by substituting the above values into this formula (the algebraic multiplication is annotated using " *", the centered dot). This yields

A picture containing table Description automatically generated

i.e., the probability that the conversation was held with a woman, given that the person had long hair is about 83%. More examples are provided below.

Point estimators

• minimum-variance mean-unbiased estimator (MVUE), minimizes the risk (expected loss) of the squared-error loss-function.

• best linear unbiased estimator (BLUE)

• minimum mean squared error (MMSE)

• median-unbiased estimator, minimizes the risk of the absolute-error loss function

• maximum likelihood (ML)

• method of moments, generalized method of moments

Error calculation allows you to see how well a machine learning

method is performing.

One way of determining this performance is to calculate a numerical error This number is sometimes a percent,

however it can also be a score or distance. The goal is usually to minimize an error

percent or distance:

however th goal may be to minimize or maximize a score. Encog supports the

following error calculation methods.

Sum of Squares Error (ESS)

Root Mean Square Error (RMS)

Mean Square Error (MSE) (default)

SOM Error (Euclidean Distance Error)

RMSE measures error of a predicted numeric value, and so applies to contexts like

regression and some recommender system techniques,

which rely on predicting a numeric value. It is not relevant to classification

techniques

like logistic regression and Naive Bayes, which predict categorical values.

It also is not relevant to unsupervied techniques like clustering.

The root-mean-square deviation (RMSD) or root-mean-square error (RMSE) is a

frequently used measure of the

differences between values predicted by a model or an estimator and the values

actually observed. Basically,

the RMSD represents the sample standard deviation of the differences between

predicted values and observed values.

These individual differences are called residuals when the calculations are

performed over the data sample that was used for estimation,

and are called prediction errors when computed out-of-sample. The RMSD serves

to aggregate the magnitudes

of the errors in predictions for various times into a single measure of predictive

power. RMSD is a good measure of accuracy,

but only to compare forecasting errors of different models for a particular variable

and not between variables, as it is scale-dependent.

The total number of clicks is the sum of the number of men and

women. The sum of two Poisson random variables also follows a Poisson distribution with

rate equal to the sum of their rates.

The Normal and Binomial distribution can approximate the Poisson distribution in

certain cases, but the expressions above do not approximate Poisson(X+Y).

In a binomial distribution, only 2 parameters, namely n and p, are needed to determine the probability. Where p is the probability of success and q is the probability of failure in a binomial trial, then the expected number of successes in n trials.

This is a binomial distribution because there are only 2 possible outcomes (we get a 5 or we don't).

Smooth the estimates: consider flipping a coin for which the probability of heads is p, where p is unknown, and our goal is to estimate p. The obvious approach is to count how many times the coin came up heads and divide by the total number of coin flips. If we flip the coin 1000 times and it comes up heads 367 times, it is very reasonable to estimate p as approximately 0.367. However, suppose we flip the coin only twice and we get heads both times. Is it reasonable to estimate p as 1.0? Intuitively, given that we only flipped the coin twice, it seems a bit rash to conclude that the coin will always come up heads, and smoothing is a way of avoiding such rash conclusions. A simple smoothing method, called Laplace smoothing (or Laplace's law of succession or add-one smoothing in R&N), is to estimate p by (one plus the number of heads) / (two plus the total number of flips). Said differently, if we are keeping count of the number of heads and the number of tails, this rule is equivalent to starting each of our counts at one, rather than zero. Another advantage of Laplace smoothing is that it avoids estimating any probabilities to be zero, even for events never observed in the data. Laplace add-one smoothing now assigns too much probability to unseen words