Predict Probability Using Custom Threshold

In our previous article we have used a titanic data set to create one machine learning model which will predict whether a passenger will survive or not. But one thing to note is that last time we used the “predict” method of model and when we used the predict method, the model uses 0.5 as the threshold and provides you the result in the form of integer values (0 or 1).

 

What do you mean by Threshold?

Threshold is the judging criteria. 

When a model learns the mapping between X and Y (or we can say mapping between your independent variables and the target) and later on while making the prediction, model provides the probability. Here probability means how confident the model is about its prediction.

Example: Suppose there is a cricket match scheduled tomorrow between India and Pakistan and the question is who will win the match?

    Can we definitely say that India will win?

No

Since most of the events in our day to day life will not surely happen, hence we have to provide a confidence level between 0 to 100. 

• There is an 85% chance that Team India is going to win.
• There is only a 15% chance that it is going to rain today.

 

Similarly, during training the model learns X to Y mapping and later at the time of prediction, it returns the probability. But based on the function that you have used, you may either get probability or you may get class names 0 or 1.

 

Methods:

1) predict():

predict method uses 0.5 as the probability cutoff.

If the predicted probability is > 0.5, then the output is tagged as 1 else 0.

 

In our previous article we had used the predict() method where the model calculated the probability internally and used 0.5 as threshold and later provided the output 0 or 1.

 

2) predict_proba():

predict_proba method doesn’t use any cutoff for classifying the data. Rather it outputs the raw probability value itself.

In this article we are going to take the same titanic data set and use predict_proba method instead of predict.

 

Modeling Exercise:

If you have worked on some machine learning model, then you would be familiar with the flow. It contains various steps in sequence:

1) Import Library

2) Reading Data

3) Preprocessing Data

4) Training the Model

5) Model Predictions

6) Evaluating the Model output

  

## Import Libraries 

import pandas as pd

import numpy as np

from sklearn.model_selection import train_test_split

from sklearn.ensemble import RandomForestClassifier

from sklearn.metrics import precision_score, recall_score, accuracy_score, f1_score, fbeta_score

## Import Data 

data = pd.read_csv("titanic-data.csv")

data = data[['Fare', 'Age', 'Sex', 'Survived']]

print("No. of records are:", data.shape)

data.head()

   

 

  

## PreProcessing

def replace_string(value):

    if value == "male":

        return 0

    if value == "female":

        return 1

data['Sex'] = data['Sex'].apply(replace_string)

data['Age'] = data['Age'].fillna(data['Age'].mean())

data['Age'].describe()

data.head()

 

  

## Model Training 

X = data.drop(['Survived'], axis = 1)

y = data['Survived']

X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42, test_size=0.3)

predictions = rc.predict(X_test)

predictions_prob = rc.predict_proba(X_test)

y_test = y_test.reset_index(drop = True)

result = pd.DataFrame()

result['Actual_Target'] = y_test

result['Predicted_Target'] = predictions

result['Predicted_Probability'] = predictions_prob[:, 1]

result.head()

 

 

 

## Confusion Matrix 

pd.crosstab(result['Predicted_Target'], result['Actual_Target'])

 

## Custom Threshold 

result['Custom_Target'] = result['Predicted_Probability'].apply(lambda r: 1 if r > 0.6 else 0)

result.head()

 

## Calculating Metrics - Predicted Target 

tp = result[(result['Actual_Target'] == 1) & (result['Predicted_Target'] == 1)].shape[0]

fn = result[(result['Actual_Target'] == 1) & (result['Predicted_Target'] == 0)].shape[0]

tn = result[(result['Actual_Target'] == 0) & (result['Predicted_Target'] == 0)].shape[0]

fp = result[(result['Actual_Target'] == 0) & (result['Predicted_Target'] == 1)].shape[0]

Accuracy = (tp + tn) / (tp + fn + tn + fp)

Precision = (tp) / (tp + fp)

Recall = (tp) / (tp + fn)

F1Score = 2*Precision*Recall/(Precision+Recall)

Precision, Recall, Accuracy, F1Score

 

Output:

(0.7029702970297029,

 0.6893203883495146,

 0.7686567164179104,

 0.6960784313725491)

 

## Calculating Metrics - Custom Target 

tp = result[(result['Actual_Target'] == 1) & (result['Custom_Target'] == 1)].shape[0]

fn = result[(result['Actual_Target'] == 1) & (result['Custom_Target'] == 0)].shape[0]

tn = result[(result['Actual_Target'] == 0) & (result['Custom_Target'] == 0)].shape[0]

fp = result[(result['Actual_Target'] == 0) & (result['Custom_Target'] == 1)].shape[0]

Accuracy = (tp + tn) / (tp + fn + tn + fp)

Precision = (tp) / (tp + fp)

Recall = (tp) / (tp + fn)

F1Score = 2*Precision*Recall/(Precision+Recall)

Precision, Recall, Accuracy, F1Score

 

Output:

(0.7674418604651163,

 0.6407766990291263,

 0.7873134328358209,

 0.6984126984126984)

  

We can see that by using a custom threshold of 0.6, accuracy and F1 score of the model has increased. But Precision and Recall are affected.

 

When do you need to use a custom threshold?

• When the target class is not balanced
• When you want to focus on one class more and want to capture as many items from a specific class.

 

How do you know what is the right threshold to be used for decisioning?

You can write one loop and under which you can check for every possible threshold value e.g. 0, 0.001, 0.002, 0.003, … . . . 0.110, 0.111, 0.112, . . . . 0.889, 0.890, .. . . . 0.999, 1.

And check for the threshold where you get maximum F1 Score / Precision / Recall etc.

 

If you want to download the jupyter notebook file for the above code, click here .