How to calculate my own cost matrix?

[vousmevoyez]

In your tutorial, I saw:

X_train, X_test, y_train, y_test, cost_mat_train, cost_mat_test = \
train_test_split(data.data, data.target, data.cost_mat)

But data is built-in. How about my own data which doesn't have .cost_mat attribute?

[felipeolifre]

Hi @vousmevoyez, what problem are you working on? The calculation of the cost matrix depends on it. Each problem has his own implicit costs, i.e., how much it costs a false positive, false negative, true positive and true negative respectively.

You can look at /costcla/datasets/base.py to see how the cost matrix is being calculated for the datasets included in the library.

If you code the matrix-calculation by your own, it's important to know that each sample in the dataset should have its own cost matrix (an array containing the costs in the following order: FP, FN, TP, TN). A very abstract example code could be:

cost_mat = np.zeros((n_samples, 4))

# TODO: Define the cost for each prediction class
cost_mat[:, 0] = false_positive_cost
cost_mat[:, 1] = false_negative_cost
cost_mat[:, 2] = true_positive_cost
cost_mat[:, 3] = true_negative_cost

Hope this helps :)

[vousmevoyez]

Thanks.

[denson]

I have several problems where the cost matrix is the same for all samples within that problem. In other words, the cost of a false positive is the same for all samples in problem 1.

I am testing this out with the sklearn make classification problem:

n_samples = 50000
n_features = 100
n_estimators = 10 #RF estimators
# Build a classification task using 3 informative features
X, y = make_classification(n_samples=n_samples,
                           n_features=n_features,
                           n_informative=30,
                           n_redundant=5,
                           n_repeated=0,
                           n_classes=2,
                           random_state=0,
                           shuffle=False,
                           class_sep=0.5)

I made the classes imbalanced:

                       Frequency  Percentage
Negative (repairable)      25020    0.667022
Positive (total loss)      12490    0.332978

Here are the first 4 rows of the cost matrix for on of the problems:

[[ 400.  300.  200.  100.]
 [ 400.  300.  200.  100.]
 [ 400.  300.  200.  100.]]

Here are the results from multiple models (CSRF is CostSensitiveRandomForestClassifier):

             pre        f1       acc       rec       sav
RF      0.866229  0.895439  0.855513  0.926689  0.142192
LR      0.742509  0.811418  0.722435  0.894426 -0.010320
DT      0.830740  0.833187  0.776605  0.835649  0.092877
RF-BMR  0.914974  0.860961  0.824696  0.812969  0.167626
LR-BMR  0.859082  0.611888  0.597569  0.475164  0.068995
DT-BMR  0.830740  0.833187  0.776605  0.835649  0.092877
CSRF    0.935275  0.308390  0.447110  0.184635  0.041826
CSDT    0.897365  0.460826  0.515675  0.310014  0.058219
CSRP    0.667626  0.800691  0.667626  1.000000 -0.141096

In all the tests I have done so far RF-BMR is better by a substantial amount. Am I missing out on better parameters for one of the other cost sensitive models or does the RF-BMR just happen to be better for this type of problem?