keras: Can't use auc_roc_score (or any other metric function based on numpy) as a custom metric?

Metrics functions must be symbolic functions (built with the Keras backend, or with Theano/TensorFlow).

Also ROC AUC is not a metric that be accumulated in mini-batches, it has to be computed for all the data at once.

The right thing to do is to run predictions on all of your test data at the end of an epoch, then run the sklearn function on your predictions, and display the result. You can do this in a callback.

#-----------------------------------------------------------------------------------------------------------------------------------------------------
# AUC for a binary classifier
def auc(y_true, y_pred):   
    ptas = tf.stack([binary_PTA(y_true,y_pred,k) for k in np.linspace(0, 1, 1000)],axis=0)
    pfas = tf.stack([binary_PFA(y_true,y_pred,k) for k in np.linspace(0, 1, 1000)],axis=0)
    pfas = tf.concat([tf.ones((1,)) ,pfas],axis=0)
    binSizes = -(pfas[1:]-pfas[:-1])
    s = ptas*binSizes
    return K.sum(s, axis=0)

#-----------------------------------------------------------------------------------------------------------------------------------------------------
# PFA, prob false alert for binary classifier
def binary_PFA(y_true, y_pred, threshold=K.variable(value=0.5)):
    y_pred = K.cast(y_pred >= threshold, 'float32')
    # N = total number of negative labels
    N = K.sum(1 - y_true)
    # FP = total number of false alerts, alerts from the negative class labels
    FP = K.sum(y_pred - y_pred * y_true)    
    return FP/N
#-----------------------------------------------------------------------------------------------------------------------------------------------------
# P_TA prob true alerts for binary classifier
def binary_PTA(y_true, y_pred, threshold=K.variable(value=0.5)):
    y_pred = K.cast(y_pred >= threshold, 'float32')
    # P = total number of positive labels
    P = K.sum(y_true)
    # TP = total number of correct alerts, alerts from the positive class labels
    TP = K.sum(y_pred * y_true)    
    return TP/P

class roc_callback(keras.callbacks.Callback):
    def __init__(self,training_data,validation_data):
        
        self.x = training_data[0]
        self.y = training_data[1]
        self.x_val = validation_data[0]
        self.y_val = validation_data[1]
        
    
    def on_train_begin(self, logs={}):
        return
 
    def on_train_end(self, logs={}):
        return
 
    def on_epoch_begin(self, epoch, logs={}):
        return
 
    def on_epoch_end(self, epoch, logs={}):        
        y_pred = self.model.predict(self.x)
        roc = roc_auc_score(self.y, y_pred)      
        
        y_pred_val = self.model.predict(self.x_val)
        roc_val = roc_auc_score(self.y_val, y_pred_val)      
        
        print('\rroc-auc: %s - roc-auc_val: %s' % (str(round(roc,4)),str(round(roc_val,4))),end=100*' '+'\n')
        return
 
    def on_batch_begin(self, batch, logs={}):
        return
 
    def on_batch_end(self, batch, logs={}):
        return   


callbacks=[roc_callback(training_data=training_data,validation_data=validation_data)] 

@JoshuaC3 The way to make @jamartinh 's solution to work with fit_generator is by making these changes:

1. In __init__, you should pass in the train and val generators instead.

2. In on_epoch_end, replace model.predict with model.predict_generator.

Here’s a sketch where only the AUROC of the val dataset is calculated at the end of every epoch:

class roc_callback(keras.callbacks.Callback):
    def __init__(self, val_gen):
        self.val_gen = val_gen
        self.val_reports = []

    def on_epoch_end(self, epoch, logs={}):        
        y_pred = self.model.predict_generator(val_gen, .....)
        y_true = self.val_gen.y
        val_roc = roc_auc_score(y_true , y_pred)
        self.val_reports.append(val_roc)

@fchollet

Also ROC AUC is not a metric that be accumulated in mini-batches, it has to be computed for all the data at once.

But its reasonable to have this metric computed on the validation set at the end of each epoch.

The right thing to do is to run predictions on all of your test data at the end of an epoch, then run the sklearn function on your predictions, and display the result. You can do this in a callback.

Correct, but the problem is that since its not in the keras metrics form, you don’t get the output to tensorboard. Am I correct?

@solensolen are you sure about the metrics method working? I’m using both the methods and the validation AUROC I get from the metrics method increases with each epoch. Towards the end of training, the metrics AUROC is way higher than the callback one (fyi my dataset is pretty imbalanced)

I thought it’s like what @isaacgerg said that it is calculated batch-by-batch and averaged which is why it is so high as some batches may not have the underpopulated class. So I decided to make my validation data the same size as my batch size so that it is calculated in one go and I am still experiencing the aforementioned phenomenon. Here is a sample from the output (batch size is 1000).

def auc_roc(y_true, y_pred):
	value, update_op = tf.metrics.auc(y_true, y_pred)
	K.get_session().run(tf.local_variables_initializer())
	return update_op

Train on 44088 samples, validate on 1000 samples
Epoch 1/15
44088/44088 [==============================] - 3s 78us/step - loss: 0.6872 - acc: 0.5674 - auc_roc: 0.5267 - val_loss: 0.6672 - val_acc: 0.6770 - val_auc_roc: 0.5473

AUC (from callback) - 0.5823
.
.
.
.

Epoch 5/15
44088/44088 [==============================] - 2s 55us/step - loss: 0.3236 - acc: 0.8919 - auc_roc: 0.9386 - val_loss: 0.5749 - val_acc: 0.7430 - val_auc_roc: 0.9279

AUC (from callback) - 0.5860

The val_auc_roc is calculated by passing the auc_roc function to the model.compile method and the AUC (from callback) is the same as the roc_callback class defined in an above post with only validation data AUC calculated. In epoch 1, the values are similar but by further epochs it’s way higher. Can anyone explain why is the val_auc_roc so different from the AUC (from callback)?

@fchollet

def auc(y_true, y_pred):   
    keras.backend.get_session().run(tf.global_variables_initializer())
    #return K.variable(value=tf.contrib.metrics.streaming_auc(y_pred, y_true)[0], dtype='float32')
    return tf.contrib.metrics.streaming_auc(y_pred, y_true)[0]

I don’t see why this yields, “tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value auc/true_positives…”

@jamartinh - how to get this to work then validation_data is a flow_from_directory generator?

I get the error

TypeError: Error when checking model : data should be a Numpy array, or list/dict of Numpy arrays.
Found: (array([[[[ 0.29411766,  0.34117648,  0.4039216 ], ...

https://www.kaggle.com/rspadim/gini-keras-callback-earlystopping-validation

gini = 2*auc -1

@NickYi1990 by default the batch size is 32, since the roc auc must use all the validation data to be correct, you need to give a batch size equal to the size of you validation set. In your case:

model.fit(X_train.values, y_train.values, validation_split=0.2, epochs=1, verbose=1, batch_size=X_train.values.shape[0]/5.)

@NickYi1990 I had the same issue - I figured out that some of the thresholds had 0 true positives / false negatives. This led to a division by 0 -> return FP / N A decent solution is: return FP / N -> return FP / (N + 1)

import tensorflow as tf
import keras.backend as K
import numpy as np


# -----------------------------------------------------------------------------------------------------------------------------------------------------
# AUC for a binary classifier
def auc(y_true, y_pred):
    ptas = tf.stack([binary_PTA(y_true, y_pred, k) for k in np.linspace(0, 1, 1000)], axis=0)
    pfas = tf.stack([binary_PFA(y_true, y_pred, k) for k in np.linspace(0, 1, 1000)], axis=0)
    pfas = tf.concat([tf.ones((1,)), pfas], axis=0)
    binSizes = -(pfas[1:] - pfas[:-1])
    s = ptas * binSizes
    return K.sum(s, axis=0)


# -----------------------------------------------------------------------------------------------------------------------------------------------------
# PFA, prob false alert for binary classifier
def binary_PFA(y_true, y_pred, threshold=K.variable(value=0.5)):
    y_pred = K.cast(y_pred >= threshold, 'float32')
    # N = total number of negative labels
    N = K.sum(1 - y_true)
    # FP = total number of false alerts, alerts from the negative class labels
    FP = K.sum(y_pred - y_pred * y_true)
    return FP / (N + 1)


# -----------------------------------------------------------------------------------------------------------------------------------------------------
# P_TA prob true alerts for binary classifier
def binary_PTA(y_true, y_pred, threshold=K.variable(value=0.5)):
    y_pred = K.cast(y_pred >= threshold, 'float32')
    # P = total number of positive labels
    P = K.sum(y_true)
    # TP = total number of correct alerts, alerts from the positive class labels
    TP = K.sum(y_pred * y_true)
    return TP / (P + 1)

Technically AUC ROC can be calculated on mini-batch as long as we have the y_true and y_pred. The only concern should be that too small batch size will reduce the accuracy of this metric and make it less meaningful. Am I right? @fchollet

This works:

def auc(y_true, y_pred):   
    ptas = tf.stack([binary_PTA(y_true,y_pred,k) for k in np.linspace(0, 1, 1000)],axis=0)
    pfas = tf.stack([binary_PFA(y_true,y_pred,k) for k in np.linspace(0, 1, 1000)],axis=0)
    pfas = tf.concat([tf.ones((1,)) ,pfas],axis=0)
    binSizes = -(pfas[1:]-pfas[:-1])
    s = ptas*binSizes
    return K.sum(s, axis=0)