Sigmoid activation output exceed [0,1]



  • Hi There,

    Can you help me with the following problem:

    Short version:
    I converted a tflite model using nncase v0.2. The sigmoid output range before conversion is [0.06289272 0.99822897] (which is correct), and after conversion is [0.0 15.686275] - which exceeds the [0,1] output range of sigmoid function.

    Long version:

    The DNN I'm trying to use is a simple autoencoder with [1,240,320,1] input, and the output has the same shape with sigmoid activation. I created a minimum example to demonstrate the problem as infer.py, which could be downloaded along with test data and results from:
    https://www.dropbox.com/s/a1ux6rj5wzlgvbi/example.zip?dl=0

    import tensorflow as tf # tf 2.1
    from PIL import Image
    import numpy as np
    import os, sys
    
    # load TFlite model
    interpreter = tf.lite.Interpreter("model.tflite")
    interpreter.allocate_tensors()
    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()
    
    # Prepare data
    input_img = Image.open("./cal/000000581100.jpg").convert('L')
    in_data = np.array(input_img).astype(np.float32)/255.0
    in_data = in_data[np.newaxis, :, :, np.newaxis]
    #print(in_data.shape)
    
    # Infer on TFlite model
    interpreter.set_tensor(input_details[0]['index'], in_data)
    interpreter.invoke()
    out_data = interpreter.get_tensor(output_details[0]['index'])
    
    print("TFLite sigmoid output: ", out_data.shape, np.min(out_data), np.max(out_data))
    #TFLite sigmoid output:  (1, 240, 320, 1) 0.06289272 0.99822897
    
    # convert and infer kmodel use nncase, runs on Ubuntu
    os.system("./ncc compile model.tflite model.kmodel -i tflite -o kmodel --dataset ./cal/ --inference-type uint8")
    os.system("./ncc infer model.kmodel . --dataset ./cal/")
    
    # read ncc infer output
    ncc_out = np.fromfile("000000581100.bin",dtype=np.float32)
    print("NCC sigmoid output: ", ncc_out.shape, np.min(ncc_out), np.max(ncc_out))
    # NCC sigmoid output:  (76800,) 0.0 15.686275
    

    I have carefully read though usage, tutorials, FAQs, issues from https://github.com/kendryte/nncase but cannot find an answer.
    Any input that helps explain the huge (and strange) discrepancy between TFLite result and NNCase output is greatly appreciated! Thanks,

    -Jacky



  • I tested your model on my nncase which is built from github source, it is pretty much the latest code base, it is still producing a value of 11, instead of 15, you can ask rise an issue with sunnycase on github and see whether he responds to it.

    Relu is supported in cafe ops from a long time, so they might have not explicitly added it to the list, since they support LeakyRelu in tflite.



  • @tensilestrength
    Thank for the reply. The Github issue says a newer version supports logistic/sigmoid: https://github.com/kendryte/nncase/issues/33, and I didn't see any error/warning durning conversion.
    The link you've sent also missing RELU but I have no problem converting models with RELU activation (also discussed in https://github.com/kendryte/nncase/issues/40).



  • https://github.com/kendryte/nncase/blob/master/docs/tflite_ops.md

    I don't find sigmoid activation being supported in nncase, please correct me if I am wrong.



  • Hi tj,

    Thanks for your reply!
    The input to the sigmoid operation is [None, 240, 320, 1]. I haven't tried yet but suspect manually apply the sigmoid function would greatly slow down inference.

    Here is summary of my model:

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    input_1 (InputLayer)         [(None, 240, 320, 1)]     0
    _________________________________________________________________
    conv2d (Conv2D)              (None, 240, 320, 16)      160
    _________________________________________________________________
    max_pooling2d (MaxPooling2D) (None, 120, 160, 16)      0
    _________________________________________________________________
    conv2d_1 (Conv2D)            (None, 120, 160, 16)      2320
    _________________________________________________________________
    conv2d_2 (Conv2D)            (None, 120, 160, 16)      2320
    _________________________________________________________________
    max_pooling2d_1 (MaxPooling2 (None, 60, 80, 16)        0
    _________________________________________________________________
    conv2d_3 (Conv2D)            (None, 60, 80, 16)        2320
    _________________________________________________________________
    up_sampling2d (UpSampling2D) (None, 120, 160, 16)      0
    _________________________________________________________________
    conv2d_4 (Conv2D)            (None, 120, 160, 16)      2320
    _________________________________________________________________
    conv2d_5 (Conv2D)            (None, 120, 160, 16)      2320
    _________________________________________________________________
    up_sampling2d_1 (UpSampling2 (None, 240, 320, 16)      0
    _________________________________________________________________
    conv2d_6 (Conv2D)            (None, 240, 320, 1)       145
    =================================================================
    Total params: 11,905
    Trainable params: 11,905
    Non-trainable params: 0
    _________________________________________________________________
    

    And code for the neural network part:

    from tensorflow.keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D
    from tensorflow.keras.models import Model
    
    nn_in = Input(shape=(240, 320, 1))  
    
    x = Conv2D(16, (3, 3), activation='relu', padding='same')(nn_in)
    x = MaxPooling2D((2, 2), padding='same')(x)
    x = Conv2D(16, (3, 3), activation='relu', padding='same')(x)
    x = Conv2D(16, (3, 3), activation='relu', padding='same')(x)
    encoded = MaxPooling2D((2, 2), padding='same')(x)
    
    x = Conv2D(16, (3, 3), activation='relu', padding='same')(encoded)
    x = UpSampling2D((2, 2))(x)
    x = Conv2D(16, (3, 3), activation='relu', padding='same')(x)
    x = Conv2D(16, (3, 3), activation='relu', padding='same')(x)
    x = UpSampling2D((2, 2))(x)
    nn_out = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)
    
    model = Model(nn_in, nn_out)
    


  • @jackyhu I'd be curious what the input ('model/conv2d_6/BiasAdd:20') to the sigmoid operation looks like. I've had trouble with nncase and sigmoid in the past and had to manually apply the sigmoid function.