gap_sdk: Wrong output from custom CNN network

Hi,

I have designed a simple 2D convolutional auto-encoder that recreates in output a sine wave for testing the functionalities of the GAP8 before I bring to it my real input that handles EEG signals.

The input of the network is a 2D matrix 16x8 where 16 are the samples from the sine wave and 8 is the number of signals (for simplicity they are 8 equal sine waves, this is in order to simulate my real input of 8 channels EEG). The sine wave is bounded from 0 to 1, and the samples are stored in float64, so I converted my .h5 model into .tflite without quantizing, and let nntool do the full integer 8 bit quantization.

So in the ‘aquant’ command I provided a slice of the sine wave from the numpy array:

np.save('signals.npy', sine_wave[0:100])

and I have also tried to quantized directly in python, but the final result is the same.

All the GAPFlow steps work fine (I used a makefile as suggested for automatizing the process), but when I set some data in the input of the network, the output buffer gives me always wrong results, e.g:

Input buffer:  (16 hard-coded sample of a sine wave equal for each of the 8 channels)
125 125 125 125 125 125 125 125 
126 126 126 126 126 126 126 126 
126 126 126 126 126 126 126 126 
126 126 126 126 126 126 126 126 
126 126 126 126 126 126 126 126 
126 126 126 126 126 126 126 126 
126 126 126 126 126 126 126 126 
126 126 126 126 126 126 126 126 
125 125 125 125 125 125 125 125 
124 124 124 124 124 124 124 124 
123 123 123 123 123 123 123 123 
122 122 122 122 122 122 122 122 
121 121 121 121 121 121 121 121 
119 119 119 119 119 119 119 119 
118 118 118 118 118 118 118 118 
116 116 116 116 116 116 116 116 
Entering main controller
Constructor
Call cluster
Running on cluster
Runner completed


result buffer:  (this should be equal to the input buffer)
21 15 20 21 21 11 9 7 
21 14 9 21 18 3 2 1 
21 13 0 11 3 0 0 0 
21 8 0 0 0 0 0 0 
21 5 0 0 0 5 0 0 
20 19 10 2 15 21 18 7 
21 21 21 21 21 21 21 16 
21 21 21 21 21 21 21 21 
21 21 21 21 21 21 21 21 
21 21 21 21 21 21 21 21 
21 21 21 21 21 21 21 21 
21 21 21 21 21 21 21 21 
21 21 21 21 21 21 21 21 
21 17 21 21 21 21 21 21 
21 18 21 21 21 21 21 21 
21 21 21 21 21 21 21 21

Even if I change the input buffer with other values, it gives me more or less this same wrong output.

My model in python infers correctly every sample of the given sine waves, but honestly I don’t know why here is so wrong. The input and output of the generated ATmodel are signed char:

int q8CNN(
	signed char * __restrict__ Input_1,
	signed char * __restrict__ Output_1)

and so I have to convert manually my float signals into int8, for this purpose I am using the following functions:

int8_t int8fromfloat(float x)
{
    return (int)(x * 127.0f);
}

float int8tofloat(int8_t x)
{
    return (float)x * (1.0f / 127.0f);
}

On this, I was also wondering if it is possible to generate somehow an ATModel that takes in input directly float32, and let the network do the quantization? Because I tried to give into nntool a .tflite model with inpu float32 and a quantize layer, but when it generates the API, the input of the model is always signed char*.

I upload here my project directory with all the files I use sine_wave.zip

I am on a Ubuntu 20.04 machine with GAPUINO_V2, running the 3.8.0_dev branch (because here it’s fixed a previous issue I had importing the model)

Please let me know if you can spot any issue.

Thank you very much for your time.

Best regards,

Manuel

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 18 (9 by maintainers)

Most upvoted comments

EDIT: Also, how does one change whether the network expects signed char or signed short as input? My generated kernels specify signed char as input to the first convolution.
void S1_Conv2d_6x1x3x3(
		signed char * __restrict__ In,
                ...) {
...
}

I think he was referring to the quantization scheme, if you quantize with 8bit the generated code will have signed char, as in your case, instead if you quantized with 16bit, the generated code will have signed short as input.

If you run qshow in nntool you will see, for each layer, the quantization strategy. In this case you will need the information about the first layer.

| 0 | input_1 | SQ8 | | -0.85<(i8-0.00)*0.00663208<0.84

In this example for instance the quantized deployed network expects the inputs in int8 representation and the [-127:128] range will be interpreted as [-0.85:0.84]. Hence -127=-0.84, …

Thank you for this suggestion!

Best,

Manuel

Manucar on Feb 25, 2021

Ciao Manuel!

You should always check the nntool execution on your input samples to see the result. Try with dump command with float and quant (-q -d flags) execution on your input sample. Remember that in nntool the inputs must always be in their floating point representation.

I tried your negative values example and also the floating point execution is returning me 0s. Maybe it is a training issue? Try the same input sample in keras/tflite and see if it differs.
On the different range I suggest you to always calibrate the quantization with samples that exploit the whole range of values. If not the network will clamp some of your layers to wrong min/max values resulting in large errors.

Best regards, Marco

marco-fariselli on Feb 17, 2021

It shouldn’t be necessary

marco-fariselli on Feb 10, 2021

Hi Manuel !! The sigmoid support is at the moment on hold due to other, more prioritized, tasks on going. I suggest however to use the latest 3.8.3_dev branch of the gap_sdk where we fixed some bugs regarding HARD sigmoid support. It should work much better now. To use it you just need to provide a tflite graph with a standard sigmoid, during quantization the HARD version will replace it automatically.

Cheers, Marco

marco-fariselli on Feb 10, 2021

Hi Manuel! Sorry for the late reply. I took a look at your code (first attached files, sine_wave.zip) and indeed there was a bug in our flow regarding non RELU activations. The HSigmoid does not show too much drop in accuracy now wrt to the standard Sigmoid. Let me know if it’s fine for you. I pushed the fixes in a development branch (https://github.com/GreenWaves-Technologies/gap_sdk/tree/3.8.0_dev). The official release will come soon.

The problem with your latest files is that there is no support now for the Minimum operator. It will be released soon. However it seems that your conversion process ended with a strange minimum + Relu instead of an HSigmoid layer. Can you share the keras code which produces the .h5 graph?

Best regards, Marco

marco-fariselli on Dec 28, 2020

Hi @Manucar !! When you give nntool any input it should be in its real value. For instance if you trained a network with [-1:1] input images you should take the original images (typically [0:255] values) and apply a normalization function to convert them to the real value, i.e. “x:x/128-1”. In your case you just need to fed the network with your npy files. Beware here because I see you have a single file with 100 samples in it, it won’t work, you should convert it into 100 files. However when you eventually deploy your network then the inputs should be in the quantized values, the network will expect signed char or signed short as inputs.

For the Sigmoid issue you could do several things:

remove it from the graph and compute that calculation after the network finished in your main.c.
retrain the network with a HARDSigmoid instead of a sigmoid, shouldn’t effect much the overall accuracy
wait for support of a Sigmoid in GapFlow

marco-fariselli on Dec 22, 2020

Hi Manuel !! Sorry for the late reply! I looked at your code and indeed there were some problems related to this network which in part I solved and will be released soon. These were the main issues:

Our resizers could not handle multi channel inputs and signed char input/output types (now it’s done and will be released). From my side the execution of the network in nntool quantized has the same results as the run on chip;
The network has at the end a sigmoid which is by default converted by the GapFlow to a HARDSigmoid when you quantize the network. It seems to degradate a lot the accuracy. Up to the layer before the sigmoid the quantization seems to not affect much numerically.

marco-fariselli on Dec 22, 2020