Zero Input Response of NN model

A story about what comes out with zero input and hardcore debugging.

I started to pay attention to the zero input response of Neural Network (NN) when I was developing an acoustic noise suppression model. So sharing the experience here.

Skipping to start of the interesting part, I got the initial version of noise suppression model running on a device, sent in the test signal and listened to the output. Noise being removed, not too bad for first version. However I noticed there was a buzzing noise at the background, even when the input was silence. At that point I thought it must due to parameters had not yet been quantized properly and the deployed device's noise floor. “It will go away when I tidy things up. “

And it didn’t. I retrained my model with consideration of quantization and noise floor, then carefully convert the weights from float to int16, deployed it again and the buzzing was still there, just as annoying as first run. “What the..”

I went back to my model development environment, ran some tests. It turned out even with perfect zero input, the model I trained would still produce those buzzing, not very visible (amplitude is small) but very audible. So nothing to do with noise floor, it is to do with Zero Input Response of my NN model. “Emm, never thought about that. “

Now the question is how the model generate those buzzing by itself. To find out, I did what a hardcore DSP engineer would do, follow the numbers. Before that, I strategically trim the model down. Then I started following the numbers right from the input, operator by operator as they flow through neuron calculation. Luckily it didn't took me long before I found the root cause, it is the cannot be more fundamental perceptron.

The maths is straightforward, output = input x W + b, when the input is zero, output = b. Then this b value would flow through the later layers and more often than not produce a static non-zero output. That was my first hardcore debugging of NN model. “Feels good, make sure do that again. “

Having a tiny static output plays perfectly within the rules, which is our loss function. The static output incurs error all the time, however, as long as the bias can provide more performance gain to offset those error then it’s a good deal. Good, nothing is broken, the model was trained as intended. What missing was the rules didn’t include, in acoustic world, tiny static output could be really annoying. The solution is simple, “use_bias=False“.

Disable bias does lead to small performance drop in my case, however, it’s not anything worth concerning, especially in popular model architecture like this one.

The key is encoder and decoder have clean zero input response, the mask generation, usually where most of the complexity belongs to can still use bias for better performance. “Hope above all helps. ”