Adding Noise for Robust Deep Neural Network Models

Deep Learning neural networks can work well for several tasks today. We can use deep neural networks for image recognition, object detection, segmentation, speech, NLP and much more. But there is one problem with deep neural networks. They are bad at handling noise during real-world testing.

The generalization power of deep neural networks reduces drastically when they encounter noisy data. But there is a way to reduce such poor generalization ability which we will learn in this article.

What Will You Learn in This Article?

• Why noise is a problem for neural networks?
• How adding noise to input data can help neural networks?
• How adding noise affects different types of deep neural network and other types of machine learning models?
• Types of Noise that we can add to the input data.

Why Noise is Problem for Neural Networks?

In this section, we will discuss why noise in the data is a problem for neural networks and many other machine learning algorithms in general?

While training neural networks, we try to get the best accuracy while training. But most of the time what matters is the generalization ability of the neural network model. The question here is, how well does the model perform on a real-world dataset?

Most of the time, it performs well. But suppose that you have trained a huge image classification neural network model which gives state of the art training accuracy. But while testing on real data which it has not seen before, it performs poorly. One of the main reasons here can be that the real-world images are not as clean as train images.

While training, we may preprocess, resize, and give the inputs to the model for training. But most of the time, we do not consider the presence of noise in the data. And this is what throws the generalization power of a neural network off-track.

In their paper, Deep networks for robust visual recognition, Yichuan Tang, and Chris Eliasmith said the following about Deep Belief Networks.

Deep Belief Networks (DBNs) are hierarchical generative models which have been used successfully to model high dimensional visual data. However, they are not robust to com-mon variations such as occlusion and random noise.

Deep networks for robust visual recognition, Yichuan Tang, Chris Eliasmith

This is mostly the case because the neural network model has not been trained on any type of noisy data. So, one of the solutions is to train the neural network by adding some type of random noise to the input data. In most of the cases, this is bound to increase the robustness to noise and generalization power of the neural network.

In the rest of the article, we will discuss, how adding noise can help neural network models and also see the results of some of the research papers which have tried to achieve similar results.

How Adding Noise to Inputs Help Neural Networks?

Neural networks are both really powerful, and also weak at the same time. We achieve the best results when using deep neural networks by using a supervised learning technique.

Suppose that you built a really good image recognition model with state of the art training accuracy. Chances are that the neural network model performs really well even on the validation set. But there are some cases in the real world where the neural network will struggle to generalize well.

Those real-world images may be blurry, or have low resolution, or may contain some sort of noise. Irrespective of the case, the neural network is bound to suffer to some extent. The best solution to this is to train the model on original input images, as well as images containing noise.

Quoting Ian Goodfellow from the Deep Learning book,

One way to improve the robustness of neural networks is simply to train them with random noise applied to their inputs.

Regularization, page 237

So, basically, we can add random some of the input data which can help the neural network to generalize better. And of course, it will make the neural network more robust against adverse images in real-world testing.

Some Other Benefits of Adding Random Noise to the Data

Generalization is one of the major benefits of training a neural network model with noise. However, there are other benefits as well. The following discussion mainly aims to lay out a brief description of those benefits.

Adding Noise can Reduce Overfitting

Random noise can lead to less overfitting of a neural network while training. It is also consistent with the fact that it can lead to better generalization. So, less overfitting leads to better validation and test scenario, which in turn leads to better generalization during real-world data testing.

Can Help to Deal with Small Train Dataset Size

If you do not have a sufficient amount of data to train a neural network, then adding noise to inputs can surely help. This will lead to the network to see more diverse data while training.

But still, there is a problem. If the dataset size is too small, and you add random noise to half of the inputs. Then the model will not get to train on a sufficient amount of data for some of the classes. This will become even more problematic when we have an imbalanced dataset.

Adding Noise can Act as a Data Augmentation Technique

Suppose that you have a very small dataset. Then you can prepare another dataset by adding noise to the whole of the original dataset. This can also be used as a data augmentation technique while generating more data. Such type of data augmentation can also help in overcoming the previous problem of training on less data for a specific class.

Injecting noise in the input to a neural network (Sietsma and Dow, 1991) can also be seen as a form of data augmentation.

Deep Learning, page 237

Adding Noise to Input Data

Data augmentation has been proved to be a very useful technique to reduce overfitting and increase the generalization performance of neural networks.

Adding Noise to Image Data

However, most of the data augmentation in images happen as rotating the image, shifting the pixels, or maybe adding some type of whitening to it. Now we can also try adding noise as a type of data augmentation technique.

This is because in the real world the images may vary a lot from what the neural network has been trained on.

In real-world applications image quality may vary drastically depending on factors such as the capture sensor used and lighting conditions. These scenarios need to be taken into consideration when performing image classification, since quality shift directly infuence its results.

Deep Convolutional Neural Networks and Noisy Images, Tiago S. Nazare, Gabriel B. Paranhos da Costa, Welinton A. Contato, and Moacir Ponti (2018)

How Adding Different Types of Noise to Input Data Affects the Classification Results?

The authors in the paper Deep Convolutional Neural Networks and Noisy Images tried adding different types of noise to the input data and then train different deep neural network models. They found that adding noise to the input data and then training a neural network model on that data is beneficial when dealing with varying images.

Specifically, quoting their conclusion,

The results indicate that training networks using data affected by some types of noise could be beneficial for applications that need to deal with images with varying quality, given that it seems to improve the resilience of the network to other types of noise and noise levels.

Deep Convolutional Neural Networks and Noisy Images, Tiago S. Nazare, Gabriel B. Paranhos da Costa, Welinton A. Contato, and Moacir Ponti (2018)

The authors in the above experiment used the Digit MNIST, the CIFAR10, and SVHN dataset. This makes it easy to collect the data if someone wants to replicate the results. If you want to get a detailed view of what the authors tried and accomplished, then do give the paper a thorough read.

The following image shows the results obtained by the authors. It shows the accuracy when training and testing were conducted on the same dataset version.

The authors also found another interesting fact. When denoising was applied to the noisy images and neural networks were trained on them, then the accuracy decreased. As pointed out, this may be due to the blurry denoised images which remove relevant information from the data. The following images show the accuracy with and without applying the denoising algorithms.

How Feature Extraction Helps in Dealing with Noisy Data?

Another study was done by authors Gabriel B. Paranhos da Costa, Wellington A. Contato, Tiago S. Nazare, Jo ̃ao E. S. Batista Neto, Moacir Ponti. In their paper An empirical study on the effects of different types of noise in image classification tasks, the authors tried adding three different types of noise to the input data.

However, their main aim was to see how different machine learning models performed after feature extraction was done on noisy images. So, they wanted to see if the noising of images helped in achieving better classification results rather than using the noisy images directly. When dealing with noisy images they found that,

… none of the classifiers were able to overcome the performance of the classifier trained and tested with the original dataset.

An empirical study on the effects of different types of noise in image classification tasks, abriel B. Paranhos da Costa, Wellington A. Contato, Tiago S. Nazare, Jo ̃ao E. S. Batista Neto, Moacir Ponti

But there is a very interesting point to note in their experiments. They did not use any deep neural network models for their experimentations. Instead, they used different linear Support Vector Machines (SVMs) for different types of noisy and noise-free data.

In their conclusion, the authors concluded the following after feature extraction was applied to denoise the images.

When denoising is applied, the results obtained by classifying images from the same category (same type of noise or denoising method) were slightly better then the ones achieved by classifying noisy images. However, due to the smoothing caused by these methods, these results did not match the classification performance of the original dataset.

An empirical study on the effects of different types of noise in image classification tasks, abriel B. Paranhos da Costa, Wellington A. Contato, Tiago S. Nazare, Jo ̃ao E. S. Batista Neto, Moacir Ponti

The paper itself contains explanations in detail along with graphs and plots of the results. If you enjoy reading machine learning papers, then you can read it here to gain even more knowledge.

Different Types of Noise

In this section, we will see what are the different types of noise that we can add to the input images before we train the neural network on them. In the papers that we discussed above, the authors used different types of noise.

Some of the most common types of noise that we can add to the images are:

• Gaussian noise.
• Salt and pepper noise.
• Poisson noise.

Much noise-related research involving images is carried out by applying any of the three noise to the image data. Many times you may also find that a number of experiments are carried out by adding all the above noise to the whole dataset but in separate. So, sometimes the experiments can deal with 2 and even 3 datasets to get the proper results.

Let’s take a look at some images and analyze how they look after applying noise.

The following image shows Gaussian noise added to the Digit MNIST dataset.

The above image shows how the digits of the dataset will look when Gaussian noise is added to them.

Now, let’s take a look at the effect of adding Salt and Pepper noise to the digits of the MNIST dataset.

The noise in the case of salt and pepper noise is much more prominent. And the noise amount, in this case, is 0.5

Gain More Knowledge

You can follow the Deep Learning book by Ian Goodfellow and Yoshua Bengio and Aaron Courville. In specific, you can read the Regularization for Deep Learning chapter. This chapter puts forth many regularization techniques for deep neural networks and adding noise to the input data is one of them.

If you want to read some research papers, then the following are some good ones:

• An empirical study on the effects of different types of noise in image classification tasks, Gabriel B. Paranhos da Costa, Welinton A. Contato, Tiago S. Nazare, Jo ̃ao E. S. Batista Neto, Moacir Ponti.
• Deep networks for robust visual recognition, Yichuan Tang, Chris Eliasmith.
• Deep Convolutional Neural Networks and Noisy Images, Tiago S. Nazare, Gabriel B. Paranhos da Costa, Welinton A. Contato, and Moacir Ponti (2018).

Summary and Conclusion

I hope that you were able to gain some better insights into the regularization of deep learning neural networks. This article discusses the effect of adding noise to the input data and then training the deep neural network on the noisy data. If you have any questions, you can leave your thoughts in the comment section and I will try my best to address them.

You can contact me using the Contact section. You can also find me on LinkedIn, and Twitter.