jmvalin | RNNoise: Learning Noise Suppression

This demo presents the RNNoise project, showing how deep learning can be applied to noise suppression. The main idea is to combine classic signal processing with deep learning to create a real-time noise suppression algorithm that's small and fast. No expensive GPUs required — it runs easily on a Raspberry Pi. The result is much simpler (easier to tune) and sounds better than traditional noise suppression systems (been there!).

Read More

Threaded | Top-Level Comments Only

From: (Anonymous)

Long time ago I used to make amateur remixes, and one tricky part was to isolate vocals from the remixed track. To do that I was using the noise removal tool: select a part of the track without vocals, run a spectral analysis on it and then substract the result to the whole track. Most of the time the result was terribly mangled, but sometimes I got something usable.

Your demo got me thinking: if I want to remove something very specific from one track instead of learning a generalized filter, can I train this model with a smaller dataset, like a few seconds from that track?

From: (Anonymous)

Hi Jean Marc,

There was a 1.6 million $ Indigogo project about Snoring noise suppression device that went bust.
https://www.indiegogo.com/projects/silent-partner-quiets-snoring-noise-like-magic-sleep/x/14463062#/

Do you think you could help them?

From: (Anonymous)

This is great, thanks for your work! Do you think this might be of use with non-audio data?

From:

jmvalin

What you're describing may be possible, but it would require a different structure for the network, with separate inputs for what you want to remove and what you want to remove it from. That and of course a good training set.

From: (Anonymous)

Hi this looks really interesting. I am working with a team to try to improve optical character recognition on old and "noisy" text images that humans can read but OCR cannot. We have been using neural net to do this with some success on training data (https://github.com/digiah/oldOCR, my github username is rcrath) using OCRopy but I have always thought treating the text as a data stream and using department to get a noise sample of the garbage text that OCR makes when it fails and subtracting that from the signal might reduce the garbage in an OCR file to make other approaches better focused. I realize that is far from what you are doing, butdo you think it would be feasible for us to try and adapt your code?

One thing that strikes my ear in the samples, most obviously in the street noise one, is that the algorithm is acting more like a gate than noise removal since the horns and traffic are clearly audible still in the speech sections.

I would love to see this adapted to guitar noise suppression!

Thanks for this work.

From: (Anonymous)

Lol, plz excuse autocorrect. I have no idea what "department" was supposed to be!

From: (Anonymous)

Do you think this approach would be helpful in reducing noise on old 78 rpm records? Because archive.org's Great 78 project has a large number of albums you could experiment with.

From:

jmvalin

As it is right now, RNNoise would not work for that application. But if you specifically trained it on that task, I think it might.

From:

jmvalin

So I don't think my code would be applicable to OCR. It seems like what you would want instead of to train your OCR with these noisy images. If you don't have enough, you could try and simulate that noise (though it has to be realistic).

About your "noise gate" comment, I understand why you say it sounds like that, but I can assure you it also cancels during active speech. The only reason it cannot easily cancel the horns on active speech is because it cannot use lookahead to tell the horns from speech. It's easy to do when you have a second of audio, but not if all you have is 10 ms.

From:

jmvalin

What you're describing is called "active noise cancellation" and it's very different from cancelling noise in a recording.

From: (Anonymous)

Hi Jean Marc,

Impressive Work! Really nice results too. I've been working on a similar project but to be exclusively used with Ardour so it's an lv2 plugin (https://github.com/lucianodato/noise-repellent). I used ME method (Rangachari and Loizou) to estimate noise with it but this method seems to work miles better. Do you mind if I make an lv2 plugin out of your library?
Also did you evaluate using discrete wavelet transform instead of fft+bark scale? I've read in the past few works that get near zero latency with that architecture.
Thank you very much for this. It already taught me few things wasn't aware of.

From:

jmvalin

You're more than welcome to make an lv2 plugin (or really anything else) out of this. That's why I used a BSD license. I've not looked at wavelets -- for what I'm doing they wouldn't make much difference either way except make things a bit more complicated.

From: (Anonymous)

Hmm, perhaps you can store the RNN's coefficients in a file so one can train the filter to match his needs?

From: (Anonymous)

I'd love to see a ladspa-plugin, so that I could watch Youtube-videos & university-lectures denoised with mpv.
Anyone any ideas how hard it would be to create a ladspa-plugin based on it?

I never done anything like this (but have basic C/C++-knowleged)…

From: (Anonymous)

Hi Jean Marc,
According to the audio samples you provided, it seems that the RNNoise has more residual noise than the speex. Do you think it will perform better with more noise samples for training?

From: (Anonymous)

Can I know how much data do you trained to achieve your demo performance

From: (Anonymous)

Great Work you are doing Jean Marc!
I was wondering if this could be applicable to active noise cancellation as well, with a real time algorithm say from a raspberry pi or maybe a hardware implementation on a fpga. Of course, this would have to be trained on a different training set; I was thinking the Aurora II? Could it be paired with another algorithm such as beamforming to also attenuate noise not from a specific relative location?

Sorry for the plethora of questions.
Thanks.

From:

jmvalin

Active noise cancellation is actually a totally different problem, so unfortunately none of this RNNoise work is applicable.

From: (Anonymous)

Looking denoised output on spectogram feels so good, great work!

But how can we modify to make it real-time spectogram since you purposely delayed it by a few seconds?

Thank you sir.

From: (Anonymous)

Hello.
First of all, thanks for great work.

Recently, I needed VAD on my application and found that RNNoise has VAD output. So I tried use VAD output of RNNoise.
I commented out code not related with VAD, and made prototype of the app. It works quite nicely despite its small calculation size. But on some audio samples, VAD fails.

I'm trying to improve VAD by adding feature size and neural net size.
Can you give me some hint to improve VAD quality?

From: (Anonymous)

Hi JMV,

In Amateur Radio noise suppression is always a big topic. I've added your project into a SDR receiver - if you're not into SDR see www.sdr-radio.com . Anyway it's working well when the audio is above the noise, when the noise and audio are both at the same level I can lose the audio.

This isn't a complaint - just an observation, your code isn't really designed for this situation. Now if only I could get you interested the Amateur Radio your skills would have a big impact on noise reduction.

Thanks for this project, I'll be following it closely.

From: (Anonymous)

BTW - I'm Simon@sdr-radio.com - I should have said that :) .

Simon G4ELI

From:

h3yfinn

Hey there, I'm trying to find a way of detecting any voice in a public(new zealand rural) area. This is so I can scrub private conversations from publicly record files and then analyse the remaining files for birdsong.
We have quite poor quality recordings so I thought your algorithm may be quite useful to remove background noise so we can more easily detect voice (which otherwise we cannot at all detect using a webrtcVAD algorithm). I'm just setting it up now. Thought I'd put this here in case you had any idea whether your algorithm would work like I hope.

Thanks a lot,

From: (Anonymous)

I was wondering myself, if you use a typical FFT\bark.scale Algorhythm on the A.I Enhanced granular wavetable frequency bands, as virtual Bin (size?) calculated predominantly by of course the FFT\barki.scale relying for its final or real.time results, which are limited by or from mainly the CPU/RAM/MEMORY Capacity.
Do you have a Unique (Transform Algorhythm) that uses the "keras" or "theano" CNTK Binary libraries? say something that could be like a hybrid of the FFT & GFT, DTMF? I understand that using a GFT\bark.scale Transfrom Algorhythm alone, could be a massive difference in data\frequency band to using a FFT, as they can potentially use unlimited/Infinite Granular Wavetable band Resolution Virtual Bin data-size/processing power per noise scale or frequency bands!
Understanding there is not large amounts of people, that acknowledge/understand the difference of those two Transform Algorhythms as to begin with.

From: (Anonymous)

The player, with the play line, and the babble/db/noise reduction on/off buttons, is nicely done.
slightly off topic but how did you do that?

From:

jmvalin

Well, the JS and CSS shouldn't be too hard to read. There's a single player and the buttons just change what it plays. That was actually modified from the Opus 1.2 demo page.

From: (Anonymous)

Hi Jean Marc, I am a college student.You great job really impresses me.I want to recurrent your job to learn more.But when I check your open code, your raw data named 'denoise_data9.h5' was lacked .Can you share this data with me? My email address is xiangkai@hust.edu.cn. Thank you very much.

From: (Anonymous)

For such purposes, you need a source separation framework and not noise cancellation. Ping me at praveshb @ iiitd dot ac dot in if you want know more.

Thanks

From: (Anonymous)

Hi, will you disclose the secret spectral non-stationarity metric mentioned on the xiph page?

From:

jmvalin

Hint: look for "spec_variability" in the source code. I'm not quite sure what the history of that metric is and whether/where it's been published, but it's at least 20 years old.

From: (Anonymous)

Hi, my name is Herru, I’ve seen your web demo https://people.xiph.org/~jm/demo/rnnoise/ about RNN for Noise Removal, it’s awesome! After testing the live recording demo, I see that the noise removal is using Javascript. I would like to ask you about the permission to use the Javascript version of it on my project (which is non commercial project), since your link to the original code was made using C and under BSD license.

From:

jmvalin

You may use the JS code under the same BSD license as the C code.

From: (Anonymous)

Thank you very much Jean-Marc.

From: (Anonymous)

Thanks, I found it. Great work on this! Will you help me settle something I've been thinking way too much about: do you pronounce it "RN-Noise," or "RNN-Noise?"

From: (Anonymous)

Dear Jean-Marc,

Many thanks for publishing your exciting work and sharing your code.
I've two points which are not 100% clear to me after reading your documentations and code:

(1) Network training input and output data samples are finite sequences of 42- and 23-element vectors, respectively. But in the operation mode, the trained network is fed sequentially with a single input vector and outputs a single vector?

(2) Is the training data extracted from overlapping spectrogram segments?

Kind regards

From:

jmvalin

1) For training, the RNN needs to be fed a full sequence so that it can backpropagate through time. When we actually use the RNN, we also feed a sequence, but we do it one frame at a time because we want the algorithm to be real-time (for batch denoising, we could feed the sequence all at once)

2) Yes, we use a frame size of 20 ms, with 10 ms overlap.

From: (Anonymous)

Thanks a lot for your quick answer!
Regarding 2), I think I have to specify my question:

Looking at your training code (rnnoise/training/rnn_train.py), you feed the network with sequences of 2000 42-element vectors/frames (= 1 training sample). Now I wonder if two distinct training samples might share a certain number of frames?

From: (Anonymous)

Hi Jean,

For active noise cancellation, can you suggest on the approach? Also, is there any project to your knowledge around that?

From:

jmvalin

No idea, you should look at the scientific literature on the topic. I don't think deep learning is going to be so useful for that. Or at least not in any form that resembles RNNoise.