CELT part 3: Pitch prediction
Dec. 24th, 2007 08:40 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
jmvalinBefore reading this, I recommend reading part 1 and part 2. As I explained in part 1, CELT achieves really low latency by using very short MDCT windows. In the current setup, we have two 256-sample overlapping (input) MDCT windows per frame. The reason for not using a single 512-sample MDCT instead is latency (the look-ahead of the MDCT is shorter). With that setup, we get 256 output samples per frame to encode (128 per MDCT window). Now, at 44.1 kHz, it means a resolution of 172 Hz, not to mention the leakage. That's far from enough to separate female pitch harmonics, much less male ones. To the MDCT, a periodic voice signal thus looks pretty much like noise, with no clear structure that can be used to our advantage.
To work around the poor MDCT resolution, we introduce a pitch predictor. Instead of trying to extract the structure from a single (small) frame, the pitch predictor looks outside the current frame (in the past of course) for similar patterns. Pitch prediction itself is not new. Most speech codecs (and all CELP codecs, including Speex) use a pitch predictor. It usually works in the excitation domain, where we find a time offset in the past (we use the decoded signal because the original isn't available to the decoder) that looks similar to the current frame. The time offset (pitch period) is encoded, along with a gain (the prediction gain). When the signal is highly periodic (as is often the case with voice), the gain is close to 1 and the error after the prediction is small.
Unlike CELP, CELT doesn't operate in the time domain, so doing pitch prediction is a bit trickier. What we need to do is find the offset in the time domain, and then apply the MDCTs (remember we have two MDCT windows per frame) and do the rest in the frequency domain. Another complication is the fact that periodicity is generally only present at lower frequencies. For speech, the pitch harmonics tend to go down (compared to the noisy part) after about 3 kHz, with very little present past 8 kHz. Most CELP codecs only have a single gain that is applied throughout the entire frame (across all frequencies). While Speex has a 3-tap predictor that allows a small amount of control on the amount of gain as a function of frequency, it's still very basic. Working in the frequency domain on the other hand, allows a great deal of flexibility. What we do is apply the pitch prediction only up to a certain frequency (e.g. 6 kHz) and divide the rest in several (e.g. 5) bands. For the example from part 2 (corresponding to mode1 of the 0.0.1 release), we use the following bands for the pitch (different from the bands on which we normalise energy):
{0, 4, 8, 12, 20, 36}
Another particulatity of the pitch predictor in CELT (unlike any other algorithm I know of) is that the pitch prediction is computed on the normalised bands. That is we apply the energy normalisation on both the current signal (X) and the delayed (pitch prediction from the past) signal (P). Because of that, the pitch gain can never exceed unity, which is a nice property when it comes to making things stable despite transmission losses. Despite a maximum value of one in the normalised domain, the "effective value" (not normalised) can be greater than one when the energy is increasing, which is the desired effect. The pitch gain for band i is computed simply g_i = <X_i, P_i>, where <,> is the inner product and X_i is the sub-vector of X that corresponds to band i (same for P_i).
Here's what the distribution of the gains look like for each band:
It's clear from the figure above that the lower bands (lower frequencies) tend to have a much higher pitch value. Because of that, a single gain for all the bands wouldn't work very well. Once the gains are computed, they need to be encoded efficiently. Again, using naive scalar quantisation and encoding each gain separately (using 3 or 4 bits each) would be a bit wasteful. So far, I've been using a trained (non-algebraic) vector quantiser (VQ) with 32 entries, which means a total of 5 bits for all gains. The advantage of VQ for that kind of data is that it eliminates all redundancy so it tends to be more efficient. The are a few disadvantages as well. Trained VQ codebooks are not as flexible and can end up taking too much space when there are many entries (I don't think 32 entries is enough for 5 gains).
The last point to address about the pitch predictor is calculating the pitch period. We could try all delays, apply the MDCTs and compute the gains for each and at the end decide which is beat. Unfortunately, the computational cost would be huge. Instead, it's easier to do it in "open loop" just like in Speex (and many other CELP codecs). We compute the generalised cross-correlation (GCC) in the frequency domain (cheaper than computing in the time domain). The cross-spectrum (before computing the IFFT) is weighted by an approximation of the psychoacoustic masking curve just so each band contributes to the result (instead of having the lower frequencies dominate everything else).
Now the results: how much benefit does pitch prediction give? Quite a bit actually, hear for yourself. Here's the same speech sample encoded with or without pitch prediction. Even on music, which is not always periodic, pitch prediction can a bit, though not as much. I think there's potential to do better on music. There's a few leads I'd like to investigate (and again, I'm open to ideas):
To work around the poor MDCT resolution, we introduce a pitch predictor. Instead of trying to extract the structure from a single (small) frame, the pitch predictor looks outside the current frame (in the past of course) for similar patterns. Pitch prediction itself is not new. Most speech codecs (and all CELP codecs, including Speex) use a pitch predictor. It usually works in the excitation domain, where we find a time offset in the past (we use the decoded signal because the original isn't available to the decoder) that looks similar to the current frame. The time offset (pitch period) is encoded, along with a gain (the prediction gain). When the signal is highly periodic (as is often the case with voice), the gain is close to 1 and the error after the prediction is small.
Unlike CELP, CELT doesn't operate in the time domain, so doing pitch prediction is a bit trickier. What we need to do is find the offset in the time domain, and then apply the MDCTs (remember we have two MDCT windows per frame) and do the rest in the frequency domain. Another complication is the fact that periodicity is generally only present at lower frequencies. For speech, the pitch harmonics tend to go down (compared to the noisy part) after about 3 kHz, with very little present past 8 kHz. Most CELP codecs only have a single gain that is applied throughout the entire frame (across all frequencies). While Speex has a 3-tap predictor that allows a small amount of control on the amount of gain as a function of frequency, it's still very basic. Working in the frequency domain on the other hand, allows a great deal of flexibility. What we do is apply the pitch prediction only up to a certain frequency (e.g. 6 kHz) and divide the rest in several (e.g. 5) bands. For the example from part 2 (corresponding to mode1 of the 0.0.1 release), we use the following bands for the pitch (different from the bands on which we normalise energy):
{0, 4, 8, 12, 20, 36}
Another particulatity of the pitch predictor in CELT (unlike any other algorithm I know of) is that the pitch prediction is computed on the normalised bands. That is we apply the energy normalisation on both the current signal (X) and the delayed (pitch prediction from the past) signal (P). Because of that, the pitch gain can never exceed unity, which is a nice property when it comes to making things stable despite transmission losses. Despite a maximum value of one in the normalised domain, the "effective value" (not normalised) can be greater than one when the energy is increasing, which is the desired effect. The pitch gain for band i is computed simply g_i = <X_i, P_i>, where <,> is the inner product and X_i is the sub-vector of X that corresponds to band i (same for P_i).
Here's what the distribution of the gains look like for each band:
It's clear from the figure above that the lower bands (lower frequencies) tend to have a much higher pitch value. Because of that, a single gain for all the bands wouldn't work very well. Once the gains are computed, they need to be encoded efficiently. Again, using naive scalar quantisation and encoding each gain separately (using 3 or 4 bits each) would be a bit wasteful. So far, I've been using a trained (non-algebraic) vector quantiser (VQ) with 32 entries, which means a total of 5 bits for all gains. The advantage of VQ for that kind of data is that it eliminates all redundancy so it tends to be more efficient. The are a few disadvantages as well. Trained VQ codebooks are not as flexible and can end up taking too much space when there are many entries (I don't think 32 entries is enough for 5 gains).
The last point to address about the pitch predictor is calculating the pitch period. We could try all delays, apply the MDCTs and compute the gains for each and at the end decide which is beat. Unfortunately, the computational cost would be huge. Instead, it's easier to do it in "open loop" just like in Speex (and many other CELP codecs). We compute the generalised cross-correlation (GCC) in the frequency domain (cheaper than computing in the time domain). The cross-spectrum (before computing the IFFT) is weighted by an approximation of the psychoacoustic masking curve just so each band contributes to the result (instead of having the lower frequencies dominate everything else).
Now the results: how much benefit does pitch prediction give? Quite a bit actually, hear for yourself. Here's the same speech sample encoded with or without pitch prediction. Even on music, which is not always periodic, pitch prediction can a bit, though not as much. I think there's potential to do better on music. There's a few leads I'd like to investigate (and again, I'm open to ideas):
- Using two pitch periods
- Frequency-domain prediction
New site
Date: 2017-02-09 07:59 am (UTC)http://hotties.pictures.erolove.in/?post.alison
loli nude free shemale galleries movies red head burping girl video redness of eyes and sore throat jordan capri blow job video
My supplementary website
Date: 2017-02-10 12:25 am (UTC)http://blondesex.twiclub.in/?register.cristina
free pornography videps erotic boundaries lesbian erotic books erotic theater erotic hotels
вышел новый хрумер
Date: 2017-02-11 06:37 am (UTC)new XRumer (https://vk.com/pages?oid=-139678140&p=программа_для_постинга_в_блоги)
Callow Job
Date: 2017-02-14 08:12 am (UTC)mobile rpg games android best dating app sexi girl wallpapers onedrive app download download play app
http://pornapps.xblog.in/?gain.jazmine
android gratis para descargar free online puzzle games all phones of htc sexy lingerie online apps android gratis tablet
Day after day gay photos service
Date: 2017-02-14 05:43 pm (UTC)http://gay-pics.porngalleries.top/?post-august
Delivered grown-up galleries
Date: 2017-02-15 02:21 pm (UTC)http://dateme.sexblog.pw/?entry.kaylie
disabled sex contacts free milf dating service totally free sex dates married dating sites free знакомства для бизнеса
Callow Job
Date: 2017-02-17 06:32 am (UTC)http://arab.sex.photo.xblog.in/?post-aniyah
planet numbar missions beauty freesexcams
Communal pictures
Date: 2017-02-18 01:25 am (UTC)http://love.dating.sexblog.pw/?post.arlene
african women dating site free sex social networking sites singapore expats dating & friends interracial bbw dating matchmaker dating
Social pictures
Date: 2017-02-18 05:51 pm (UTC)http://jessica.casual.blogporn.in/?diagram.deasia
mamba dating site online hookup site swingers groups free sex social networking couples for couples
Sexual pictures
Date: 2017-02-19 02:30 am (UTC)http://dateme.sexblog.pw/?post.anabel
free shemale meet adult adverts to find a date free online dating no credit card married men single women
Pictures from venereal networks
Date: 2017-02-19 09:58 pm (UTC)http://date.top.yopoint.in/?gain.juliette
best dating site in ireland uk sex contact matchmaking uk top dating sites reviews best interracial dating sites 2015
New Poke out
Date: 2017-02-20 05:57 am (UTC)http://blondesex.twiclub.in/?gain.alison
sex sites bondage erotic body download erotic games video xxx prono
Latest install
Date: 2017-02-20 09:43 am (UTC)http://travel.tobuy.in/?gain.arianna
online erotic books erotic ebooks free erotic blu ray sexy movies erotic kids
Latest spot
Date: 2017-02-22 07:45 am (UTC)http://arab.egypt.adultnet.in/?entry.maggie
sri dundee scholars canli bellydancr
My brand-new website
Date: 2017-02-22 08:01 am (UTC)http://pattaya.girls.blogporn.in/?post.aracely
promise au graduate contraception foreskin
Mature galleries
Date: 2017-02-23 09:28 am (UTC)http://sissyblog.twiclub.in/?dispatch.monserrat
erotic dance free adult movies free exotic stories free xxx male erotic photography
Free galleries
Date: 2017-02-23 01:12 pm (UTC)http://sunni.muslim.purplesphere.in/?entry-aileen
hindus grannies stopped turismo ap
http://angeladorer.com/
Date: 2017-02-23 02:55 pm (UTC)[url=http://angeladorer.com/]бесплатное порно видео молоденьких[/url]
[url=http://angeladorer.com/]торрент бесплатно порно молоденькие[/url]
новые косметические средства
Date: 2017-02-24 12:56 pm (UTC)[url=http://eyes-coverr.blogspot.com/]Eyes Cover Маска Для Глаз [/url]
http://gastrenit.blogspot.ru/
[url=http://eyescover2017.blogspot.com/] Маска Для Глаз [/url]
[url=http://2017firefit.blogspot.com/] Капли для похудения Fire Fit [/url]
[url=http://firefitnowa2017.blogspot.com/] Fire Fit [/url]
[url=http://www.firecover.ru/kostochka-na-noge-prichiny/]Косточка на ноге причины[/url]
[url=http://www.firecover.ru/kostochka-na-noge-iz-za-chego/]Косточка на ноге из за чего[/url]
[url=http://www.firecover.ru/chto-delat-esli-bolit-kostochka-na-noge/]Что делать если болит косточка на ноге возле большого пальца лечение[/url]
[url=http://www.firecover.ru/kostochka-na-noge-gde-lechit/]Косточка на ноге где лечить[/url]
[url=http://www.firecover.ru/kostochka-na-noge-nachalnaya-stadiya-foto/]Косточка на ноге начальная стадия фото[/url]
Communal pictures
Date: 2017-02-24 04:07 pm (UTC)http://brazil.porndairy.in/?pic-rachael
panty video xxx poop samantha kay boobs free aflam arabiya sex bengali sex action
Matured placement
Date: 2017-02-25 12:13 pm (UTC)http://teen.polish.yopoint.in/?entry.jacquelyn
weaird cartoon porn bittorrent porn 2009 jelsoft enterprises ltd black asian hard core porn how porn stars do anal porn addiction gospel
Pictures from community networks
Date: 2017-02-25 09:04 pm (UTC)http://ashlynn.pornpost.in/?stage.daniella
free uk adult dating sites thai dating i danmark free local adult chat rooms adult chat for mobile find gay men free
Social pictures
Date: 2017-02-27 10:04 am (UTC)http://vanessa.blog.porndairy.in/?epoch.fatima
hiv dating sites uk seks dating dating sites for asian women most popular dating apps woman russian
Возможность зарабатывать в свободное время
Date: 2017-02-28 01:06 am (UTC)Возможность для людей с международными контактами. Собственный бизнес. (http://oksanamelen.blogspot.ru/p/blog-page_5.html)
Matured site
Date: 2017-02-28 02:36 pm (UTC)out dating site trendy bathing suits beauty salons leicester
http://feminisation.xblog.in/?post.ann
cheap sleepwear rules for husbands women adult sex pan ties hotels & cleaning lady photos buy white widow seeds girl fuc k