Dec. 2nd, 2006

jmvalin: (Default)
After a couple days fighting with this annoying overflow bug, I think I've managed to solve the problem. As you can see, some of the fixes are not very nice. It basically comes down to
  • Adding explicit saturation (SATURATE) before 32-bit to 16-bit conversions.
  • Scaling the signal up/down for some operations to avoid having to add saturation all over the place, especially in critical loops
  • PSHR* is evil. Well, not quite but can you shot the danger in the PSHR32 definition?

    • The moral of the story: saturating isn't great, but it still beats overflowing!

Profile

jmvalin: (Default)
jmvalin

March 2023

S M T W T F S
   1234
567891011
12131415161718
1920212223 2425
262728293031 

Most Popular Tags

Page Summary

Style Credit

Expand Cut Tags

No cut tags
Page generated Jun. 20th, 2025 07:57 am
Powered by Dreamwidth Studios