CS245:            Introduction to Interactive Sound Synthesis

     

Prerequisites:  [Trig, Physics, C/C++ Programming, Data Structures, Operating Systems]      

Textbook:       Audio Anecdotes – Tools and techniques for Digital Audio

            Ken Greenebaum, Ronen Barzel (ISBN: 1-56881-104-7)

Description:    Sound synthesis offers an exciting alternative to the use of pre-recorded sounds in simulation and video game software. Synthesis techniques allow sound to directly correlate to the simulated interactions which generate the sound. For instance collision sounds may be dynamically synthesized using parameters from the simulation or physics engine such as the material types involved, the speed of impact, and the angle of impact. As a result every collision is unique and conveys much more information than simply re-playing a pre-recorded engineered sound.  The entire sonic chain may be modeled including the sound source, the transmission environment and the listener. The results are directional sounds which convey a great deal of information to the user and correspondingly appear very rich.

This course introduces the fundamentals needed to understand, create and manipulate digital sound: the physics of sound, the human perception of sound, and digital sampling theory. The class surveys significant algorithms and techniques used in the field emphasizing tradeoffs in accuracy and performance. Assignments provide a hands-on opportunity to experiment with implementing and using many of these algorithms and implementation techniques. The instructor, Ken Greenebaum, provides insights from his 15 years creating digital media systems for companies including Silicon Graphics and Microsoft. Demos, videos, and guest speakers will be used to enhance the lecture.

(A follow-on course may be offered providing a deeper exploration of advanced algorithms)

 

Class requirements:

(1)   Mid-term                                                                                                          -25%

(2)   Final                                                                                                                -25%

(3)   Project(s)                                                                                                         -50%

Project: Construct dynamic audio synthesis engine. To keep this project manageable it will be broken into 10 pieces assigned synchronously to the topics introduced in lecture.

(1)       Audio “Hello World” using either DirectSound or PABLIO libraries

(2)       Parametric Sine wave synthesizer

(3)       Mixer with de-zippering

(4)       Rate and quantization conversion to engine

(5)       Spectral filtering

(6)       FM synthesis

(7)       Additive synthesis

(8)       Subtractive synthesis

(9)       Physical modeling

(10)   3D Spatialization

Grading policy:

(1)       Correctness                                                                                                     – 70%

(2)       Design and representation                                                                               – 20%

(3)       Programming style                                                                                           – 10%

Programming requirements:

(1)       C/C++ submitted with handwritten Makefiles (MUST BUILD)

(2)       DirectSound or PABLIO audio libraries

(3)       VisualStudio/Windows or Linux required

Office Hours:

By appointment before or after class

Course webpage: http://cs245.sworks.com/

Contact info:

Email:        keng@sworks.com

Phone:       206.310.5253

 

Contents:

            Week 1

·         Introduction

·         Physics of sound (Compression rarefaction of a medium, collision, cricket (comb), speaker cone)

·         Attenuation ((spherical dissipation), point, dipole, planar radiators)

·         Propagation of sound waves (Speed, Reinforcement/cancellation, Reflection, Diffraction, Doppler Effect)

·         Measurement (The Decibel, Sound Pressure Level)

·         EZ- synthesis (Square-waves, saw-tooth)

·         Assignment: Audio “Hello World” using DirectSound or PABLIO audio libraries

            Week 2

·         Human perception of sound (Range of Hearing, Fletcher/Munson equal loudness curves, latency perception, Spatial perception (inter-aural time difference, inter-aural amplitude difference, cosine model), Theory of Signal Detection (Receiver Operator Curves))

·         Sampling theory (Nyquist limit, Quantization, Sampling rate, Reconstruction filter)

·         Sine-wave synthesis (sin(x), Lookup table, Oscillators)

·         Assignment: Parametric Sine wave synthesis

            Week 3

·         Wave table synthesis (Mixing, Enveloping, De-zippering techniques)

·         Computer techniques (Realtime programming, Streaming using threads, Buffering: circular ‘ring-buffer’ queue)

·         Assignment: mixer with de-zippering

            Week 4

·         Dynamic rate conversion (Why rate convert?, Challenges?, Optimal solutions, Practical solutions)

·         Compression concepts (Encodings: (non-linear, mu-law, delta), Lossless: entropy coding)

·         Assignment: add rate conversion to mixer to playback simultaneous sounds of different quantization and sample rate

            Week 5

·         Compression concepts cont. (Lossy compression (Auditory masking, MP3, WMA))

·         Spectral filtering (FIR, IIR filters, FFT)

·         Assignment: add parametric filtering to sound engine

            Week 6

·         Sound and interaction

·         Causality

·         Synthesis vs. playback of pre-recorded sounds (1980’s standup arcade games (tentative guest speaker: David Thiel Gotlieb programmer, Microsoft Researcher and the creator of Q*Bert’s voice))

·         Dynamic parametric synthesis

·         Synthetic Foley sound

·         Timbre trees

·         DirectAnimation

·         Assignment: No assignment

            Week 7

·         Midterm

·         Synthesis techniques overview

·         No Assignment

            Week 8

·         FM Synthesis (Explosions, Musical sounds, Exploring the FM audio space)

·         Computer techniques:Pipelining (Accurate timing, Synchronization, Doppler via accumulation buffer)

·         Assignment: add FM synthesizes to sound engine (mixer), find 3 interesting sounds in the sound space

            Week 9

·         Additive Synthesis

·         What makes wood, metal, hollow, solid, etc. sound that way?

·         Resonance/Damping

·         Fundamental frequencies/Harmonics

·         Assignment: Add parametric Additive synthesis to sound engine

            Week 10

·         Subtractive Synthesis

·         What makes wind, explosions sound that way?

·         Noise generators (white, pink, 1/f noise)

·         Filtering

·         Assignment: Add parametric Subtractive synthesis to mixer, create 2 interesting sounds

            Week 11

·         Physical Modeling

·         What makes actual objects have their characteristic sound?

·         The spring mass model

·         Modeling bars/bells

·         Modeling strings

·         Modeling complex systems: Finite element Analysis

·         Simplification: modal synthesis

·         Assignment: (2-week) pick physical system and excitation model to add to sound engine

            Week 12

·         Exciting physical models (Plucking, Striking, Scraping (Friction))

·         Assignment: (part-2)

            Week 13

·         Spatialization (Surround sound, Panning vs. fading, Head Related Transform)

·         Spatialization issues (Speaker array, headphones, crosstalk cancellation, moving sources)

·         Assignment: add parametric spatializer to sound engine

            Week 14

·         Granular synthesis (sounds of nature, rivers, the wind)

·         Ambient synthesis (city streets, baseball game)

·         Review

·         Assignment: No assignment (semester assignment due + Final)