Ssynth: a Real Time Additive Synthesizer With Flexible Control V. Verfaille, J. Boissinot, M. M. Wanderley & Ph. Depalle Sound Processing and Control Laboratory Schulich School of Music – McGill University
May 25-27 2006
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Context of the Ssynth project
Context of the Ssynth project What? Ssynth: real time additive synthesizer synthesis from the control viewpoint (design & implementation) application: conception of new musical instruments further development of Escher [Wanderley, Schnell &Rovan, 1998] How? implements advanced and flexible control functionalities provides interpolation and extrapolation of musical playing of digital instruments generates high quality sounds (instrumental sound database), with a coherent control May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Structure
Structure
Modularity in design of digital musical instruments [Wanderley, 2001] Ssynth composed of two parts: 1 2
Pd patches: different mapping strategies and layers additive synthesizer: 1-order or 3-order phase polynomial models [McAulay & Quatieri, 1986]
scalar, vectorized and recursive formulations written in C: stand alone program or Pd object Pd scheduler to have output audio
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Mapping strategies
Mapping strategy Modularity: decoupling gestural control / synthesizer [Wanderley, 2001] 78
Chapter 6 : ESCHER - An Application Example WX7
Data glove
C.P.
control parameters
mapping 1 A.P.
abstract parameters
mapping 2 S.P.
synthesis parameters
synth
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Mapping strategies
Mapping strategy 2 mapping layers (parameter conversion):
higher-level parameters −→ synthesis parameters (partial amplitudes and frequencies)
=⇒ nb of synthesis parameters &
user controls
gestural controller control parameters Mapping for synthesis abstract
Interpolator / navigator
parameters
Interoperability System
synthesis parameters (spec. env.)
(f0, i, d) additive synthesis
SYNTHESIS
2
gesture data −→ higher-level parameters (fundamental frequency, intensity, and dynamics)
INSTRUMENT
1
gesture
sound
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Mapping strategies: layer 1
Mapping strategy (layer 1) gesture data −→ higher-level parameters (fundamental frequency, intensity, and dynamics) [Wanderley, 2001]: 88 Chapter 6 : ESCHER - An Application Example WX7 breath
dynamics (Y) loudness
lip pressure fingering
breath
vibrato (X, relative) fundamental frequency (X, absolute)
dynamics (Y) loudness
lip pressure fingering
vibrato (X, relative) fundamental frequency (X, absolute)
breath dynamics (Y) loudness lip pressure fingering
vibrato (X, relative) fundamental frequency (X, absolute)
Figure 6.13: A possible pedagogical use of mapping strategies.
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Mapping strategies: layer2 (morphing)
Mapping strategy (layer 2) additive frames organized as a 3-dimensional mesh: pitch (7 values) dynamics (3 values; related to loudness and brightness) instrument: clarinet, oboe, saxophone & trumpet
sound synthesis: trajectories in this 3D mesh!
timbre
cs
mi
na
dy
trajectory
fundamental frequency
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Mapping strategies: layer2 (morphing)
Morphing strategies 2 standpoint for the morphing: additive: controlling (ai , fi ) of each partial mixed: controlling fi (additive) separately from ai via S (substractive) |X[m,k]| / dB →
50
0
|H[m,k]| / dB →
−50
0 −20 −40
|S[m,k]| / dB →
40 20 0 −20
0
1000
2000
3000
4000 Time/s →
5000
6000
7000
8000
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Mapping strategies: layer2 (morphing)
More morphing strategies Morphing between N tones: sustain: interpolator within 1 instrument: pitch-shifting and weighting of additive frames with 6= f0 and d morpher between several instruments: weighting of interpolated frames (from several instruments)
attack and release: time-warping additive data (improve timbre quality) [Tellman, Haken & Holloway, 1995]
D4 1
1
B3 Instrument
A3
1
1
pp
mf 1
residual noise [Serra & Smith, 1990]: filtering a white noise with morphed residual envelope (not yet provided)
1
D4 Pitch 1
1
A3 pp
mp
mf
Dynamics
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Mapping strategies: layer2 (spectral envelope)
Control of the spectral envelope spectral envelope = function of frequency simplifies the control of partials amplitudes (useful to morph sounds) properties: envelope fit, smoothness (tuned depending on the application) [Schwarz & Rodet, 1999] various models [Noll, 1964; Oppenheim & Schafer, 1975; Galas & Rodet, 1990]
some models more suited to provide a stable spectral envelope for a given gestural control =⇒ convert a spectral envelope models: formants, cepstrum, LPC class (auto-regressive filter, correlation function and reflection coefficients) [Oppenheim & Schafer, 1975; Kay, 1988; Schwarz & Rodet, 1999] May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Mapping strategies: layer2 (spectral envelope)
Control of the spectral envelope
ac2table
table2ac
Correlation Function r(k)
poly
2ta b
le
2ac poly ly po 2 c a
Boxes parametric models
Cep. Coef c(k) cep2poly
Table E(f)
table2dcep dcep2table ep le2c tab table 2 cep
poly2cep
Table a(i), f(i)
spectral models for2table
Formants
table2for
F(j), G(j), Bw(j)
AR Filter coef. (a(k)), G
rc2ac ac2rc
poly for2 for 2 o p ly rc2 po poly ly 2rc
temporal models Arrows exact conversions:
left_to_right right_to_left
Reflection Coefficients k(j)
approximated conv: left_to_right right_to_left
Figure: Conversions between spectral envelope models, from [Verfaille et al., 2006]
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Mapping strategies: layer2 (spectral envelope)
Other control aspects
About signal sampling rates: input (control): internal control rate: 100 Hz can be changed (database has to be resampled)
output: audio: Fs = 44100 Hz stand-alone version: Ssynth could generate various output signals with 6= Fs (haptics)
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Sound database
Sound database The sound database contains: additive analysis & fundamental frequency estimation performed using Additive spectral envelope models of the partials clarinet & oboe as in Escher, + saxophone & trumpet sounds taken from Studio-on-Line (Ircam) but not yet: more instruments using McGill Master Samples [Opolko & Wapnick, 1987]
spectral envelope models of the residual noise Add a ‘haptic signals’ database, or generate from ‘physical models’? May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Comparing Ssynth with other systems
Comparing Ssynth with other systems
Other systems with sound database interpolation/morphing: Escher: controlled using MIDI, performed additive synthesis through FFT−1 algorithm [Rodet & Depalle, 1992] Loris: enhanced bandwidth synthesis, real time morphing, eg. controlled by the continuum fingerboard using MIDI [Fitz et al., 2003]
Diphone: articulation based on morphing , no real time control [Depalle et al., 1993]
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Comparing Ssynth with Escher
Comparing Ssynth with Escher criteria model interpolation extrapolation instruments directivity polyphony mapping messages
Escher FFT −1
Ssynth temporal (3-order phase poly.) pitch, loudness, dynamic, instrument √ — clar., oboe clar., oboe, sax., trumpet √ (soon) √ — in jMax in Pd MIDI OSC
Table: Comparison between Ssynth and Escher.
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Comparing Ssynth with Escher
Comparing Ssynth with Escher
Summary: Ssynth benefits from better sound synthesis quality with real time synthesis (3-order phase model) control messages with OSC protocol instead of MIDI bigger sound database (to grow up) refined morphing strategies (both interpolation & extrapolation)
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Conclusions
Conclusions
Ssynth is a system developed for interpolating and extrapolating digital musical instruments. It: is based on: modular mapping (2 layers) additive/substractive control for morphing additive synthesis
allows for interpolating & extrapolating the database can synthesize polyphonic sound handles OSC control messages
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Bibliography
About morphing
Ph. Depalle, X. Rodet, Th. Galas, and G. Eckel, “Generalized diphone control,” in Proc. Int. Comp. Music Conf., Tokyo, 1993, pp. 184–7. P. Depalle, G. Garcia, and X. Rodet, “Reconstruction of a castrato voice: Farinelli’s voice,” in Proc. IEEE Workshop on Appl. of Digital Sig. Proc. to Audio and Acoustics, 1995, pp. 242–5. K. Fitz, L. Haken, S. Lefvert C. Champion, and M. O’Donnell, “Cell-Utes and Flutter-Tongued Cats: Sound Morphing Using Loris and the Reassigned Bandwidth-Enhanced Model,” Computer Music J., vol. 27, no. 3, pp. 44–65, 2003. E. Tellman, L. Haken, and B. Holloway, “Timbre Morphing of Sounds with Unequal Numbers of Features,” J. Audio Eng. Soc., vol. 43, no. 9, pp. 678–89, 1995.
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Bibliography
About spectral envelope T. Galas and X. Rodet, “An improved cepstral method for deconvolution of source-filter systems with discrete spectra: Application to musical sounds,” in Proc. Int. Comp. Music Conf., Glasgow, 1990, pp. 82–8. S. M. Kay, Modern Spectral Estimation: Theory Application, Prentice-Hall, 1988. A. M. Noll, “Short-time Spectrum and “Cepstrum” Techniques for Vocal Pitch Detection,” J. Acoust. Soc. Am., vol. 36, no. 2, pp. 296–302, 1964. A. V. Oppenheim and R. W. Schafer, Digital Signal Processing, Prentice Hall, Englewood Cliffs, 1975. L. Rabiner and R. Schafer, Digital Processing of Speech Signals, Englewood Cliffs, New Jersey: Prentice-Hall, 19, 1978. D. Schwarz and X. Rodet, “Spectral envelope estimation and representation for sound analysis-synthesis,” in Proc. Int. Comp. Music Conf., Beijing, 1999, pp. 351–4.
May 25-27, 2006 — McGill University
Ssynth: a Real Time Additive Synthesizer With Flexible Control Bibliography
About additive synthesis, sound database & gestural control R. J. McAulay and T. F. Quatieri, “Speech Analysis/Synthesis Based on a Sinusoidal Representation,” IEEE Trans. on Acoustics, Speech, and Sig. Proc., vol. 34, no. 4, pp. 744–54, 1986. F. Opolko and J. Wapnick, “McGill University Master Samples, Montreal, QC, Canada: McGill University,” 1987. X. Rodet and Ph. Depalle, “Spectral envelopes and inverse FFT synthesis,” in 93rd Conv. Audio Eng. Soc., San Francisco, AES preprint 3393 (H-3), 1992. X. Serra and J. O. Smith, “A sound decomposition system based on a deterministic plus residual model,” J. Acoust. Soc. Am., sup. 1, vol. 89, no. 1, pp. 425–34, 1990. M. Wanderley, N. Schnell, and J. B. Rovan, “Escher - modeling and performing composed instruments in real-time,” in Proc. IEEE Int. Conf. on Systems, Man and Cybernetics (SMC’98), San Diego, 1998, pp. 1080–4.
May 25-27, 2006 — McGill University
