Adaptive Digital Audio Effects (A-DAFx) work with Daniel Arfib
Vincent Verfaille
CNRS-LMA 31, Joseph Aiguier 13402 Marseille Cedex France
Limerick - December, 2001
1
Adaptive digital audio effects (A-DAFx)
Guidelines of the Presentation
1- Aim of an A-DAFx 2- Feature extraction 3- Mapping between features and Fx control 4- Effects implemented 5- Conclusions
Limerick - December, 2001
2
Adaptive digital audio effects (A-DAFx)
Structure of an adaptive digital audio effect (A-DAFx) • Steps : 1- analysis / feature extraction (Rossignol, Métois) 2- mapping between analysis parameters and effect control parameters 3- DAFx / synthesis
Limerick - December, 2001
3
Adaptive digital audio effects (A-DAFx)
Aim of an adaptive digital audio effect (A-DAFx)
• Principle: dynamic control of an effect • Interest: evolutive effects • First examples: compressor, adaptive delay, selective time-stretching • Sounds: evolutive sounds, in order to have varying features
Limerick - December, 2001
4
Adaptive digital audio effects (A-DAFx)
1- Aim of an A-DAFx 2- Feature extraction 3- Mapping between features and Fx control 4- Effects implemented 5- Conclusions
Limerick - December, 2001
5
Adaptive digital audio effects (A-DAFx)
Global features • Global features (from a simple calculus) : - voiced/unvoiced indicator, - energy (RMS), - centroid, - fundamental frequency.
Example: global features extracted for a sung voice
Limerick - December, 2001
6
Adaptive digital audio effects (A-DAFx)
High-level features High-level features (from a spectral line analysis; Serra, Rodet) : - harmonic frequencies, - harmonicity of the sound, - odd/even harmonic indicator (Arfib), - energy of two sound components : harmonics, residual; more: - harmonics’ synchronism (Dubnov), - harmonicity of each partial, - energy of three sound components: harmonics, transient, residual. Computer assisted partial tracking:
Example of harmonic ’s frequencies extraction for a sound with a strong reverberation
Limerick - December, 2001
7
Adaptive digital audio effects (A-DAFx)
1- Aim of an A-DAFx 2- Feature extraction 3- Mapping between features and Fx control 4- Effects implemented 5- Conclusions
Limerick - December, 2001
8
Adaptive digital audio effects (A-DAFx)
Mapping between features and Fx control
• Linear combination N to 1 (Fg) • Including a non linearity M
• Fitting the mapped curve to effect parameter’s boundaries Δm and ΔM
Structure of the mappings between features extracted and Fx controls
Limerick - December, 2001
9
Adaptive digital audio effects (A-DAFx)
1- Aim of an A-DAFx 2- Feature extraction 3- Mapping between features and Fx control 4- Effects implemented 5- Conclusions
Limerick - December, 2001
10
Adaptive digital audio effects (A-DAFx)
Analysis methods: phase vocoder Analysis and effect based on a phase vocoder technique:
w1(t)
x(t)
x
u(t)
analysis
w2(t)
STFT
DAFx U(f)
transformation
Limerick - December, 2001
V(f)
ISTFT
v(t)
x
y(t)
synthesis
11
Adaptive digital audio effects (A-DAFx)
Analysis methods: phase vocoder We add the feature extraction and the mapping between features and the Fx controls: F(t)
feature extraction
w1(t)
x(t)
x
u(t)
analysis
STFT
mapping
w2(t)
Δ(t)
A-DAFx U(f)
transformation
Limerick - December, 2001
V(f)
ISTFT
v(t)
x
y(t)
synthesis
12
Adaptive digital audio effects (A-DAFx)
Selective time-stretching • fixed synthesis hop size • variable analysis hop size to stretch or expand time 1st example: flute sentence
Fundamental frequency Wave form of flute sound
Wave form of first time-stretching effect example
Wave form of second time-stretching effect example
Limerick - December, 2001
Control curve: time-stretching factor
Control curve: time-stretching factor
13
Adaptive digital audio effects (A-DAFx)
Selective time-stretching 2nd example: voice sentence (Pierre Schaeffer) - original sound RMS
- time-stretched (RMS) Control curve for a selective time-stretching
Mapping function
Control curve for a selective time-stretching
Mapping function
(-RMS)
Limerick - December, 2001
14
Adaptive digital audio effects (A-DAFx)
Whisperization Whisperation: giving random values to the phase of each FFT bin
Randomization of a 2048- samples window
parameter: window size The smaller the window, the better the whispered effect
Limerick - December, 2001
15
Adaptive digital audio effects (A-DAFx)
Whisperization Example: Suzanne Vega’s voice
Control curve (extracted from F0)
Suzanne Vega whispering
Suzanne Vega’s song
Limerick - December, 2001
16
Adaptive digital audio effects (A-DAFx)
Robotization • Robotization of the voice: giving a 0 value to phase of all FFT bins
0-phasing for a 64-samples window
0-phasing for a 1024-samples window
• Parameter: analysis hop size (period: inverse of the robot fundamental frequency) • For fixed pitch, the smaller the grain, the better the effect (to avoid comb filtering effect) • For varying pitch, greater grains are needed. Limerick - December, 2001
17
Adaptive digital audio effects (A-DAFx)
Robotization
Example: Pierre Schaeffer’s sentence • original sound • robotization (Hop size = 500 samples) • a-robotization H.S. in [460;540] H.S. in [300;400]
Wave form
RMS
Effect control curve
Limerick - December, 2001
18
Adaptive digital audio effects (A-DAFx)
Adaptive granular delay Granular delay: copying and displacing the grains, with several repetitions. Control parameters: - delay time, - « feedback » gain, - window size. Original sound Fx: - normal delay: g=0.4, - delay time a-delay - feedback a-delay
Wave form
RMS
t=300 ms a-delay, g=0.4, t in [200;400] ms
a-delay, g in [0;0.4], t=300 ms
Limerick - December, 2001
a-delay, g in [0;0.4], t=300 ms
19
Adaptive digital audio effects (A-DAFx)
1- Aim of an A-DAFx 2- Feature extraction 3- Mapping 4- Effects implemented 5- Conclusions
Limerick - December, 2001
20
Adaptive digital audio effects (A-DAFx)
Perspectives for the A-DAFx
• non real-time A-DAFx tool with Matlab: many news perceptively different effects depending on the mapping between features extracted and Fx control parameters; • real-time version as plug-ins (VST - Max/MSP): to test other mapping strategies (boundaries of features unknown); • real-time version with gesture control (Max/MSP): expressivity, link with musical gesture and gesture controllers (Arfib).
Limerick - December, 2001
21
Adaptive Digital Audio Effects (A-DAFx) work with Daniel Arfib
Vincent Verfaille
CNRS-LMA 31, Joseph Aiguier 13402 Marseille Cedex France
Limerick - December, 2001
22
Adaptive digital audio effects (A-DAFx)
Application to gliding sounds
• Gliding sounds : sounds with pitch changes (vibrato, portamento, glissando, musical sentences) • Interests : - interesting perceptive effects (example: vibrato, cf. Desain, & Honing ; transitions, cf. Strawn) - evolution of a great number of parameters - used in electroacoustic music
Limerick - December, 2001
23
Adaptive digital audio effects (A-DAFx)
A-DAFx proposed 1 - Selective time-stretching 2 - robotization 3 - whisperization 4 - granular delay More: 5 - filtering 6 - pitch shifting 7 - vibrato 8 - compressor/limitor etc
Limerick - December, 2001
24
