Speech
Speech production
• what’s so good about speech? 1. Speech is a unique faculty to humans and one of the most important, requiring the precise control and co-ordination of over eighty different muscles, making speech the highest learned skill a human can achieve. 2. Speaking requires generally around 1,500 muscle commands every second but needs only, as children, a few years to perfect. 3. Although a primary means of communication in itself, speech can convey other messages through accents, tone, pitch, and quality. 4. …..but only limited achievements in machine speech recognition and synthesis 2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 1
• The act of speech involves three major anatomical subsystems: 1. respiratory system including the lungs, rib cage, and diaphragm;
2. phonatory system which includes the larynx;
3. articulatory features the lips, teeth, tongue, and jaw.
2001 Christian Martyn Jones
Respiratory system
Speech and Natural Language Processing
Speech – Slide 2
Phonatory system
nasal cavity
alveolar ridge velum
teeth lips
epiglottis
tongue
esophagus
glottis larynx
lungs
diaphragm
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 3
Articulatory system
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 4
Articulation of consonants
• ….considering in tern how we create
• How we classify the production of consonants involves
consonants vowels other sound
1. the place of articulation (the relative position of the lips, teeth, and tongue),
2. the manner of articulation ….using the glottis, tongue, teeth, lips, and nasal cavity
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 5
(how the air-stream from the lungs is obstructed stops, fricatives, affricates, nasals, liquids and glides),
3. and whether the vocal cords are set to vibrate
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 6
1
…..will use American English
Annotation
….rather than British English but why?
• ASR takes an acoustic waveform as input and produces as output a string of words
• my work • most other researchers work
• TtS takes a sequence of text words to produce an acoustic waveform.
….and will also simplify • not consider regional accents
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 7
2001 Christian Martyn Jones
…two main alphabet standards
standard originally developed by international phonetic association in 1888 with the idea to transcribe all human languages it is more than just a set of symbols (eg one to one relationship to sounds) and differ for different languages
• 2. ARPAbet using ASCII characters rather than more ’nonstandard’ characters makes it much easier to create phonetic dictionaries, syntactic and semantic rules, and build them into ASR systems.....as we will see. Speech and Natural Language Processing
Speech – Slide 8
Place of articulation
• 1. International Phonetic Alphabet (IPA)
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 9
• …..refers to the relative positions of the lips, teeth and tongue. • There are six distinct types of classification: bilabial, labiodental, interdental, alveolar, alveo-palatal, and velar.
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 11
• The six places of articulation
nasal cavity soft palate uvula hard palate palate alveolar ridge lips tip of tongue blade of tongue back of tongue jaw
2001 Christian Martyn Jones
Speech and Natural Language Processing
bilabial, labiodental, interdental, alveolar, alveopalatal, and velar describe the parts of the vocal tract which are responsible for the obstruction of the air flow from the lungs
• the degree of obstruction the airstream incurs must also be considered …...this is the manner of articular.
Speech – Slide 12
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 15
2
Manner of articulation
Voicing
• The manner of articulation describes…
• Voicing
...how, and to what degree, air from the lungs is obstructed • Terms used: stops, fricatives, affricates, nasals, lateral, retroflex, and glides 2001 Christian Martyn Jones
vibration of the vocal cords in order to change the characteristics of the airstream through the mouth or nose and the overall acoustic nature of the phone.
• ...sounds that are generated with the vocal cords vibrating are voiced, and conversely those sound requiring static vocal cords are voiceless.
Speech and Natural Language Processing
Speech – Slide 16
2001 Christian Martyn Jones
Articulation of vowels
Speech and Natural Language Processing
Speech – Slide 19
Features to consider
• The articulation of American English vowels is not as defined as consonants and can vary a great deal from speaker to speaker, especially due to dialect variations. • Vowels generally present little obstruction and require wide open mouth positions.
• What we look for are: 1. tongue elevation, 2. part of tongue involved, 3. tongue muscle tension, 4. mouth shape.
• The description of vowels will consider the motion of the tongue and lips.
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 21
2001 Christian Martyn Jones
Elevation of the tongue
Speech and Natural Language Processing
Speech – Slide 22
Examples
• normally categorised in terms of simply: high, mid, or low positions.
• /iy/ in ‘beet’ and /ih/ in ‘bit’ • /ey/ in ‘bait’ and /eh/ in ‘bet’ • /ae/ in ‘bat’
high front
mid front
• /uw/ in ‘boot’ and /uh/ in ‘book’
low front
• /ow/ in ‘boat’ and /ao/ in ‘bought’ • /aa/ in ‘bott
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 23
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 24
3
Region of the tongue
Tongue tenseness
• We classify the region of the tongue as either: front, central, or back
• vowels requiring above the ‘normal’ level of muscle tension are termed tense
we have already seen front vowels:
• whilst those in which this degree of tension is not needed are simply relaxed.
'beet’, ‘bit’, ‘bait’, ‘bet’, 'bat’
• Consider list ‘beat’ and ‘bit, ‘bait’ and ‘bet’, and ‘boot’ and ‘book’
and back vowels such as: ‘boot’, ‘book’, ‘boat’, ‘bought’, and ‘bott’
- the first phone in each group is tense, with the tongue slightly higher in the mouth than the relaxed second phone
central vowels such as:
- in addition, front vowels which are tense are articulated with the tongue slightly ahead of a similarly relax phone, whilst back tense vowels will be pronounced with the tongue further back.
/ah/ in ‘but’ and schwa vowel in ‘machine’
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 25
2001 Christian Martyn Jones
Mouth shape
Speech and Natural Language Processing
Speech – Slide 26
Other phonetic articulations
• a range of lip positions for all phones
• these require more dynamic movement of the tongue and lips than can be described within the constraints of consonants and vowels, and include:
• however vowels possess some degree of generalisation
Off-glides, diphthongs, and r-colourisation consonant /h/
vowels: 'beet’, ‘bit’, ‘bait’, ‘bet’, 'bat’, ‘machine’, and ‘but’
whilst the back vowels of: ‘boot’, ‘book’, ‘boat’, ‘bought’:
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 27
2001 Christian Martyn Jones
Off-glides
Speech and Natural Language Processing
Speech – Slide 29
Diphthongs
•
Already considered are the high, front, tense /iy/ ('beet'), the mid, front, tense /ey/ ('bait'), the high, back, tense /uw/ ('boot'), and the mid, back, tense /ow/ ('boat') vowels, however the articulation of each is more than these simple descriptors can suggest.
•
Similar in composition to off-glides, diphthongs are complex vowels consisting of a vowel sound followed by the glide /y/ or /w/. However, what separates diphthongs from off-glides is that diphthongs involve considerably greater tongue motion
•
The front vowels (/iy/ and /ey/) are composed of a pure vowel /i/ (idealised position) and /e/ followed immediately by the glide /y/ (blade of tongue near hard palate), and similarly the back vowels involve vowels /u/ and /o/ followed by /w/.
•
The diphthongs of American English are /aw/ in ‘bout’, /ay/ in ‘bite’, and /oy/ in ‘boy’.
•
Each of the constituent phones are pronounced and these vowels are known as off-glides indicating the increased motion of the tongue and lips.
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 30
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 31
4
R-colourisation
/h/
•
The combination of vowel and /r/ sounds are termed as rcolourisation, and differ from off-glides and diphthongs in that they are in fact two symbols representing a single sound.
•
The r-coloured, mid, central vowel /er/ as in ‘bird’ involves the articulation of two previously described phones at one time:
•
the mid, central vowel pronunciation of the schwas as in ‘machine’
have already mentioned consonant /h/ assumes the tongue and lip positions of the proceeding vowel. As the articulation of the consonant /h/ is very much dependent on that of the vowel, and vowels do not require the vibration of the vocal cords, it is often described as a voice-less vowel although grammatically a consonant.
together with the tongue curl associated with the retroflexed /r/.
•
The tongue is again in motion during pronunciation, however does not exceed the boundaries of tongue elevation and region as is the case with diphthongs.
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 32
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 33
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 36
Deaf studies • will consider the field of speechreading addressing the visual similarity in speech.
• Research number of studies since 1950s to the present day major works from Jeffers, Nitchie, Berger, Fisher, my own studies look at here is Nitchie 1979
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 34
….less info for speechreaders
lip cues for all…..
• Larry Thronson, a sign language instructor and counsellor at the Central Coast Centre for Independent Living in California has said: “Lipreading is guesswork; under ideal conditions, only about 30 to 40 percent (of the speech) is retained”
• With my research into lip syncing I found that it can be more like 50% to 60% of the information is lost
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 37
•
how much attention we pay to the visual and auditory information can vary depending on the ‘cocktail party effect’ and also subject matter:
•
Predictable words in a conversation are spoken less clearly, as are references to objects present or passed between speakers in view, however, when an object is mentioned for the first time it is named more clearly than subsequent times
•
When speakers are face-to-face their speech becomes degraded even when they are not actually facing each other. Although it can be seen that visual cues can greatly assist communications, speakers generally rarely look at each other during conversations which seems to contradict suggestions that speakers adopt their articulation to the needs of the listener. If this was the case then one may speak less clearly when being watched, however the reverse is true.
•
Although we generally use visual cues less frequently than would initially be imagined (typically less that 50% of the time), when the listener does look to the speaker’s face the speaker then articulates much clearer than before
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 38
5
Eye gaze
Bad lip sync.
•
Comfort:
•
In the situations of video teleconferencing or cartoon-style animation, the viewer is aware that the image is synthetic and is less prone to feel uneasy looking at the speakers face. Therefore, the viewer will look at the image for a greater percentage of the overall time implying that the level of accuracy of the articulation should be as high as possible.
•
However, the psychology of human subjects is not always consistent and in fact listeners find that excessive precision in the articulation of the speaker to be generally annoying.
•
Similarly, during face-to-face conversations subjects subconsciously negotiate an acceptable level of mutual eye contact depending on a level of intimacy and comfort.
•
Note that the very latest computer facial models provide very life-like persona, and it is plausible that in the future such models may well be perceived to be real-life images, in which case listeners will find themselves again unable to make prolonged eye contact during the communication.
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 39
•
Although most people have not had any formal speechreading training and therefore cannot accurately lip-read, bad lip synchronisation is easily detectable and highly unacceptable to the viewer
•
Extreme cases has been sighted by Groß at Oxford University, England, in which films dubbed in different languages can not be synchronised with the visuals and can cause irritation to audience
•
In countries such as Germany and France which regularly dub British or American films, the spectators appear to be able to ignore the discrepancy between what they see and what they hear as if the brain can be trained to accept bad lip synchronisation when the situation is nonsensical.
•
However, if the same audience watch a German actor dubbed in to another language such as French then they strongly object to the inconsistency between the speech and mouth synchronisation.
2001 Christian Martyn Jones
Cartoon lip sync. •
•
Speech and Natural Language Processing
Speech – Slide 40
Audio/visual mis-matches
Cartoon lip synchronisation does not require the same level of accuracy as its human counterpart. Instead, animators generally supply only keyframes in the articulation and the human mind ‘fills’ in the gaps. The audience perceive the characters in animation to be life-like but at the same time not human and thus are happy to ignore inaccuracies in the lip synchronisation which would otherwise be unacceptable. Therefore animators are able to get away with poor articulation models for simple characters.
• McGurk effect • what do you….
Disney animators found that although mouth motion required exact synchronisation, other speechreading cues such as head, body, or gestures needed to be synchronised three to four frames ahead of the visual action
...hear? ...see? …with both?
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 41
2001 Christian Martyn Jones
McGurk research
• ……see results
Speech and Natural Language Processing
Speech – Slide 42
why does it occur?
• The work involved filming a female subject whilst repeated uttering ‘ba-ba’, ‘ga-ga’, ‘pa-pa’, ‘ka-ka’, and generating four dubbed audio-video sequences in which the original sound-track and lip movements were combined correctly and mis-matched as: [ba] (voice)-[ga] (lips), [ga] (voice)-[ba] (lips), [pa] (voice)[ka] (lips), and [ka] (voice)-[pa] (lips).
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 43
•
The McGurk effect can be explained in terms of the visual similarity
•
Speechreading studies show that lip positions for [ga] are frequently misread as [da], [ka] misread as [ta], and [pa] for [ba].
•
McGurk assumed that the acoustic information for [ba] and [da] contained some common features which were not present in [ga]. Thus a [ba] (voice)-[ga] (lips) presentation provided the viewer with visual information common to [ga] and [da] and auditory information with features common to [da] and [ba]. The spectator would then respond with the phone-code for which there was most data: [da].
•
A similar explanation was presupposed for the [pa] (voice)-[ka] (lips) effects and reversing the audio and visual stimuli. When the acoustic information does not bear any similarity with the articulation, in the case of [ka] (voice)-[pa] (lips) the viewer invariably must guess the spoken message and responds with combinations of [kapka], [pakpa] etc.
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 44
6
….further work
…by now
•
Burnham and Dodd researching in Australia found that the McGurk effect was observed in infants as young as 4 month-old and that the auditory [ba] and visual [ga] could be perceived as [dh] as well as [da].
•
Also noted the effect to transcend language and phonological constraints.
•
Work by Massaro has considered the use of synthetic faces on the McGurk effect. Massaro has extended the sensory mismatch to include auditory /b/ and visual /d/ to perceived /w/.
• you should have an appreciation of: speech, how it is produced, what articulators are involved, how to annotate speech using symbols, how speech appears visually, similarities in visual speech, and mis-information cues and difficulties for hearing impaired.
• next…....how this affects the acoustics of speech...... 2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 45
2001 Christian Martyn Jones
Speech and Natural Language Processing
Speech – Slide 46
7
