THE ELECTRICAL ENGINEERING AND APPLIED SIGNAL PROCESSING SERIES Edited by Alexander Poularikas The Advanced Signal Processing Handbook: Theory and Implementation for Radar, Sonar, and Medical Imaging Real-Time Systems Stergios Stergiopoulos The Transform and Data Compression Handbook K.R. Rao and P.C. Yip Handbook of Multisensor Data Fusion David Hall and James Llinas Handbook of Neural Network Signal Processing Yu Hen Hu and Jenq-Neng Hwang Handbook of Antennas in Wireless Communications Lal Chand Godara Noise Reduction in Speech Applications Gillian M. Davis Signal Processing Noise Vyacheslav P. Tuzlukov Digital Signal Processing with Examples in MATLAB® Samuel Stearns Applications in Time-Frequency Signal Processing Antonia Papandreou-Suppappola The Digital Color Imaging Handbook Gaurav Sharma Pattern Recognition in Speech and Language Processing Wu Chou and Biing Huang Juang

Forthcoming Titles Propagation Data Handbook for Wireless Communication System Design Robert Crane Smart Antennas Lal Chand Godara Nonlinear Signal and Image Processing: Theory, Methods, and Applications Kenneth Barner and Gonzalo R. Arce

‹E\&5&3UHVV//&

Forthcoming Titles (continued) Soft Computing with MATLAB® Ali Zilouchian Signal and Image Processing Navigational Systems Vyacheslav P. Tuzlukov Wireless Internet: Technologies and Applications Apostolis K. Salkintzis and Alexander Poularikas

‹E\&5&3UHVV//&

PATTERN RECOGNITION in SPEECH and LANGUAGE PROCESSING Edited by

WU CHOU Avaya Labs Research

BIING HWANG JUANG Georgia Institute of Technology

CRC PR E S S Boca Raton London New York Washington, D.C. ‹E\&5&3UHVV//&

/LEUDU\RI&RQJUHVV&DWDORJLQJLQ3XEOLFDWLRQ'DWD 3DWWHUQUHFRJQLWLRQLQVSHHFKDQGODQJXDJHSURFHVVLQJHGLWHGE\:X&KRXDQG %LLQJ+ZDQJ-XDQJ SFP ,QFOXGHVELEOLRJUDSKLFDOUHIHUHQFHVDQGLQGH[ ,6%1DONSDSHU $XWRPDWLFVSHHFKUHFRJQLWLRQ3DWWHUQUHFRJQLWLRQV\VWHPV,&KRX:X,, -XDQJ%+%LLQJ+ZDQJ 7.63 d³GF 

 

7KLVERRNFRQWDLQVLQIRUPDWLRQREWDLQHGIURPDXWKHQWLFDQGKLJKO\UHJDUGHGVRXUFHV5HSULQWHGPDWHULDO LVTXRWHGZLWKSHUPLVVLRQDQGVRXUFHVDUHLQGLFDWHG$ZLGHYDULHW\RIUHIHUHQFHVDUHOLVWHG5HDVRQDEOH HIIRUWVKDYHEHHQPDGHWRSXEOLVKUHOLDEOHGDWDDQGLQIRUPDWLRQEXWWKHDXWKRUVDQGWKHSXEOLVKHUFDQQRW DVVXPHUHVSRQVLELOLW\IRUWKHYDOLGLW\RIDOOPDWHULDOVRUIRUWKHFRQVHTXHQFHVRIWKHLUXVH 1HLWKHUWKLVERRNQRUDQ\SDUWPD\EHUHSURGXFHGRUWUDQVPLWWHGLQDQ\IRUPRUE\DQ\PHDQVHOHFWURQLF RUPHFKDQLFDOLQFOXGLQJSKRWRFRS\LQJPLFURÀOPLQJDQGUHFRUGLQJRUE\DQ\LQIRUPDWLRQVWRUDJHRU UHWULHYDOV\VWHPZLWKRXWSULRUSHUPLVVLRQLQZULWLQJIURPWKHSXEOLVKHU $OO ULJKWV UHVHUYHG$XWKRUL]DWLRQ WR SKRWRFRS\ LWHPV IRU LQWHUQDO RU SHUVRQDO XVH RU WKH SHUVRQDO RU LQWHUQDO XVH RI VSHFLÀF FOLHQWV PD\ EH JUDQWHG E\ &5& 3UHVV //& SURYLGHG WKDW  SHU SDJH SKRWRFRSLHGLVSDLGGLUHFWO\WR&RS\ULJKW&OHDUDQFH&HQWHU5RVHZRRG'ULYH'DQYHUV0$ 86$ 7KH IHH FRGH IRU XVHUV RI WKH 7UDQVDFWLRQDO 5HSRUWLQJ 6HUYLFH LV ,6%1  7KHIHHLVVXEMHFWWRFKDQJHZLWKRXWQRWLFH)RURUJDQL]DWLRQVWKDWKDYHEHHQJUDQWHG DSKRWRFRS\OLFHQVHE\WKH&&&DVHSDUDWHV\VWHPRISD\PHQWKDVEHHQDUUDQJHG 7KHFRQVHQWRI&5&3UHVV//&GRHVQRWH[WHQGWRFRS\LQJIRUJHQHUDOGLVWULEXWLRQIRUSURPRWLRQIRU FUHDWLQJQHZZRUNVRUIRUUHVDOH6SHFLÀFSHUPLVVLRQPXVWEHREWDLQHGLQZULWLQJIURP&5&3UHVV//& IRUVXFKFRS\LQJ 'LUHFWDOOLQTXLULHVWR&5&3UHVV//&1:&RUSRUDWH%OYG%RFD5DWRQ)ORULGD 7UDGHPDUN1RWLFH 3URGXFWRUFRUSRUDWHQDPHVPD\EHWUDGHPDUNVRUUHJLVWHUHGWUDGHPDUNVDQGDUH XVHGRQO\IRULGHQWLÀFDWLRQDQGH[SODQDWLRQZLWKRXWLQWHQWWRLQIULQJH

9LVLWWKH&5&3UHVV:HEVLWHDWZZZFUFSUHVVFRP ‹E\&5&3UHVV//& 1RFODLPWRRULJLQDO86*RYHUQPHQWZRUNV ,QWHUQDWLRQDO6WDQGDUG%RRN1XPEHU /LEUDU\RI&RQJUHVV&DUG1XPEHU 3ULQWHGLQWKH8QLWHG6WDWHVRI$PHULFD 3ULQWHGRQDFLGIUHHSDSHU

‹E\&5&3UHVV//&

Preface

#RRTQCEJGU VQ VJG RTQDNGOU QH FGUKIPKPI URGGEJ CPF NCPIWCIG RTQEGUUKPI CNIQTKVJOU HQT JWOCP OCEJKPG EQOOWPKECVKQP WUGF VQ DG VCMGP HTQO VJG RGTURGEVKXGU QH NKP IWKUVKEU CPF URGGEJ UEKGPEG WPVKN VJG NCVG U &WG VQ VJG CFXCPEGU KP EQORWV KPI CPF UVCVKUVKECN OQFGNKPI FCVC FTKXGP RCVVGTP TGEQIPKVKQP OGVJQFU JCXG DGEQOG C HCUV OQXKPI TGUGCTEJ CTGC FWTKPI VJG RCUV VYQ FGECFGU CPF EQPVTKDWVGF OWEJ VQ VJG RTQITGUU KP VJKU ſGNF #U VJG GTC QH KPHQTOCVKQP CIG EQPVKPWGU VQ FGXGNQR YG YKVPGUU CP GXGT KPETGCUKPI PGGF KP KPVGNNKIGPV JWOCPOCEJKPG EQOOWPKECVKQPU CU YGNN CU VJG ETGCVKQP QH OCEJKPG WPFGTUVCPFCDNG OGVCFCVC HQT 9GD EQPVGPV CPF QVJGT KPHQTOCVKQP UQWTEGU 6JKU JCPFDQQM KU VQ ſNN VJG PGGF QH C U[UVGOCVKE CPF WRVQFCVG RTGUGPVCVKQP QH PGY RCVVGTP TGEQIPKVKQP CRRTQCEJGU KP URGGEJ CPF NCPIWCIG RTQEGUUKPI 6JG DQQM UVCTVU YKVJ HWPFCOGPVCNU CPF TGEGPV VJGQTGVKECN CFXCPEGU KP RCVVGTP TGEQI PKVKQP YKVJ CP GORJCUKU QP ENCUUKſGT FGUKIP ETKVGTKC CPF QRVKOK\CVKQP RTQEGFWTGU +V EQXGTU UGXGTCN TGEGPV TGUGCTEJ CFXCPEGU KP VJKU CTGC UWEJ CU VJG OKPKOWO GTTQT TCVG /%' OGVJQF VJG OKPKOWO $C[GU TKUM CRRTQCEJ CFCRVKXG U[UVGO FGUKIP CPF FGEKUKQP TWNGU PGWTCN PGVYQTMU FKUVTKDWVGF TGEQIPK\GTU CPF FGEKUKQP HWUKQP 6JGUG OGVJQFU FGRCTV HTQO VJG EQPXGPVKQPCN RCTCFKIO YJKEJ NKPMU C ENCUUKſGT FGUKIP VQ VJG ENCUUKECN RTQDNGO QH FKUVTKDWVKQP GUVKOCVKQP +PUVGCF OQTG OGCPKPIHWN ETKVGTKC CTG KPVTQFWEGF YJKEJ UKIPKſECPVN[ KORTQXG VJG FKUETKOKPCVKQP RQYGT QH C ENCUUKſGT RCTVKEWNCTN[ YJGP CRRNKGF VQ URGGEJ RTQDNGOU KP YJKEJ VJG PQVKQP QH FCVC FKUVTKDWVKQP KU FKHſEWNV VQ TGCNK\G 6JG UGEQPF RCTV QH VJG DQQM KU VJGTGHQTG URGEKCNN[ HQEWUGF QP VJG CRRTQCEJGU CPF OGVJQFU CRRNKGF VQ URGGEJ RTQEGUUKPI +V EQXGTU VQRKEU UWEJ CU $C[GU OKPKOWO TKUM CRRTQCEJ VQ URGGEJ TGEQIPKVKQP NCTIG XQECDWNCT[ URGGEJ TGEQIPKVKQP DCUGF QP UVCVKU VKECN OGVJQFU TGEQIPKVKQP QH URQPVCPGQWU URGGEJ KP FKCNQIWG KPVGTCEVKQP URGGEJ CPF URGCMGT XGTKſECVKQP CPF CWFKQ KPHQTOCVKQP TGVTKGXCN CPF KPFGZKPI 6JGUG EJCRVGTU RTQXKFG C EQORTGJGPUKXG EQXGTCIG QH TGEGPV CFXCPEGU KP CRRN[KPI RCVVGTP TGEQIPK VKQP VQ TGCN U[UVGOU KP URGGEJ CPF CWFKQ RTQEGUUKPI 6JG VJKTF RCTV QH VJG DQQM KU FGXQVGF VQ VQRKEU QH RCVVGTP TGEQIPKVKQP KP NCPIWCIG RTQEGUUKPI +V EQPVCKPU EJCRVGTU KP NCPIWCIG OQFGNKPI DCUGF QP NCVGPV UGOCPVKE KP FGZKPI UCNKGPV KPHQTOCVKQP TGRTGUGPVCVKQP CPF RTQEGUUKPI KP PCVWTCN NCPIWCIG FKC NQIWG U[UVGO UVCVKUVKECN OCEJKPG VTCPUNCVKQP OGVJQFU KP VQRKE FGVGEVKQP VTCEMKPI CPF PCOG KFGPVKV[ KFGPVKſECVKQP 6JGUG VQRKEU CTG PGY VTGPFU KP NCPIWCIG RTQEGUU KPI CPF UKIPKſECPV RTQITGUU JCU DGGP OCFG KP TGEGPV [GCTU +V JCU C FKTGEV KORCEV VQ VJG RTCEVKEG CPF KORNGOGPVCVKQP QH KPHQTOCVKQP RTQEGUUKPI U[UVGOU HQT 9GD EQPVGPV DTQCFECUV PGYU CPF QVJGT EQPVGPVTKEJ KPHQTOCVKQP TGUQWTEGU 6JKU DQQM KU C EQNNGEVKXG GHHQTV OQVKXCVGF D[ VJG GZEKVGOGPV QH VJG PGY CFXCPEGU KP ‹E\&5&3UHVV//&

VJKU ſGNF CPF VJG WTIGPV PGGF VQ DTKPI VJGUG CFXCPEGU VQ C IGPGTCN CWFKGPEG 6JG EQP VTKDWVKPI CWVJQTU QH VJKU DQQM CTG NGCFKPI GZRGTVU KP VJG ſGNF QH URGGEJ CPF NCPIWCIG RTQEGUUKPI #VVGORVU CTG OCFG VQ OCMG GCEJ EJCRVGT UGNHEQPVCKPGF CPF EQORTG JGPUKDNG HQT TGCFGTU YKVJ IGPGTCN DCEMITQWPF KP RCVVGTP TGEQIPKVKQP CPF KPHQTOCVKQP RTQEGUUKPI +V KU KPVGPFGF VQ DG C JCPFDQQM QT TGHGTGPEG VGZVDQQM HQT TGUGCTEJGTU ITCFWCVG UVWFGPVU CPF CFXCPEGF WPFGTITCFWCVG UVWFGPVU YJQ YCPV VQ HQNNQY VJG PGY CFXCPEGU KP RCVVGTP TGEQIPKVKQP 5WHſEKGPV TGHGTGPEGU CTG RTQXKFGF CV VJG GPF QH GCEJ EJCRVGT VQ UGTXG CU CP GPVT[ RQKPV HQT CP KPVGTGUVGF TGCFGT VQ RWTUWG HWTVJGT 9G YQWNF NKMG VQ VJCPM CNN EQPVTKDWVQTU QH VJKU DQQM 9KVJQWV VJGKT EQOOKVOGPV CPF SWCNKV[ QH YQTM VJKU DQQM YQWNF PQV DG RQUUKDNG 9G CRRTGEKCVG VJG UWRRQTV CPF GPEQWTCIGOGPV HTQO QWT EQNNGCIWGU CV #XC[C .CDU 4GUGCTEJ FWTKPI VJG RTGRCTCVKQP QH VJKU DQQM +V YCU C RNGCUCPV YQTMKPI GZRGTKGPEG YKVJ %4% 2TGUU  VJGKT VGEJPKECN UWRRQTV YCU XGT[ JGNRHWN VQ WU

9W %JQW $KKPI*YCPI ,WCPI Basking Ridge, New Jersey September, 2002

‹E\&5&3UHVV//&

Contributors

A. Abella 5RGGEJ 4GUGCTEJ #66 .CDQTCVQTKGU (NQTJCO 2CTM 0, James Allan &GRCTVOGPV QH %QORWVGT 5EKGPEG 7PKXGTUKV[ QH /CUUCEJWUGVVU #OJGTUV #OJGTUV /# T. Alonso 5RGGEJ 4GUGCTEJ #66 .CDQTCVQTKGU (NQTJCO 2CTM 0, Jerome R. Bellegarda 5RQMGP .CPIWCIG )TQWR #RRNG %QORWVGT +PE %WRGTVKPQ %# William Byrne %GPVGT HQT .CPIWCIG CPF 5RGGEJ 2TQEGUUKPI ,QJPU *QRMKPU 7PKXGTUKV[ $CNVKOQTG /&

7PKXGTUKVyG FG 2CTKU 5WF 1TUC[ %GFGZ (TCPEG Vaibhava Goel 6, 9CVUQP 4GUGCTEJ %GPVGT +$/ ;QTMVQYP *GKIJVU 0; Allen L. Gorin 5RGGEJ 4GUGCTEJ #66 .CDQTCVQTKGU (NQTJCO 2CTM 0, Qiang Huo &GRCTVOGPV QH %QORWVGT 5EKGPEG CPF +PHQTOCVKQP 5[UVGOU 6JG 7PKXGTUKV[ QH *QPI -QPI *QPI -QPI %JKPC Biing-Hwang Juang #XC[C .CDU 4GUGCTEJ $CUMKPI 4KFIG 0,

Wu Chou #XC[C .CDU 4GUGCTEJ $CUMKPI 4KFIG 0,

Shigeru Katagiri +PVGNNKIGPV %QOOWPKECVKQP .CDQTCVQT[ CPF 5RGGEJ 1RGP .CDQTCVQT[ 0KRRQP 6GNGITCRJ CPF 6GNGRJQPG %QTRQTCVKQP 6QM[Q ,CRCP

Sadaoki Furui &GRCTVOGPV QH %QORWVGT 5EKGPEG 6QM[Q +PUVKVWVG QH 6GEJPQNQI[ 6QM[Q ,CRCP

Lori Lamel .+/5+%045 7PKXGTUKVyG FG 2CTKU 5WF 1TUC[ %GFGZ (TCPEG

Jean-Luc Gauvain .+/5+%045

Qi (Peter) Li $GNN .CDQTCVQTKGU

‹E\&5&3UHVV//&

.WEGPV 6GEJPQNQIKGU /WTTC[ *KNN 0, John Makhoul $$0 6GEJPQNQIKGU %CODTKFIG /# Hermann Ney .GJTUVWJN HWGT +PHQTOCVKM 8+ *WOCP .CPIWCIG 6GEJPQNQI[ CPF 2CVVGTP 4GEQIPKVKQP %QORWVGT 5EKGPEG &GRCTVOGPV 7PKXGTUKV[ QH 6GEJPQNQI[ #CEJGP )GTOCP[ F. J. Och .GJTUVWJN HWGT +PHQTOCVKM 8+ *WOCP .CPIWCIG 6GEJPQNQI[ CPF 2CVVGTP 4GEQIPKVKQP %QORWVGT 5EKGPEG &GRCTVOGPV 7PKXGTUKV[ QH 6GEJPQNQI[ #CEJGP )GTOCP[

‹E\&5&3UHVV//&

G. Riccardi 5RGGEJ 4GUGCTEJ #66 .CDQTCVQTKGU (NQTJCO 2CTM 0, Richard M. Schwartz $$0 6GEJPQNQIKGU %CODTKFIG /# J. H. Wright 5RGGEJ 4GUGCTEJ #66 .CDQTCVQTKGU (NQTJCO 2CTM 0,

Contents

1 Minimum Classification Error (MCE) Approach in Pattern Recognition Wu Chou #XC[C .CDU 4GUGCTEJ #XC[C +PE 75#  +PVTQFWEVKQP  1RVKOCN %NCUUKſGT HTQO $C[GU &GEKUKQP 6JGQT[  &KUETKOKPCPV (WPEVKQP #RRTQCEJ VQ %NCUUKſGT &GUKIP  5RGGEJ 4GEQIPKVKQP CPF *KFFGP /CTMQX /QFGNKPI  *KFFGP /CTMQX /QFGNKPI QH 5RGGEJ  /%' %NCUUKſGT &GUKIP 7UKPI &KUETKOKPCPV (WPEVKQPU  /%' %NCUUKſGT &GUKIP 5VTCVGI[  1RVKOK\CVKQP /GVJQFU  1VJGT 1RVKOK\CVKQP /GVJQFU  *// CU C &KUETKOKPCPV (WPEVKQP  4GNCVKQP DGVYGGP /%' CPF //+  &KUEWUUKQPU CPF %QOOGPVU  'ODGFFGF 5VTKPI /QFGN $CUGF /%' 6TCKPKPI  5VTKPI /QFGN $CUGF /%' #RRTQCEJ  %QODKPGF 5VTKPI /QFGN $CUGF /%' #RRTQCEJ  &KUETKOKPCVKXG (GCVWTG 'ZVTCEVKQP  8GTKſECVKQP CPF +FGPVKſECVKQP  5RGCMGT 8GTKſECVKQP CPF +FGPVKſECVKQP  7VVGTCPEG 8GTKſECVKQP  5WOOCT[ 2 Minimum Bayes-Risk Methods in Automatic Speech Recognition Vaibhava Goel £ and William ByrneÝ £ +$/ Ý ,QJPU *QRMKPU 7PKXGTUKV[  /KPKOWO $C[GU4KUM %NCUUKſECVKQP (TCOGYQTM  .KMGNKJQQF 4CVKQ $CUGF *[RQVJGUKU 6GUVKPI  /CZKOWO #2QUVGTKQTK 2TQDCDKNKV[ %NCUUKſECVKQP  2TGXKQWU 5VWFKGU QH #RRNKECVKQP 5GPUKVKXG #54  2TCEVKECN /$4 2TQEGFWTGU HQT #54  5WOOCVKQP QXGT *KFFGP 5VCVG 5GSWGPEGU  /$4 4GEQIPKVKQP YKVJ 0DGUV .KUVU  /$4 4GEQIPKVKQP YKVJ .CVVKEGU  5GIOGPVCN /$4 2TQEGFWTGU  5GIOGPVCN 8QVKPI  418'4 ‹E\&5&3UHVV//&



 

 G418'4 'ZRGTKOGPVCN 4GUWNVU  2CTCOGVGT 6WPKPI YKVJKP VJG /$4 %NCUUKſECVKQP 4WNG  7VVGTCPEG .GXGN /$4 9QTF CPF -G[YQTF 4GEQIPKVKQP  418'4 CPF G418'4 HQT /WNVKNKPIWCN #54 5WOOCT[ #EMPQYNGFIGOGPVU

3 A Decision Theoretic Formulation for Robust Automatic Speech Recognition Qiang Huo 6JG 7PKXGTUKV[ QH *QPI -QPI *QPI -QPI %JKPC  +PVTQFWEVKQP  1RVKOCN $C[GUŏ &GEKUKQP 4WNG HQT #54  #FCRVKXG &GEKUKQP 4WNGU %QPUVTWEVGF HTQO 6TCKPKPI 5CORNGU  2NWIKP $C[GUŏ &GEKUKQP 4WNGU YKVJ /CZKOWONKMGNKJQQF &GPUKV[ 'UVKOCVG  /CZKOWO&KUETKOKPCPV &GEKUKQP 4WNGU /KPKOK\KPI VJG 'O RKTKECN %NCUUKſECVKQP 'TTQT  &KUEWUUKQP  8KQNCVKQPU QH /QFGNKPI #UUWORVKQPU KP #54  6[RGU QH &KUVQTVKQPU  6QYCTFU #FCRVKXG CPF 4QDWUV #54  +ORTQXKPI #FCRVKXG &GEKUKQP 4WNGU XKC &GEKUKQP 2CTCOGVGT #FCRVC VKQP  &GEKUKQP 2CTCOGVGT #FCRVCVKQP HQT 5VCVKQPCT[ 1RGTCVKPI %QP FKVKQPU  &GEKUKQP 2CTCOGVGT #FCRVCVKQP HQT 5NQYN[ %JCPIKPI 1RGT CVKPI %QPFKVKQPU  &GEKUKQP 2CTCOGVGT #FCRVCVKQP HQT 5YKVEJKPI 1RGTCVKPI %QP FKVKQPU  &KUEWUUKQP  4QDWUV &GEKUKQP 4WNGU  &GEKUKQP 4WNG 4QDWUVPGUU  /KPKOCZ %NCUUKſECVKQP 4WNG  $C[GUKCP 2TGFKEVKXG %NCUUKſECVKQP 4WNG  &KUEWUUKQP  5WOOCT[ 4 Speech Pattern Recognition using Neural Networks Shigeru Katagiri 066 %QOOWPKECVKQP 5EKGPEG .CDQTCVQTKGU  +PVTQFWEVKQP  $C[GU &GEKUKQP 6JGQT[  2TGRCTCVKQPU  &GEKUKQP 4WNG  /KPKOWO 'TTQTTCVG %NCUUKſECVKQP ‹E\&5&3UHVV//&



  

 2TQDCDKNKV[ (WPEVKQP 'UVKOCVKQP  &KUETKOKPCVKXG 6TCKPKPI 5RGGEJ 4GEQIPK\GTU $CUGF QP 0GWTCN 0GVYQTMU  2TGRCTCVKQPU  %NCUUKſECVKQP 'TTQT /KPKOK\CVKQP  5SWCTGF 'TTQT /KPKOK\CVKQP  %TQUU 'PVTQR[ /KPKOK\CVKQP (WUKQP QH /WNVKRNG %NCUUKſECVKQP &GEKUKQPU  2TKPEKRNGU  'ZCORNGU QH 'ODQFKOGPV %QPENWFKPI 4GOCTMU #RRGPFKZ /CZKOK\KPI /WVWCN +PHQTOCVKQP

5 Large Vocabulary Speech Recognition Based on Statistical Methods Jean-Luc Gauvain and Lori Lamel .+/5+ (TCPEG  +PVTQFWEVKQP  1XGTXKGY  .CPIWCIG /QFGNKPI  6GZV 2TGRCTCVKQP  8QECDWNCT[ 5GNGEVKQP  0ITCO 'UVKOCVKQP  ./ #FCRVCVKQP  2TQPWPEKCVKQP /QFGNKPI  #EQWUVKE /QFGNKPI  #EQWUVKE (TQPVGPF  /QFGNKPI #NNQRJQPGU  *// 2CTCOGVGT 'UVKOCVKQP  *// #FCRVCVKQP  &GEQFKPI  5RGGEJ0QPURGGEJ &GVGEVKQP  &GEQFKPI 5VTCVGIKGU  'HſEKGPE[  %QPſFGPEG /GCUWTGU  +PFKECVKXG 2GTHQTOCPEG .GXGNU  &KEVCVKQP  5RGGEJ 4GEQIPKVKQP HQT &KCNQI 5[UVGOU  6TCPUETKRVKQP HQT #WFKQ +PFGZCVKQP  2QTVCDKNKV[ CPF .CPIWCIG &GRGPFGPEKGU 6 Toward Spontaneous Speech Recognition and Understanding Sadaoki Furui 6QM[Q +PUVKVWVG QH 6GEJPQNQI[  +PVTQFWEVKQP  (QWT %CVGIQTKGU QH 5RGGEJ 4GEQIPKVKQP 6CUMU  5RQPVCPGQWU 5RGGEJ 4GEQIPKVKQP CPF 7PFGTUVCPFKPI  4GXKGY  %CVGIQT[ + JWOCPVQJWOCP FKCNQIWG ‹E\&5&3UHVV//&











 %CVGIQT[ ++ JWOCPVQJWOCP OQPQNQIWG  %CVGIQT[ +++ JWOCPVQOCEJKPG FKCNQIWG ,CRCPGUG 0CVKQPCN 2TQLGEV QP 5RQPVCPGQWU 5RGGEJ %QTRWU CPF 2TQ EGUUKPI 6GEJPQNQI[  2TQLGEV 1XGTXKGY  %QTRWU #WVQOCVKE 6TCPUETKRVKQP QH 5RQPVCPGQWU 2TGUGPVCVKQP  4GEQIPKVKQP 6CUM  .CPIWCIG CPF #EQWUVKE /QFGNKPI  4GEQIPKVKQP 4GUWNVU  #PCN[UKU QP +PFKXKFWCN &KHHGTGPEGU  &KUEWUUKQP #WVQOCVKE 5RGGEJ 5WOOCTK\CVKQP CPF 'XCNWCVKQP  5WOOCTK\CVKQP QH 'CEJ 5GPVGPEG 7VVGTCPEG  5WOOCTK\CVKQP QH /WNVKRNG 7VVGTCPEGU  'XCNWCVKQP  &KUEWUUKQP 5RQPVCPGQWU 5RGGEJ 4GEQIPKVKQP CPF 7PFGTUVCPFKPI 4GUGCTEJ +U UWGU  .CPIWCIG /QFGNU CPF %QTRQTC  /GUUCIGFTKXGP 5RGGEJ 4GEQIPKVKQP CPF 7PFGTUVCPFKPI  5VCVKUVKECN #RRTQCEJGU CPF 5RGGEJ 5EKGPEG  4GUGCTEJ QP VJG *WOCP $TCKP  &[PCOKE 5RGEVTCN (GCVWTGU %QPENWUKQP

7 Speaker Authentication Qi Li£ and Biing-Hwang Juang Ý £ $GNN .CDU Ý #XC[C .CDU 4GUGCTEJ  +PVTQFWEVKQP  5RGCMGT 4GEQIPKVKQP CPF 8GTKſECVKQP  8GTDCN +PHQTOCVKQP 8GTKſECVKQP  2CVVGTP 4GEQIPKVKQP KP 5RGCMGT #WVJGPVKECVKQP  $C[GUKCP &GEKUKQP 6JGQT[  5VQEJCUVKE /QFGNU HQT 5VCVKQPCT[ 2TQEGUU  5VQEJCUVKE /QFGNU HQT 0QP5VCVKQPCT[ 2TQEGUU  5RGGEJ 5GIOGPVCVKQP  5VCVKUVKECN 8GTKſECVKQP  5RGCMGT 8GTKſECVKQP 5[UVGO  8GTDCN +PHQTOCVKQP 8GTKſECVKQP  7VVGTCPEG 5GIOGPVCVKQP  5WDYQTF *[RQVJGUKU 6GUVKPI  %QPſFGPEG /GCUWTG %CNEWNCVKQP  5GSWGPVKCN 7VVGTCPEG 8GTKſECVKQP  8+8 'ZRGTKOGPVCN 4GUWNVU  5RGCMGT #WVJGPVKECVKQP D[ %QODKPKPI 58 CPF 8+8 ‹E\&5&3UHVV//&



5WOOCT[

8 HMMs for Language Processing Problems Richard M. Schwartz and John Makhoul $$0 6GEJPQNQIKGU 8GTK\QP  +PVTQFWEVKQP  7UG QH 2TQDCDKNKVKGU  *KFFGP /CTMQX /QFGNU  0COG 5RQVVKPI  6QRKE %NCUUKſECVKQP  6JG /QFGN  'UVKOCVKPI *// 2CTCOGVGTU  %NCUUKſECVKQP  'ZRGTKOGPVU  +PHQTOCVKQP 4GVTKGXCN  # $C[GUKCP /QFGN HQT +4  6TCKPKPI VJG +4 *//  2GTHQTOCPEG  'XGPV 6TCEMKPI  7PUWRGTXKUGF 6QRKE &GVGEVKQP  5WOOCT[ 9 Statistical Language Models With Embedded Latent Semantic Knowledge Jerome R. Bellegarda #RRNG %QORWVGT +PE  +PVTQFWEVKQP  5EQRG .QECNKV[  5[PVCEVKECNN[Ō&TKXGP 5RCP 'ZVGPUKQP  5GOCPVKECNN[Ō&TKXGP 5RCP 'ZVGPUKQP  1TICPK\CVKQP  .CVGPV 5GOCPVKE #PCN[UKU  (GCVWTG 'ZVTCEVKQP  5KPIWNCT 8CNWG &GEQORQUKVKQP  )GPGTCN $GJCXKQT  .5# (GCVWTG 5RCEG  9QTF %NWUVGTKPI  9QTF %NWUVGT 'ZCORNG  &QEWOGPV %NWUVGTKPI  &QEWOGPV %NWUVGT 'ZCORNG  5GOCPVKE %NCUUKſECVKQP  (TCOGYQTM 'ZVGPUKQP  5GOCPVKE +PHGTGPEG  %CXGCVU  0ITCO .5# .CPIWCIG /QFGNKPI  .5# %QORQPGPV  +PVGITCVKQP YKVJ 0ITCOU ‹E\&5&3UHVV//&





 

 %QPVGZV 5EQRG 5GNGEVKQP 5OQQVJKPI  9QTF 5OQQVJKPI  &QEWOGPV 5OQQVJKPI  ,QKPV 5OQQVJKPI 'ZRGTKOGPVU  'ZRGTKOGPVCN %QPFKVKQPU  'ZRGTKOGPVCN 4GUWNVU  %QPVGZV 5EQRG 5GNGEVKQP +PJGTGPV 6TCFG1HHU  %TQUU&QOCKP 6TCKPKPI  &KUEWUUKQP %QPENWUKQP

10 Semantic Information Processing of Spoken Language – How May I sm Help You? A. L. Gorin, A. Abella, T. Alonso, G. Riccardi, and J. H. Wright, #66 .CDQTCVQ TKGU

    

+PVTQFWEVKQP %CNN%NCUUKſECVKQP .CPIWCIG /QFGNKPI HQT 4GEQIPKVKQP CPF 7PFGTUVCPFKPI &KCNQI %QPENWUKQPU

11 Machine Translation Using Statistical Modeling Herman Ney, and F. J. Och #CEJGP 7PKXGTUKV[ QH 6GEJPQNQI[ )GTOCP[  +PVTQFWEVKQP  5VCVKUVKECN &GEKUKQP 6JGQT[ CPF .KPIWKUVKEU  6JG 5VCVKUVKECN #RRTQCEJ  $C[GU &GEKUKQP 4WNG HQT 9TKVVGP .CPIWCIG 6TCPUNCVKQP  4GNCVGF #RRTQCEJGU  #NKIPOGPV CPF .GZKEQP /QFGNU  %QPEGRV QH #NKIPOGPV /QFGNNKPI  *KFFGP /CTMQX /QFGNU  /QFGNU +$/ Ō  6TCKPKPI  5GCTEJ  #NIQTKVJOKE &KHHGTGPEGU DGVYGGP 5RGGEJ 4GEQIPKVKQP CPF .CPIWCIG 6TCPUNCVKQP  #NKIPOGPV 6GORNCVGU (TQO 5KPING 9QTFU VQ 9QTF )TQWRU  %QPEGRV  6TCKPKPI  5GCTEJ  'ZRGTKOGPVCN 4GUWNVU  6JG 6CUM CPF VJG %QTRWU ‹E\&5&3UHVV//&

 1HƀKPG 4GUWNVU  +PVGITCVKQP KPVQ VJG 8 '4$/1$+. 2TQVQV[RG 5[UVGO  (KPCN 'XCNWCVKQP  5RGGEJ 6TCPUNCVKQP 6JG +PVGITCVGF #RRTQCEJ  2TKPEKRNG  2TCEVKECN +ORNGOGPVCVKQP  5WOOCT[  4GHGTGPEGU 12 Modeling Topics for Detection and Tracking James Allan 7PKXGTUKV[ QH /CUUCEJWUGVVU #OJGTUV  6QRKE &GVGEVKQP CPF 6TCEMKPI  6QRKE CPF 'XGPVU  6&6 6CUMU  %QTRQTC  'XCNWCVKQP  $CUKE 6QRKE /QFGNU  8GEVQT 5RCEG  .CPIWCIG /QFGNU  +ORNGOGPVKPI VJG /QFGNU  0COGF 'PVKVKGU  &QEWOGPV 'ZRCPUKQP  %NWUVGTKPI  6KOG &GEC[  %QORCTKPI /QFGNU  0GCTGUV 0GKIJDQTU  &GEKUKQP 6TGGU  /QFGNVQ/QFGN  /KUEGNNCPGQWU +UUWGU  &GHGTTCN  /WNVKOQFCN +UUWGU  /WNVKNKPIWCN +UUWGU  7UKPI 6&6 +PVGTCEVKXGN[  &GOQPUVTCVKQPU  6KOGNKPGU  /QFGNKPI 'XGPVU  %QPENWUKQP

‹E\&5&3UHVV//&

1 Minimum Classification Error (MCE) Approach in Pattern Recognition Wu Chou Avaya Labs Research, Avaya Inc., USA

CONTENTS        

+PVTQFWEVKQP 1RVKOCN %NCUUKſGT HTQO $C[GU &GEKUKQP 6JGQT[ &KUETKOKPCPV (WPEVKQP #RRTQCEJ VQ %NCUUKſGT &GUKIP 5RGGEJ 4GEQIPKVKQP CPF *KFFGP /CTMQX /QFGNKPI /%' %NCUUKſGT &GUKIP 7UKPI &KUETKOKPCPV (WPEVKQPU 'ODGFFGF 5VTKPI /QFGN $CUGF /%' 6TCKPKPI 8GTKſECVKQP CPF +FGPVKſECVKQP 5WOOCT[ #EMPQYNGFIGOGPV 4GHGTGPEGU

1.1 Introduction 2CVVGTP TGEQIPKVKQP KU C HCUV OQXKPI TGUGCTEJ CTGC 6JG CFXGPV QH RQYGTHWN EQORWV KPI FGXKEGU CPF VJG UWEEGUU QH UVCVKUVKECN CRRTQCEJGU UWEJ CU JKFFGP /CTMQX OQFGN HQT URGGEJ CPF NCPIWCIG RTQEGUUKPI VTKIIGTGF C TGPGYGF RWTUWKV HQT OQTG RQYGTHWN UVCVKUVKECN OGVJQFU VQ HWTVJGT TGFWEG VJG RCVVGTP TGEQIPKVKQP GTTQT TCVG CPF KORTQXG VJG TQDWUVPGUU QH VJG RCVVGTP ENCUUKſGT CETQUU XCTKQWU CFXGTUG EQPFKVKQPU #OQPI VJKU PGY RWTUWKV VJG WUG QH FKUETKOKPCPV HWPEVKQP OGVJQFU KP RCVVGTP TGEQIPKVKQP JCU GOGTIGF CU C RTQOKUKPI CRRTQCEJ CPF KV KU CRRNKGF UWEEGUUHWNN[ VQ URGGEJ CPF NCPIWCIG RTQEGUUKPI 6JKU EJCRVGT KU KPVGPFGF VQ RTQXKFG C TGXKUKV VQ VJG UVCVKUVKECN HQTOWNCVKQP QH VJG OKPKOWO ENCUUKſECVKQP GTTQT /%' DCUGF FKUETKOKPCVKXG OGVJ QFU KP URGGEJ CPF NCPIWCIG RTQEGUUKPI VCMG C ETKVKECN XKGY QH VJG CRRTQCEJ RTQXKFG C EQORTGJGPUKXG QXGTXKGY QH VJG ſGNF CPF JQRGHWNN[ KPURKTG QVJGT KPPQXCVKQPU VJCV YQWNF RQVGPVKCNN[ NGCF VQ PGY FKUETKOKPCVKXG OGVJQFU KP RCVVGTP TGEQIPKVKQP #NVJQWIJ VJG UVCVKUVKECN HQTOWNCVKQP QH /%' DCUGF FKUETKOKPCVKXG OGVJQFU JCU KVU TQQV KP VJG ENCUUKECN $C[GU FGEKUKQP VJGQT[ KV FGRCTVU HTQO VJG EQPXGPVKQPCN RCTCFKIO

­

6JKU EJCRVGT KU FGXGNQRGF DCUGF QP Œ&KUETKOKPCPVHWPEVKQPDCUGF OKPKOWO TGEQIPKVKQP GTTQT TCVG RCVVGTPTGEQIPKVKQP CRRTQCEJ VQ URGGEJ TGEQIPKVKQPŒ D[ 9W %JQW CRRGCTGF KP Proceedings of The IEEE 8QN  0Q  E  +'''

‹E\&5&3UHVV//&

YJKEJ NKPMU C TGEQIPKVKQP VCUM VQ VJG RTQDNGO QH FKUVTKDWVKQP GUVKOCVKQP +PUVGCF KV VCMGU C FKUETKOKPCPV HWPEVKQP DCUGF UVCVKUVKECN RCVVGTP ENCUUKſECVKQP CRRTQCEJ CPF HQT C IKXGP HCOKN[ QH FKUETKOKPCPV HWPEVKQP QRVKOCN ENCUUKſGTTGEQIPK\GT FGUKIP KPXQNXGU ſPFKPI C UGV QH RCTCOGVGTU YJKEJ OKPKOK\G VJG GORKTKECN RCVVGTP TGEQIPKVKQP GTTQT TCVG 6JG WUG QH FKUETKOKPCPV HWPEVKQP KP RCVVGTP TGEQIPKVKQP YCU UVCTVGF OCP[ [GCTU CIQ 1PG ENCUUKECN GZCORNG QH WUKPI FKUETKOKPCPV HWPEVKQP HQT ENCUUKſGT FGUKIP KP UVCVKUVKECN NKVGTCVWTG KU VJG VYQ ENCUU ENCUUKſECVKQP RTQDNGO WUKPI NKPGCT FKUETKOKPCPV HWPEVKQPU = ? +P RCTVKEWNCT C YKPFQY DCUGF OGVJQF YCU FGUETKDGF KP =? HQT VJG VYQ ENCUU ENCUUKſECVKQP RTQDNGO WUKPI NKPGCT FKUETKOKPCPV HWPEVKQPU VJCV OKPKOK\G VJG RTQDCDKNKV[ QH ENCUUKſECVKQP GTTQT TCVG 6JG HQEWU QH VJKU EJCRVGT KU QP VJG TGEGPV FGXGNQROGPV QH VJG IGPGTCN /%' DCUGF FKUETKOKPCVKXG OGVJQFU 6JG FKUETKOKPCPV HWPEVKQPU VJCV YG GPEQWPVGT CTG WUWCNN[ PQPNKPGCT CPF QHVGP TGNCVGF VQ VJG UVTWEVWTG QH VJG UVCVKUVKECN HTCOGYQTM WUGF KP URGGEJ CPF NCPIWCIG RTQEGUUKPI UWEJ CU JKFFGP /CTMQX OQFGNU 6JG TGCUQP QH VCMKPI C FKUETKOKPCPV HWPEVKQP DCUGF CRRTQCEJ VQ ENCUUKſGT FGUKIP CU YKNN DG HWTVJGT GNCDQTCVGF KU FWG OCKPN[ VQ VJG HCEV VJCV YG NCEM EQORNGVG MPQYNGFIG QH VJG HQTO QH VJG FCVC FKUVTKDWVKQP CPF VTCKPKPI FCVC CTG KPCFGSWCVG RCTVKEWNCTN[ KP FGCNKPI YKVJ URGGEJ CPF NCPIWCIG RTQDNGOU 6JG RGTHQTOCPEG QH C TGEQIPK\GT KU PQTOCNN[ FGſPGF D[ KVU GZRGEVGF TGEQIPKVKQP GTTQT TCVG CPF CP QRVKOCN TGEQIPK\GT UJQWNF DG VJG QPG VJCV CEJKGXGU VJG NGCUV GZRGEVGF TCVG QH TGEQIPKVKQP GTTQT 6JG FKHHGTGPEG DGVYGGP VJG FKUVTKDWVKQP GUVKOCVKQP DCUGF CRRTQCEJ CPF VJG FKUETKOKPCPV HWPEVKQP DCUGF /%' CRRTQCEJ NKGU KP VJG YC[ VJG TGEQIPKVKQP GTTQT KU GZRTGUUGF CPF KP VJG EQORWVCVKQPCN UVGRU VJCV YQWNF NGCF VQ VJG OKPKOK\CVKQP QH UWEJ GTTQT HWPEVKQPU # MG[ VQ VJG FGXGNQROGPV QH VJG /%' OGVJQF KU C PGY GTTQT HWPEVKQP YJKEJ KPEQTRQTCVGU VJG TGEQIPKVKQP QRGTCVKQP CPF RGTHQTOCPEG KP C HWPEVKQPCN HQTO HTQO YJKEJ VJG RGTHQTOCPEG QH VJG ENCUUKſGT ECP DG FKTGEVN[ GXCNWCVGF CPF QRVKOK\GF %NCUUKſGT FGUKIP YKVJQWV CUUWOKPI VJG MPQYNGFIG QH ENCUU RQUVGTKQT RTQDCDKNKVKGU YJKEJ CTG VJG DCUKU QH VJG FKUVTKDWVKQP GUVKOCVKQP DCUGF ENCUUKGT FGUKIP JCU DGGP UVWFKGF KP OCP[ CTGCU +P RCTVKEWNCT 6U[RMKP =? CPF #OCTK = ? RKQPGGTGF VJKU CRRTQCEJ HQT UGNHNGCTPKPI CPF UGNHQTICPK\KPI PGVU 6JG[ HQTOWNCVGF VJG RTQDNGO QH UGNHNGCTPKPI KPVQ C ENCUUKſECVKQP RTQDNGO YJKEJ EQPUKUVU QH QRVKOCN RCTVKVKQPKPI QH VJG QDUGTXCVKQP URCEG KPVQ TGIKQPU
 HQT YJKEJ VJG GZRGEVGF TKUM  KU OKPKOK\GF +P CFFKVKQP C OCVJGOCVKECN OKPKOK\CVKQP RTQEGFWTG IGPGTCNK\GF RTQDCDKNKUVKE FG UEGPV )2& CNIQTKVJO QT UVQEJCUVKE CRRTQZKOCVKQP YCU RTQRQUGF CU C OGCPU HQT ENCUUKſGT FGUKIP WPFGT VJKU HTCOGYQTM 5KPEG VJGP XCTKQWU NQUU HWPEVKQPU JCXG DGGP WUGF KP FGUKIPKPI ENCUUKſGTU KPENWFKPI VJQUG RQRWNCT OGCPUSWCTG GTTQT DCUGF NQUU HWPEVKQPU *QYGXGT OCP[ VTCEVCDNG NQUU HWPEVKQPU FQ PQV JCXG C FKTGEV TGNCVKQP VQ VJG TGEQIPKVKQP GTTQT TCVG OKPKOK\CVKQP CPF VJGTGHQTG CNDGKV DCUGF QP FKUETKOKPCPV HWPEVKQPU VJG[ CTG PQV FKTGEVN[ TGNCVGF VQ TGEQIPKVKQP GTTQT TCVG YJKEJ UJQWNF DG VJG OQUV UGPUKDNG EJQKEG HQT ENCUUKſGT FGUKIP 1XGT VJG RCUV FGECFG VJG /%' DCUGF CRRTQCEJ JCU DGGP FGXGNQRGF VQ QXGTEQOG VJG HWPFCOGPVCN NKOKVCVKQPU QH VJG VTCFKVKQPCN CRRTQCEJ CPF VQ FKTGEVN[ NKPM VJG ENCUUKſGT FGUKIP RTQDNGO VQ ENCUUKſECVKQP GTTQT TCVG OKPKOK\CVKQP +P QTFGT VQ CNNGXKCVG VJG FGRGPFGPE[ QP VJG ENCUU RQUVGTKQT FKUVTKDWVKQPU C FKUETKOKPCPV HWPEVKQP DCUGF /%' CRRTQCEJ YCU RTQRQUGF D[ ,WCPI GV CN =? CU CP CNVGTPCVKXG VQ QRVKOCN ENCUUKſGT FG




‹E\&5&3UHVV//&



UKIP #NVJQWIJ VJKU CRRTQCEJ CRRNKGU VQ VJG RCVVGTP TGEQIPKVKQP RTQDNGO KP IGPGTCN KV ſPFU XCTKQWU CRRNKECVKQPU KP URGGEJ CPF NCPIWCIG RTQEGUUKPI +V YCU ſTUV CRRNKGF VQ F[PCOKE VKOG YCTRKPI DCUGF RCVVGTP TGEQIPKVKQP U[UVGOU = ? #RRNKECVKQP VQ JKFFGP /CTMQX OQFGN DCUGF EQPVKPWQWU URGGEJ TGEQIPKVKQP U[UVGOU YCU HQTOWNCVGF CU C UGIOGPVCN CPF UVTKPI OQFGN DCUGF /%' CRRTQCEJ = ? CPF UWEEGUUHWN CR RNKECVKQPU QH VJKU CRRTQCEJ YGTG TGRQTVGF KP =     ? 6JKU CRRTQCEJ YCU HWTVJGT GZVGPFGF VQ HQTO C EQODKPGF UVTKPI OQFGN KP YJKEJ VTCKPKPI QH QVJGT OQFGN EQORQPGPVU KP URGGEJ CPF NCPIWCIG RTQEGUUKPI ECP DG CEJKGXGF WPFGT C WPK ſGF /%' HTCOGYQTM = ? +V YCU CRRNKGF VQ FKUETKOKPCVKXG OQFGN EQODKPCVKQP = ? CPF VQ CRRNKECVKQPU KP URGCMGT KFGPVKſECVKQP CPF XGTKſECVKQP =  ? 6JG DCUKE KFGC QH VJG /%' CRRTQCEJ YCU HWTVJGT FGXGNQRGF HQT CRRNKECVKQPU KP WVVGT CPEG XGTKſECVKQP RTQDNGOU =  ? # IGPGTCN HTCOGYQTM QH EQODKPKPI FG VGEVKQP CPF XGTKſECVKQP KP URGGEJ TGEQIPKVKQP CPF WPFGTUVCPFKPI YCU CNUQ RTQRQUGF KP YJKEJ VJG FKUETKOKPCPV HWPEVKQP DCUGF RCVVGTP TGEQIPKVKQP CRRTQCEJ YCU CRRNKGF KP DQVJ FGVGEVKQP CPF XGTKſECVKQP RTQEGUUGU = ? 9G DGIKP KP VJG PGZV UGEVKQP YKVJ C DTKGH TGXKGY QH VJG $C[GU FGEKUKQP VJGQT[ CPF KVU CRRNKECVKQP VQ VJG HQTOWNCVKQP QH UVCVKUVKECN RCVVGTP TGEQIPKVKQP RTQDNGO 9G KP VTQFWEG VJG FKUETKOKPCPV HWPEVKQP DCUGF UVCVKUVKECN RCVVGTP TGEQIPKVKQP CRRTQCEJ KP 5GEVKQP  +P 5GEVKQP  YG RTQXKFG C DTKGH KPVTQFWEVKQP VQ URGGEJ TGEQIPKVKQP CPF JKFFGP /CTMQX OQFGNKPI 6JG FKUETKOKPCPV HWPEVKQP DCUGF /%' RCVVGTP TGEQIPKVKQP CRRTQCEJ CPF KVU CRRNKECVKQP VQ *// DCUGF URGGEJ TGEQIPKVKQP U[UVGOU CTG KPVTQ FWEGF KP 5GEVKQP  %QORCTKUQPU CTG OCFG VQ QVJGT ETKVGTKC KP URGGEJ TGEQIPKVKQP CPF KP RCTVKEWNCT YG UVWF[ VJG TGNCVKQP DGVYGGP /%' CPF //+ OCZKOWO OWVWCN KPHQTOCVKQP ETKVGTKC KP ENCUUKſGT FGUKIP KP VJG UGEQPF JCNH QH 5GEVKQP  +P 5GEVKQP  YG UVWF[ VJG GODGFFGF UVTKPI OQFGN DCUGF /%' CRRTQCEJ CPF KVU GZVGPUKQP VQ VJG JKIJGT NGXGN EQODKPGF UVTKPI OQFGN 9G FKUEWUU KUUWGU CPF CRRNKECVKQPU KP FKUETKO KPCVKXG OQFGN EQODKPCVKQP FKUETKOKPCVKXG NCPIWCIG OQFGN GUVKOCVKQP CPF FKUETKO KPCVKXG HGCVWTG GZVTCEVKQP WPFGT VJG IGPGTCN VJGQTGVKECN HTCOGYQTM QH VJG EQODKPGF UVTKPI OQFGN 5GEVKQP  KU FGXQVGF VQ CRRNKECVKQPU QH FKUETKOKPCPV HWPEVKQP DCUGF RCVVGTP TGEQIPKVKQP CRRTQCEJ KP XGTKſECVKQP CPF KFGPVKſECVKQP 6JG FKUETKOKPCPV HWPEVKQP CRRTQCEJ KU UVWFKGF HQT XCTKQWU CRRNKECVKQPU KP URGGEJ CPF NCPIWCIG RTQ EGUUKPI UWEJ CU URGCMGT KFGPVKſECVKQP CPF XGTKſECVKQP WVVGTCPEG XGTKſECVKQP TGEQI PKVKQP DCUGF QP IGPGTCNK\GF EQPſFGPEG OGCUWTGU FGVGEVKQP CPF XGTKſECVKQP DCUGF CRRTQCEJ KP URGGEJ TGEQIPKVKQP CPF WPFGTUVCPFKPI 6JG EJCRVGT KU UWOOCTK\GF YKVJ FKUEWUUKQPU KP 5GEVKQP 

1.2 Optimal Classifier from Bayes Decision Theory




(QT CP ENCUU ENCUUKſECVKQP RTQDNGO C ENCUUKſGT KU VQ ENCUUKH[ GCEJ TCPFQO UCORNG  6JG Z KPVQ QPG QH VJG / ENCUUGU 9G FGPQVG VJGUG ENCUUGU D[

  ENCUUKſGT   FGſPGU C OCRRKPI HTQO VJG UCORNG URCEG ¾ VQ VJG FKUETGVG



‹E\&5&3UHVV//&

   
 

ECVGIQTKECN UGV

¾   .GV 


 DG VJG LQKPV RTQDCDKNKV[ FKUVTKDWVKQP QH  CPF

 C SWCPVKV[ YJKEJ KU CUUWOGF VQ DG MPQYP VQ VJG FGUKIPGT QH VJG ENCUUKſGT +P QVJGT YQTFU VJG FGUKIPGT JCU HWNN MPQYNGFIG QH VJG TCPFQO PCVWTG QH VJG UQWTEG (TQO VJG UGV QH LQKPV RTQDCDKNKV[ FKUVTKDWVKQPU VJG OCTIKPCN CPF VJG EQPFKVKQPCN RTQDCDKNKV[ FKUVTKDWVKQPU ECP DG GCUKN[ ECNEWNCVGF +P QTFGT VQ EJCTCEVGTK\G VJG RGTHQTOCPEG QH VJG ENCUUKſGT GXGT[ ENCUU RCKT
  ECP DG CUUQEKCVGF YKVJ C EQUV QT NQUU HWPEVKQP 
 YJKEJ UKIPKſGU VJG EQUV QH ENCUUKH[KPI QT TGEQIPK\KPI C ENCUU  QDUGTXCVKQP KPVQ C ENCUU  GXGPV 6JG NQUU HWPEVKQP KU IGPGTCNN[ PQPPGICVKXG YKVJ     TGRTGUGPVKPI EQTTGEV ENCUUKſECVKQP 6JG NQUU HWPEVKQP KU C HWPEVKQP HTQO
  YJGTG KU VJG UGV QH TGCN PWODGTU +P ENCUUKſECVKQP YG OCMG C FGEKUKQP

 HQT QDUGTXKPI C TCPFQO UCORNG  5KPEG 


  KU VJG ENCUU RQUVGTKQT RTQDCDKNKV[ VJCV VJG TCPFQO KPRWV  KU HTQO

 VJG CXGTCIG NQUU CUUQEKCVGF YKVJ OCMKPI C FGEKUKQP

 
 ECP DG FGſPGF CU =?



   


 
   




















6JKU NGCFU VQ C TGCUQPCDNG RGTHQTOCPEG OGCUWTG HQT VJG ENCUUKſGT KG VJG GZRGEVGF NQUU FGſPGF CU

 


   




YJGTG

 TGRTGUGPVU VJG ENCUUKſGTŏU FGEKUKQP CUUWOKPI QPG QH VJG őXCNWGUŒ
 
     
  DCUGF QP C TCPFQO QDUGTXCVKQP  FTCYP HTQO C RTQDCDKNKV[ FKUVTKDWVKQP 
 6JG FGEKUKQP HWPEVKQP

 FGRGPFU QP VJG ENCUUKſGT FGUKIP 1DXKQWUN[ KH VJG ENCUUKſGT KU UQ FGUKIPGF VJCV HQT GXGT[ 




    



   

  VJG GZRGEVGF NQUU KP GSWCVKQP  YKNN DG OKPKOK\GF (QT OCP[ CRRNKECVKQPU KPENWFKPI URGGEJ TGEQIPKVKQP VJG NQUU HWPEVKQP  
KU WUWCNN[ EJQUGP VQ DG VJG \GTQQPG NQUU HWPEVKQP FGſPGF D[




 

             







YJKEJ CUUKIPU PQ NQUU VQ EQTTGEV ENCUUKſECVKQP CPF C WPKV NQUU VQ CP[ GTTQT TGICTFNGUU QH VJG ENCUU 9KVJ VJKU V[RG QH NQUU HWPEVKQP VJG GZRGEVGF NQUU  KU VJWU VJG GTTQT RTQDCDKNKV[ QH ENCUUKſECVKQP QT TGEQIPKVKQP 6JG EQPFKVKQPCN NQUU DGEQOGU



   



 








 

  







6JG QRVKOCN ENCUUKſGT VJCV CEJKGXGU OKPKOWO  KU VJWU VJG QPG VJCV KORNGOGPVU VJG HQNNQYKPI

 
 KH 

      


 




‹E\&5&3UHVV//&

(QT OKPKOWO GTTQT TCVG ENCUUKſECVKQP VJG ENCUUKſGT GORNQ[U VJG FGEKUKQP TWNG QH  YJKEJ KU ECNNGF VJG őOCZKOWO C RQUVGTKQTŒ /#2 FGEKUKQP 6JG OKPKOWO GTTQT TCVG CEJKGXGF D[ /#2 FGEKUKQP KU ECNNGF ő$C[GU TKUMŒ 9JGP CNN RQUVGTKQT RTQDCDKNKVKGU CTG MPQYP VJG ENCUUKſGT DCUGF QP /#2 TWNG KU CP QRVKOCN ENCUUKſGT DCUGF QP VJG $C[GU FGEKUKQP VJGQT[ *QYGXGT KH VJGUG RTQDCDKNKVKGU CTG PQV MPQYP QT VJG FGEKUKQP TWNG KU PQV DCUGF QP VJG ENCUU RQUVGTKQT RTQDCDKNKV[ VJGP YG ECPPQV WUG VJKU TGUWNV FKTGEVN[ +P RTCEVKEG VJGUG RTQDCDKNKVKGU JCXG VQ DG GUVKOCVGF HTQO C VTCKPKPI FCVC UGV YKVJ MPQYP ENCUU NCDGNU 6JG ENCUUKECN $C[GU FGEKUKQP VJGQT[ VJWU GHHGEVKXGN[ VTCPUHQTOU VJG ENCUUKſGT FGUKIP RTQDNGO KPVQ C FKUVTKDWVKQP GUVKOCVKQP RTQDNGO 6JKU KU VJG DCUKU QH VJG $C[GUKCP UVCVKUVKECN CRRTQCEJ VQ RCVVGTP TGEQIPKVKQP YJKEJ ECP DG UVCVGF CU IKXGP QT EQNNGEV C UGV QH VTCKPKPI FCVC QDUGTXCVKQPU

½ 
¾     

 YKVJ MPQYP ENCUU NCDGNU GUVKOCVG VJG C RQUVGTKQT RTQDCDKNKVKGU 
  
          HQT CP[
VQ KORNGOGPV VJG OCZKOWO C RQUVGTKQT FGEKUKQP HQT OKPKOWO $C[GU TKUM 6JG C RQUVGTKQT RTQDCDKNKV[ 
  
 ECP DG TGYTKVVGP CU


 
  

  
 

 



5KPEG 

 KU PQV C HWPEVKQP QH VJG ENCUU KPFGZ CPF VJWU JCU PQ GHHGEV KP VJG /#2 FGEKUKQP VJG PGGFGF RTQDCDKNKUVKE MPQYNGFIG ECP DG TGRTGUGPVGF D[ VJG ENCUU RTKQT 
  CPF VJG EQPFKVKQPCN RTQDCDKNKV[ 

    6JGTG CTG UGXGTCN KUUWGU CUUQEKCVGF YKVJ VJKU ENCUUKECN CRRTQCEJ (KTUV VJG FKUVTKDW VKQPU WUWCNN[ JCXG VQ DG RCTCOGVGTK\GF KP QTFGT HQT VJGO VQ DG RTCEVKECNN[ WUGHWN HQT VJG KORNGOGPVCVKQP QH VJG /#2 TWNG QH  6JG ENCUUKſGT FGUKIPGT VJGTGHQTG JCU VQ FGVGTOKPG VJG TKIJV RCTCOGVTKE HQTO QH VJG FKUVTKDWVKQPU (QT OQUV QH VJG TGCN YQTNF RTQDNGOU VJKU KU C FKHſEWNV VCUM 1WT EJQKEG QH VJG FKUVTKDWVKQP HQTO KU QHVGP NKO KVGF D[ VJG OCVJGOCVKECN VTCEVCDKNKV[ QH VJG RCTVKEWNCT FKUVTKDWVKQP HWPEVKQPU CPF KU XGT[ NKMGN[ VQ DG KPEQPUKUVGPV YKVJ VJG CEVWCN FKUVTKDWVKQP 6JKU OGCPU VJCV VJG VTWG /#2 FGEKUKQP ECP TCTGN[ DG KORNGOGPVGF CPF VJG OKPKOWO $C[GU TKUM IGPGTCNN[ TGOCKPU CP WPCEJKGXCDNG NQYGT DQWPF 5GEQPF IKXGP C RCTCOGVGTK\GF FKUVTKDWVKQP HQTO VJG WPMPQYP RCTCOGVGTU FGſPKPI VJG FKUVTKDWVKQP JCXG VQ DG GUVKOCVGF HTQO C ſPKVG COQWPV QH NCDGNGF VTCKPKPI FCVC TGSWKTKPI VJCV VJG GUVKOCVKQP OGVJQF JCU VQ DG CDNG VQ RTQFWEG EQPUKUVGPV RCTCOGVGT XCNWGU YJGP VJG UK\G QH VJG VTCKPKPI UCORNGU XCTKGU 6JKTF KV TGSWKTGU C VTCKPKPI FCVC UGV QH UWHſEKGPV UK\G KP QTFGT VQ JCXG TGNK CDNG RCTCOGVGT GUVKOCVGU $WV KP RTCEVKEG CPF HQT URGGEJ CPF NCPIWCIG RTQEGUUKPI KP RCTVKEWNCT VTCKPKPI FCVC CTG CNYC[U URCTUG EQORCTGF VQ CNN RQUUKDNG TGCNK\CVKQPU CPF XCTKCVKQPU KP JWOCP URGGEJ CPF NCPIWCIG 6JGUG VJTGG DCUKE KUUWGU RQKPV QWV C HWPFCOGPVCN HCEV VJCV KU FGURKVG VJG EQPEGRVWCN QRVKOCNKV[ QH VJG $C[GU FGEKUKQP VJGQT[ CPF KVU CRRNKECVKQPU VQ RCVVGTP TGEQIPKVKQP KV ECPPQV CNYC[U DG CEEQORNKUJGF KP RTCEVKEG DGECWUG OQUV RTCEVKECN ő/#2Œ FGEKUKQPU KP URGGEJ CPF NCPIWCIG RTQ EGUUKPI CTG PQV VTWG /#2 FGEKUKQPU 6JKU WPFGTUVCPFKPI KU ETKVKECN HQT VJG FKUEWUUKQP VJCV HQNNQYU

‹E\&5&3UHVV//&

1.3 Discriminant Function Approach to Classifier Design &KUETKOKPCPV HWPEVKQPU QP VJG QVJGT JCPF CTG VJQUG HWPEVKQPU YJKEJ EJCTCEVGTK\G VJG FGEKUKQP TWNG QH VJG ENCUUKſGT 6JG[ OC[ QT OC[ PQV DG RTQDCDKNKV[ QT NKMGNKJQQF DCUGF HWPEVKQPU CPF VJG[ ECP EQOG HTQO FKHHGTGPV RCTCOGVTKE HCOKNKGU KPENWFKPI VJQUG HCOKNKGU YJKEJ JCXG PQ TGNCVKQP VQ VJG RCTCOGVTKE HQTO QH VJG ENCUU RQUVGTKQT FKUVTKDWVKQP



 CU TGSWKTGF KP VJG ENCUUKECN $C[GU FGEKUKQP VJGQT[ 1PG YGNN UVWFKGF HCOKN[ QH FKUETKOKPCPV HWPEVKQP KU VJG NKPGCT FKUETKOKPCPV HWPEVKQP YJKEJ JCU EQORWVCVKQPCN CFXCPVCIGU CPF FWG VQ KVU CPCN[VKE HQTO JCU TGEGKXGF EQPUKFGTCDNG CVVGPVKQP CPF VJGQTGVKECN FGXGNQROGPV HQT KVU FGUKIP 6Q KNNWUVTCVG VJG EQPEGRV YG EQPUKFGT VJG ECUG QH C VYQ ENCUU  ½  ¾  ENCUUKſECVKQP RTQDNGO 6JG ENCUUKſGT WUGU C FKUETKOKPCPV HWPEVKQP 
 UWEJ VJCV




KH KH


   VJGP Z KU ENCUUKſGF VQ ½ 
   VJGP Z KU ENCUUKſGF VQ ¾



.KPGCT FKUETKOKPCPV HWPEVKQPU CTG VJQUG HWPEVKQPU QH VJG HQTO


  
  ¼ YJGTG 




  ¾        CPF ¼ C TGCN PWODGT 1T OQTG IGPGTCNN[

 ½


  ¼  ½ ½
   
  





YJGTG





  ½      ¼  

  ½         ¼



YJGTG KU VJG VTCPURQUKVKQP PQVCVKQP CPF VJG  CTG MPQYP NKPGCTN[ KPFGRGPFGPV HWPEVKQPU QH  (QT / ENCUU ENCUUKſECVKQP RTQDNGO WUKPI FKUETKOKPCPV HWPEVKQPU C UGV QH FKUETKOKPCPV HWPEVKQPU 

      CTG WUGF CPF VJG ENCUUKſGT 
 KU FGſPGF UWEJ VJCV 
   KHH     


 

9JGP VJG NQUU HWPEVKQP 


 KU URGEKſGF VJG RTQDNGO QH QRVKOCN ENCUUKſGT FGUKIP WUKPI FKUETKOKPCPV HWPEVKQPU DGEQOGU C OKPKOK\CVKQP RTQDNGO QH ſPFKPI C DGUV UGV QH FKUETKOKPCPV HWPEVKQPU  

      HTQO C ENCUU QH FKUETKOK PCPV HWPEVKQPU YJKEJ OKPKOK\GU VJG GZRGEVGF NQUU  CU FGſPGF KP 'S  +P QVJGT YQTFU VJG ENCUUKſGT FGUKIP RTQDNGO KU VQ ſPF

 
 
       



‹E\&5&3UHVV//&



  
´µ
 ´  µ













YJGTG

  KU VJG IKXGP HCOKN[ QH FKUETKOKPCPV HWPEVKQPU +H VJG NQUU HWPEVKQP KU IKXGP CU KP 'S  CPF VJG RCTVKEWNCT UGV QH FKUETKOKPCPV HWPEVKQPU WUGF KP VJG ENCUUKſGT CTG VJG őVTWGŒ ENCUU RQUVGTKQT RTQDCDKNKV[

  VJGP 'S  KORNG OGPVU VJG UCOG /#2 FGEKUKQP TWNG CU FGſPGF KP 'S  *QYGXGT KV KU KORQTVCPV VQ RQKPV QWV VJCV VJG FKUETKOKPCPV HWPEVKQP CRRTQCEJ VQ VJG QRVKOCN ENCUUKſGT FGUKIP CU URGEKſGF KP 'S  QHVGP JCU CP KPſPKVG PWODGT QH UQNWVKQPU GXGP HQT VJG UCOG  KU CP QRVKOCN UQNWVKQP VQ 'S ENCUUKſGT +V KU GCU[ VQ UGG VJCV KH  

   

 VJGP HQT CP[ 
      

      KU CPQVJGT QRVKOCN UQNWVKQP CPF FGſPGU VJG UCOG ENCUUKſGT #ICKP VJKU KU SWKVG FKHHGTGPV HTQO VJG FKUVTKDWVKQP GUVKOCVKQP DCUGF CRRTQCEJ KP RCVVGTP ENCUUKſECVKQP +H VJG FKUETKOK PCPV HWPEVKQPU CTG NKOKVGF VQ VJG ENCUU RQUVGTKQT RTQDCDKNKVKGU CP[ FGXKCVKQP HTQO VJG őVTWGŒ ENCUU RQUVGTKQT RTQDCDKNKV[

  YKNN TGUWNV KP C FKHHGTGPV ENCUUKſGT CPF YKNN DG KPHGTKQT VQ VJG QRVKOCN /#2 ENCUUKſGT VJCV CEJKGXGU VJG OKPKOWO $C[GU TKUM

 

    

      

 

 

6JG WUG QH FKUETKOKPCPV HWPEVKQPU KP UVCVKUVKECN RCVVGTP TGEQIPKVKQP KU VQ UQNXG VJG ENCUUKſGT FGUKIP RTQDNGO YJGP VJG GZCEV HQTO CPF XCNWG QH VJG ENCUU RQUVGTKQT RTQDC DKNKVKGU

  CTG PQV MPQYP GXGP YKVJ VJG JGNR QH VTCKPKPI FCVC QT VJG ENCUUKſGT JCU VQ DG DCUGF QP C RCTVKEWNCT ENCUU QH FKUETKOKPCPV HWPEVKQPU 6JGUG FKUETKOKPCPV HWPEVKQPU KP VJG ENCUUKſGT OC[ EQOG HTQO GKVJGT VJG OQFGN WUGF VQ EJCTCEVGTK\G VJG IGPGTCVKQP RTQEGUU QH VJG TGEQIPKVKQP QDLGEVU QT VJG RTCEVKECN EQPUKFGTCVKQP QH OCVJ GOCVKECN VTCEVCDKNKV[ CPF CNIQTKVJOKE EQORNGZKV[ %NCUUKH[KPI JWOCP URGGEJ OGGVU DQVJ UEGPCTKQU +P RCTVKEWNCT VJG OGVJQF QH JKFFGP /CTMQX OQFGNKPI KU C RTGXC NGPV CRRTQCEJ KP RTQXKFKPI UVCVKUVKECN EJCTCEVGTK\CVKQP QH JWOCP URGGEJ CPF VJG HWNN EQORNGZKV[ QH ENCUUKH[KPI URQPVCPGQWU JWOCP URGGEJ KU UVKNN VQQ ITGCV VQ JCPFNG +P VJG PGZV UGEVKQP YG IKXG C DTKGH FKUEWUUKQP QH VJG URGGEJ TGEQIPKVKQP RTQDNGO CPF *//DCUGF CEQWUVKE OQFGNKPI DGHQTG KPVTQFWEKPI C FKUETKOKPCPV HWPEVKQP DCUGF CR RTQCEJ VQ URGGEJ TGEQIPKVKQP

 

1.4 Speech Recognition and Hidden Markov Modeling 5RGGEJ TGEQIPKVKQP KU C RTQDNGO QH TGEQIPK\KPI C YQTF UGSWGPEG HTQO JWOCP URGGEJ +V ECP DG XKGYGF CU C EQOOWPKECVKQP RTQDNGO 6JG JWOCP DTCKP UGTXGU CU VJG VGZV IGPGTCVQT YJKEJ IGPGTCVGU VJG YQTF UVTKPI   6JG YQTF UVTKPI IQGU VQ VJG CEQWUVKE EJCPPGN YJKEJ EQPUKUVU QH C URGCMGTŏU CTVKEWNCVQT[ CRRCTCVWU CPF QVJGT CEQWUVKE RTQ EGUUGU VJCV EQPXGTV VJG VGZV UVTKPI KPVQ CP CWFKDNG CEQWUVKE YCXGHQTO 6JG CEQWUVKE EJCPPGN KP XGTDCN EQOOWPKECVKQP CEVU CU C FCVC VTCPUFWEGT CPF EQORQUGT 6JG URGGEJ TGEQIPK\GT KU C FGEQFGT YJKEJ RGTHQTOU CP KPXGTUG QRGTCVKQP VQ FGEQFG VJG OGUUCIG HTQO VJG URGGEJ YCXGHQTO 6JGTGHQTG C FGEQFGT RGTHQTOU C OCZKOWO C RQUVGTKQT  UWEJ VJCV FGEKUKQP VJCV FGVGTOKPGU VJG YQTF UGSWGPEG 

   
 
   

  
  Ï Ï ‹E\&5&3UHVV//&



YJGTG


  KU VJG UEQTG HTQO CEQWUVKE OQFGNKPI CPF

  KU VJG UEQTG HTQO VJG NCPIWCIG OQFGN # V[RKECN URGGEJ TGEQIPKVKQP U[UVGO EQPUKUVU QH VJG HQNNQYKPI DCUKE EQORQPGPVU 







#EQWUVKE HGCVWTG GZVTCEVKQP #EQWUVKE HGCVWTG GZVTCEVKQP KU VQ GZVTCEV VJG HGC VWTGU HQT URGGEJ TGEQIPKVKQP HTQO VJG URGGEJ YCXGHQTO +V V[RKECNN[ KPENWFGU C UJQTVVKOG EGRUVTCN CPCN[UKU YJKEJ IGPGTCVGU C HGCVWTG XGEVQT QH NQY HTG SWGPE[  EGRUVTCN EQGHſEKGPVU HQT GXGT[  8CTKQWU UKIPCN RTQEGUU KPI RTQEGFWTGU CTG RGTHQTOGF VQ UGRCTCVG VJG UCNKGPV CEQWUVKE KPHQTOCVKQP HQT URGGEJ TGEQIPKVKQP RWTRQUGU (TQO PQY QP YG YKNN WUG VJG PQVCVKQP  
½      Ì  VQ TGRTGUGPV VJG CEQWUVKE QDUGTXCVKQP HGCVWTG XGEVQT UGSWGPEG #EQWUVKE OQFGNKPI #EQWUVKE OQFGNKPI RTQXKFGU UVCVKUVKECN OQFGNKPI HQT VJG CEQWUVKE QDUGTXCVKQP   *KFFGP /CTMQX OQFGNKPI KU VJG RTGXCNGPV EJQKEG HQT VJKU RWTRQUG CNVJQWIJ VJG PGWTCN PGVYQTM DCUGF CRRTQCEJ KU CNUQ WUGF KP OCP[ U[UVGOU 6JG OQFGN WPKVU ECP DG DCUGF QP UGOCPVKECNN[ OGCPKPIHWN WPKVU UWEJ CU YQTFU QT RJQPGVKECNN[ OGCPKPIHWN UWDYQTF WPKVU UWEJ CU RJQPGOGU .CPIWCIG OQFGNKPI .CPIWCIG OQFGNKPI RTQXKFGU NKPIWKUVKE CPF ITCOOCT EQPUVTCKPVU VQ VJG VGZV UGSWGPEG   +V KU QHVGP DCUGF QP UVCVKUVKECN 0ITCOU NCPIWCIG OQFGNU #P 0ITCO NCPIWCIG OQFGN KU QH VJG HQTO

Ò
½      Ò ½  YJKEJ KU VJG RTQDCDKNKV[ QH QDUGTXKPI YQTF Ò IKXGP VJG YQTF JKUVQT[ ½      Ò ½  &GEQFKPI GPIKPG 6JG FGEQFKPI GPIKPG UGCTEJGU HQT VJG DGUV YQTF UGSWGPEG IKXGP VJG HGCVWTG CPF VJG OQFGN (QT URGGEJ TGEQIPKVKQP DCUGF QP *// OQF GNKPI VJKU KU CEJKGXGF VJTQWIJ 8KVGTDK FGEQFKPI (QT C FKUETGVG QDUGTXCVKQP RTQDCDKNKV[ DCUGF U[UVGO YQTF UVTKPI  KU IKXGP D[  

  

Ï



 Ï
 

YJGTG Å KU VJG DGUV UVCVG UGSWGPEG IKXGP EQPVKPWQWU FGPUKV[ *//U  

CPF VJG OQFGN

 Ï
 

   


Ï

 

 

(QT



YJKEJ KU DCUGF QP VJG NQINKMGNKJQQF UEQTG CNQPI VJG DGUV UVCVG UGSWGPEG  É 

1.4.1 Hidden Markov Modeling of Speech 5RGGEJ KU IGPGTCVGF HTQO JWOCP CTVKEWNCVQT CPF KV KU WPKSWG KP OCP[ YC[U 9JGP YG URGCM QWT CTVKEWNCVQT[ CRRCTCVWU VJG NKRU LCY VQPIWG CPF XGNWO OQFWNCVGU VJG CKT RTGUUWTG CPF ƀQY VQ RTQFWEG CP CWFKDNG UGSWGPEG QH UQWPFU &WG VQ VJG RJ[UKECN EQPUVTCKPVU VJG CTVKEWNCVQT EQPſIWTCVKQP ECPPQV WPFGTIQ XGT[ FTCUVKE EJCPIGU CPF FWTKPI VJG UJQTV KPVGTXCN YJGTG VJG CTVKEWNCVQT[ EQPſIWTCVKQP UVC[U TGNCVKXGN[ EQP UVCPV C TGIKQP QH őSWCUKUVCVKQPCTKV[Œ KP VJG RTQFWEGF URGGEJ ECP QHVGP DG QDUGTXGF

‹E\&5&3UHVV//&

*KFFGP /CTMQX OQFGNKPI KU C RQYGTHWN UVCVKUVKECN HTCOGYQTM HQT VKOG XCT[KPI SWCUK UVCVKQPCT[ RTQEGUU CPF C RQRWNCT EJQKEG HQT UVCVKUVKECN OQFGNKPI QH URGGEJ UKIPCN )KXGP C URGGEJ WVVGTCPEG NGV
Ü ½ Ü
Ü
 DG C HGCVWTG XGEVQT UGSWGPEG GZVTCEVGF HTQO VJG URGGEJ YCXGHQTO YJGTG Ü  FGPQVGU C UJQTVVKOG XGEVQT OGCUWTG OGPV CPF KV KU EQPXGPVKQPCNN[ C EGRUVTCN XGEVQT (WTVJGT EQPUKFGT C ſTUVQTFGT UVCVG /CTMQX EJCKP IQXGTPGF D[ C UVCVG VTCPUKVKQP RTQDCDKNKV[ OCVTKZ

   YJGTG  KU VJG RTQDCDKNKV[ QH OCMKPI C VTCPUKVKQP HTQO UVCVG VQ UVCVG  #UUWOG VJCV CV
 VJG UVCVG QH VJG U[UVGO  KU URGEKſGF D[ CP KPKVKCN UVCVG RTQDCDKNKV[ 
 
 6JGP HQT CP[ UVCVG UGSWGPEG Õ
 
 VJG RTQDCDKNKV[ QH Õ DGKPI IGPGTCVGF D[ VJG /CTMQX EJCKP KU




  







    

  





 Õ 
 
¼ ¼ ½ ½¾    Ì ½Ì 

 5WRRQUG VJG U[UVGO YJGP CV UVCVG   RWVU QWV CP QDUGTXCVKQP Ü  CEEQTFKPI VQ C FKUVTKDWVKQP Ø Ü 
 Ü    
       6JG JKFFGP /CTMQX OQFGN WUGF CU C FKUVTKDWVKQP HQT VJG URGGEJ WVVGTCPEG
KU VJGP FGſPGF CU 
 
   

 
 
  
 Õ  
  
 Õ   Õ  
 ¼
 Ø ½ Ø Ø Ü   YJGTG
 
       KU VJG RCTCOGVGT UGV HQT VJG OQFGN #U ECP DG UGGP KP   Ø  FGſPGU VJG FKUVTKDWVKQP HQT UJQTVVKOG QDUGTXCVKQPU CPF
EJCTCEVGTK\GU VJG DGJCXKQT CPF KPVGTTGNCVKQPUJKR DGVYGGP FKHHGTGPV UVCVGU QH VJG

URGGEJ IGPGTCVKQP RTQEGUU +P QVJGT YQTFU VJG UVTWEVWTG QH C JKFFGP /CTMQX OQFGN RTQXKFGU C TGCUQPCDNG OGCPU HQT EJCTCEVGTK\KPI VJG FKUVTKDWVKQP QH C URGGEJ UKIPCN 0QTOCNN[  VJG VQVCN PWODGT QH UVCVGU KU OWEJ UOCNNGT VJCP  VJG VKOG FWTCVKQP QH VJG URGGEJ WVVGTCPEG 6JG UVCVG UGSWGPEG Õ FKURNC[U C EGTVCKP FGITGG QH UVCDKNKV[ COQPI CFLCEGPV ¼ U FWG VQ VJG CDQXG OGPVKQPGF őSWCUKUVCVKQPCTKV[Œ 6JG WUG QH *//U CU URGGEJ FKUVTKDWVKQPU KU UJQYP VQ DG RTCEVKECNN[ GHHGEVKXG +V UJQWNF DG PQVGF VJCV VJG EJQKEG QH UVCVG QDUGTXCVKQP FKUVTKDWVKQPU Ø Ü  KU PQV URGEKſGF &KHHGTGPV EJQKEGU QH URGGEJ FKOGPUKQPU HQT VJG QDUGTXCVKQP URCEG OC[ TGSWKTG FKHHGTGPV HQTOU QH VJG UVCVG QDUGTXCVKQP FKUVTKDWVKQP (QT EGRUVTCN XGEVQTU C OKZVWTG )CWUUKCP FGPUKV[ KU EQOOQPN[ GORNQ[GF /QTGQXGT TGICTFNGUU QH VJG RTCEVKECN GHHGEVKXGPGUU QH *// KP URGGEJ TGEQIPKVKQP KV UJQWNF PQV DG VCMGP CU VJG VTWG FKUVTKDWVKQP HQTO QH URGGEJ CPF VJGTGHQTG CP[ TGEQIPKVKQP U[UVGO QT FGEKUKQP TWNG VJCV QRGTCVGU DCUGF QP *// KU PQV IQKPI VQ CEJKGXG VJG OKPKOWO GTTQT TCVG CU KORNKGF KP VJG VTWG $C[GU /#2 FGEKUKQP +P QTFGT VQ CRRN[ *//U VQ URGGEJ TGEQIPKVKQP VJTGG DCUKE RTQDNGOU JCXG VQ DG TG UQNXGF PCOGN[ VJG GXCNWCVKQP RTQDNGO VJG FGEQFKPI RTQDNGO CPF VJG GUVKOCVKQP RTQDNGO = ? 6JG GXCNWCVKQP RTQDNGO KU VQ GUVKOCVG VJG RTQDCDKNKV[    QH QDUGTXKPI VJG URGGEJ HGCVWTG XGEVQT UGSWGPEG IKXGP VJG JKFFGP /CTMQX OQFGN 6JG FGEQFKPI RTQDNGO KU VQ ſPF C DGUV UVCVG UGSWGPEG Õ YJKEJ KU QRVKOCN KP C EGTVCKP UGPUG IKXGP VJG URGGEJ HGCVWTG UGSWGPEG  5KPEG UVCVGU KP *// CTG TGNCVGF VQ YQTFU CPF YQTF ENCUUGU VJG YQTF UGSWGPEG KP URGGEJ WVVGTCPEG ECP DG KFGPVKſGF D[ VTCEKPI VJTQWIJ VJG YQTF NCDGNU KP UVCVG UGSWGPEG Õ 6JG GUVKOCVKQP RTQDNGO KU VQ GUVKOCVG













‹E\&5&3UHVV//&




*// RCTCOGVGTU
HTQO C IKXGP UGV QH VTCKPKPI UCORNGU CEEQTFKPI VQ UQOG OGCP KPIHWN ETKVGTKQP 6JG EQPXGPVKQPCN CRRTQCEJ KU DCUGF QP VJG OCZKOWO NKMGNKJQQF

/. RTKPEKRNG CPF VJG OQFGN RCTCOGVGT UGV
KU GUVKOCVGF UQ VJCV VJG NKMGNKJQQF QP VJG VTCKPKPI FCVC KU OCZKOK\GF 8CTKQWU JKIJN[ GHſEKGPV /. DCUGF CNIQTKVJOU CTG FGXGNQRGF KP URGGEJ TGEQIPKVKQP HQT *//U UWEJ CU $CWO9GNEJ CNIQTKVJO =? CPF UGIOGPVCN MOGCPU CNIQTKVJO =? /QTG FKUEWUUKQPU QH RCTCOGVGT GUVKOCVKQP RTQD NGO HQT *//U ECP DG HQWPF KP =? +V UJQWNF DG PQVGF VJCV VJG EQPXGPVKQPCN /. OGVJQF KP URGGEJ TGEQIPKVKQP FQGU PQV PGEGUUCTKN[ NGCF VQ C OKPKOWO GTTQT TCVG RGT HQTOCPEG HQT VJG TGEQIPK\GT 6JKU KU FWG OCKPN[ VQ  VJG NKMGN[ OKUOCVEJ DGVYGGP VJG EJQUGP FKUVTKDWVKQP HQTO CPF VJG CEVWCN URGGEJ FCVC FKUVTKDWVKQP CPF  VJG ſPKVG VTCKPKPI MPQYP FCVC UGV YJKEJ KU QHVGP KPCFGSWCVG

1.5 MCE Classifier Design Using Discriminant Functions #U KV KU PQVGF YKVJQWV VJG MPQYNGFIG QH VJG HQTO QH VJG ENCUU RQUVGTKQT RTQDCDKNK VKGU TGSWKTGF KP VJG ENCUUKECN $C[GU FGEKUKQP VJGQT[ ENCUUKſGT FGUKIP D[ FKUVTKDWVKQP GUVKOCVKQP QHVGP FQGU PQV NGCF VQ CP QRVKOCN RGTHQTOCPEG 6JKU OQVKXCVGU VJG GHHQTV QH UGCTEJKPI HQT QVJGT CNVGTPCVKXG ETKVGTKC KP ENCUUKſGT FGUKIP +P RCTVKEWNCT ETKVGTKC QH //+ OCZKOWO OWVWCN KPHQTOCVKQP CPF /&+ OKPKOWO FKUETKOKPCVKXG KPHQT OCVKQP CTG WUGF KP OCP[ CRRNKECVKQPU = ? #NVJQWIJ VJGUG OGVJQFU FGOQPUVTCVG UKIPKſECPV RGTHQTOCPEG CFXCPVCIGU QXGT VJG VTCFKVKQPCN /. CRRTQCEJ VJG[ CTG PQV DCUGF QP C FKTGEV OKPKOK\CVKQP QH C NQUU HWPEVKQP YJKEJ NKPMU VQ VJG ENCUUKſECVKQP GTTQT TCVG &Q6W GV CN =? UVWFKGF /%' UQNWVKQP HQT VJG VYQ ENCUU PQPRCTCOGVTKE ENCUUK ſECVKQP RTQDNGO WUKPI NKPGCT FKUETKOKPCPV HWPEVKQPU 6JG[ GORNQ[GF C YKPFQYGF UEJGOG VQ QXGTEQOG VJG RTQDNGO QH UKPIWNCT ITCFKGPV HWPEVKQPU CUUQEKCVGF YKVJ VJG GTTQT EQWPV KPFKECVQT HWPEVKQP QH VJG ENCUUKſGT # IGPGTCN CRRTQCEJ HQT OWNVKENCUU CPF PQPNKPGCT FKUETKOKPCPV HWPEVKQPU CTG RTQRQUGF D[ ,WCPI GV CN =? 6JKU IGP GTCN CRRTQCEJ KU ECNNGF őOKPKOWO ENCUUKſECVKQP GTTQT /%' OGVJQFŒ KP YJKEJ VJG ENCUUKſGT FGUKIP CPF RCTCOGVGT GUVKOCVKQP CTG VQ EQTTGEVN[ FKUETKOKPCVG VJG QDUGTXC VKQPU HQT DGUV TGEQIPKVKQPENCUUKſECVKQP TGUWNVU TCVJGT VJCP VQ ſV VJG FKUVTKDWVKQPU VQ VJG FCVC

1.5.1 MCE Classifier Design Strategy .GV WU EQPUKFGT C UGV QH ENCUU FKUETKOKPCPV HWPEVKQPU


          FG ſPGF D[ VJG RCTCOGVGT UGV
 6JG ENCUUKſGT   KU VJG QPG VJCV HQT CP QDLGEV   



KHH



  








6JG IGPGTCN /%' ENCUUKſGT FGUKIP UVTCVGI[ KU DCUGF QP C URGEKCN V[RG QH NQUU HWPEVKQP 2CTCOGVGTU QH VJG ENCUUKſGT CTG GUVKOCVGF KP UWEJ C YC[ VJCV OKPKOK\KPI VJG GZRGEVGF

‹E\&5&3UHVV//&

NQUU TGNCVGU VQ C OKPKOK\CVKQP QH VJG TGEQIPKVKQP GTTQT TCVG QH VJG ENCUUKſGT 6JKU KU CEJKGXGF VJTQWIJ C VJTGG UVGR RTQEGUU  6JG OKUENCUUKſECVKQP OGCUWTG KP VJG /%' DCUGF CRRTQCEJ KU FGſPGF CU





  


     













   
  






YJGTG  KU C RQUKVKXG PWODGT =? 6JKU OKUENCUUKſECVKQP OGCUWTG KU C EQPVKPWQWU HWPEVKQP QH VJG ENCUUKſGT RCTCOGVGTU  CPF CVVGORVU VQ GOWNCVG VJG FGEKUKQP TWNG IKXGP VJG FKUETKOKPCPV HWPEVKQP  
 (QT CP  ENCUU WVVGTCPEG  

    KORNKGU OKUENCUUKſECVKQP CPF

    OGCPU C EQTTGEV FGEKUKQP 9JGP  CR RTQCEJGU  VJG VGTO KP VJG DTCEMGV KU VJG  PQTO QP VJG FKUETGVG KPVGIGT UGV          YJKEJ EQPXGTIGU VQ VJG  
PQTO CPF DGEQOGU 


 

   $[ XCT[KPI VJG XCNWG QH  CPF   QPG ECP VCMG CNN VJG EQORGVKPI ENCUUGU KPVQ EQPUKFGTCVKQP CEEQTFKPI VQ VJG KPFKXKFWCN UKIPKſECPEG YJGP UGCTEJKPI HQT VJG ENCUUKſGT RCTCOGVGT   6JG NQUU HWPEVKQP KU WUGF HQT TGEQIPKVKQP GTTQT TCVG OKPKOK\CVKQP 6JG OKUENCUUKſ ECVKQP OGCUWTG QH  KU GODGFFGF KP C UOQQVJ \GTQQPG HWPEVKQP HQT YJKEJ CP[ OGODGT QH VJG UKIOQKF HWPEVKQP HCOKN[ KU CP QDXKQWU ECPFKFCVG # IGPGTCN HQTO QH VJG loss function ECP VJGP DG FGſPGF CU


   


 



YJGTG KU C UKIOQKF HWPEVKQP QPG GZCORNG QH YJKEJ KU



 



 




 



YKVJ  PQTOCNN[ UGV VQ  CPF  UGV VQ ITGCVGT QT GSWCN VQ QPG %NGCTN[ YJGP

  KU OWEJ UOCNNGT VJCP \GTQ YJKEJ KORNKGU EQTTGEV ENCUUKſECVKQP XKTVWCNN[ PQ NQUU KU KPEWTTGF 9JGP

  KU RQUKVKXG KV NGCFU VQ C RGPCNV[ YJKEJ DGEQOGU GUUGPVKCNN[ C ENCUUKſECVKQPTGEQIPKVKQP GTTQT EQWPV  6JG ENCUUKſGT RCTCOGVGT GUVKOCVKQP KU DCUGF QP VJG OKPKOK\CVKQP QH VJG GZRGEVGF NQUU (QT CP[ WPMPQYP QDLGEV   VJG ENCUUKſGT RGTHQTOCPEG KU OGCUWTGF D[


   

  





  


 



YJGTG
 KU VJG KPFKECVQT HWPEVKQP 6JG GZRGEVGF NQUU YJKEJ KU TGNCVGF VQ TGEQIPK VKQP GTTQT TCVG KU IKXGP D[


   
  



6JKU VJTGGUVGR FGſPKVKQP GOWNCVGU VJG ENCUUKſECVKQP QRGTCVKQP CU YGNN CU VJG TGEQI PKVKQP GTTQT TCVG DCUGF RGTHQTOCPEG GXCNWCVKQP KP C UOQQVJ HWPEVKQPCN HQTO UWKVCDNG

‹E\&5&3UHVV//&

HQT ENCUUKſGT RCTCOGVGT QRVKOK\CVKQP +V UJQWNF DG RQKPVGF QWV VJCV KH VJG EQTTGEV HQTO QH VJG RQUVGTKQT RTQDCDKNKV[




 KU WUGF VJG $C[GU OKPKOWO TKUM KU VJGP GZRTGUUGF CU








 
  



  
 


   


   CPF  TGRTGUGPVU VJG GPVKTG 
 






YJGTG    


 UKIPCN URCEG 6JKU ECP DG CRRTQZKOCVGF D[ VJG NQUU HWPEVKQP KP /%' CRRTQCEJ CU HQNNQYU


  




 







 






   




   









   

 





#P KORQTVCPV RQKPV JGTG KU VJCV CRRTQZKOCVKQP CEEWTCE[ QH 'S  ECP DG EQP VTQNNGF D[ XCT[KPI VJG EQPUVCPVU KP VJG UOQQVJ /%' NQUU HWPEVKQP $CUGF QP VJG ETKVGTKQP QH  YG ECP EJQQUG VQ OKPKOK\G QPG QH VYQ SWCPVKVKGU HQT VJG ENCUUKſGT RCTCOGVGT UGCTEJ QPG KU VJG GZRGEVGF NQUU CPF VJG QVJGT VJG GORKTKECN NQUU

1.5.2 Optimization Methods 6JG RWTRQUG QH VJG VTCKPKPI RTQEGUU KP VJG /%' CRRTQCEJ KU VQ ſPF C UGV QH RCTCOGVGTU  UQ VJCV C RTGUETKDGF NQUU KU OKPKOK\GF #U OGPVKQPGF RTGXKQWUN[ VJG VYQ MKPFU QH NQUU YG HQEWU QP CTG VJG GZRGEVGF NQUU CPF VJG GORKTKECN NQUU 1.5.2.1 Expected Loss (QT C ENCUUKſECVKQP RTQDNGO KPXQNXKPI FKHHGTGPV ENCUUGU VJG GZRGEVGF NQUU KU FG ſPGF CU


   
 





  ¾








8CTKQWU OKPKOK\CVKQP CNIQTKVJOU ECP DG WUGF VQ OKPKOK\G VJG GZRGEVGF NQUU 6JG IGPGTCNK\GF RTQDCDKNKUVKE FGUEGPV )2& CNIQTKVJO KU C RQYGTHWN CNIQTKVJO VJCV ECP DG WUGF VQ CEEQORNKUJ VJKU VCUM =? +P )2& DCUGF OKPKOK\CVKQP CNIQTKVJO VJG VCTIGV HWPEVKQP 
 KU OKPKOK\GF CEEQTFKPI VQ CP KVGTCVKXG RTQEGFWTG

      
  

 YJGTG  KU C RQUKVKXG FGſPKVG OCVTKZ =?  KU C UGSWGPEG QH RQUKVKXG PWODGTU CPF 
  

 KU VJG ITCFKGPV HWPEVKQP QH VJG NQUU HWPEVKQP CV      CPF  KU 













VJG VJ VTCKPKPI UCORNG WUGF KP VJG UGSWGPVKCN VTCKPKPI RTQEGUU 6JG EQPXGTIGPEG RTQRGTVKGU QH )2& CNIQTKVJO YCU UVWFKGF KP VJG NKVGTCVWTG GI =   ? CPF UQOGVKOGU WPFGT VJG PCOG QH UVQEJCUVKE CRRTQZKOCVKQP 7PFGT XGT[ IGPGTCN EQPFKVKQPU VJG HQNNQYKPI EQPXGTIGPEG RTQRGTVKGU ECP DG GUVCDNKUJGF =?

‹E\&5&3UHVV//&


½ 



 
  


      
 such that for all t, the inner product                     where is the Hessian matrix of second order partial derivatives;
  £       is the unique such that     
      
  

Property 1 Suppose the following conditions are satisfied:










½












































Then,
given by


 






    











will converge to £ almost surely (i.e. with probability one). %QPFKVKQP
 ECP DG EQPUKFGTCDN[ YGCMGPGF 'XGP YKVJQWV EQPFKVKQP NQYKPI KU UVKNN VTWG   
  






 VJG HQN



YJGTG
 KU C UWDUGSWGPEG QH
 +P VJKU ECUG
 YKNN EQPXGTIG VQ C NQECN OKPKOWO RQKPV £ YJGTG   
   #FCRVKPI VJG OQFGN RCTCOGVGTU WUKPI C UCORNG D[ UCORNG WRFCVKPI HQTOWNC CU KP 'S  KU OQUV GHſEKGPV KP VGTOU QH VJG WUG QH CXCKNCDNG VTCKPKPI UCORNGU $WV VJG UKPING UCORNG DCUGF ITCFKGPV GUVKOCVKQP ECP DG PQKU[ NGCFKPI VQ ƀWEVWCVKQPU FWT KPI VJG RCTCOGVGT GUVKOCVKQP RTQEGUU $CVEJ OQFG CFCRVCVKQP UEJGOGU DCUGF QP C ITCFKGPV GUVKOCVG YJKEJ KU CP CXGTCIG QH GXGT[  UCORNGU ECP CNUQ DG WUGF 1VJGT XCTKCVKQPU VQ VJG QTKIKPCN )2& CNIQTKVJOU CTG CNUQ RQUUKDNG EJQKEGU VQ URGGF WR VJG EQPXGTIGPEG CPF TGFWEG VJG ƀWEVWCVKQP FWTKPI VJG ENCUUKſGT VTCKPKPI RTQEGUU 6JG EQPXGTIGPEG RTQRGTVKGU QH VJGUG TGNCVGF CFCRVCVKQP CNIQTKVJOU CTG DCUGF QP XCTKQWU UVCVKUVKECN EQPXGTIGPEG VJGQTKGU UWEJ CU OCTVKPICNG VJGQT[ RQVGPVKCN HWPEVKQPU GVE CPF KV KU UVKNN C XGT[ CEVKXG CTGC QH TGUGCTEJ *QYGXGT HTQO CP CRRNKECVKQP RQKPV QH XKGY KP QTFGT VQ CRRN[ VJKU CNIQTKVJO VQ URGGEJ TGEQIPKVKQP UWEJ CU C URGGEJ TGEQIPKVKQP U[UVGO WUKPI *//U VJG )2& CNIQTKVJO JCU VQ CEEQOOQFCVG XCTKQWU EQPUVTCKPVU KORQUGF QP VJG *// UVTWEVWTGU +P RCTVKEWNCT VJG )2& CNIQTKVJO KU CP WPEQPUVTCKPGF OKPKOK\CVKQP UEJGOG VJCV PGGFU OQFKſECVKQP HQT UQNXKPI OKPKOK\C VKQP RTQDNGOU YKVJ EQPUVTCKPVU #U YKNN DG UJQYP UJQTVN[ QPG ECP WVKNK\G RCTCOGVGT URCEG VTCPUHQTOCVKQPU VQ TGUQNXG VJKU KUUWG +P VJKU OGVJQF VJG QTKIKPCN RCTCOGVGTU CTG WRFCVGF VJTQWIJ VJG KPXGTUG VTCPUHQTO HTQO VJG VTCPUHQTOGF RCTCOGVGT URCEG VQ VJG QTKIKPCN RCTCOGVGT URCEG 6JKU KU FQPG KP UWEJ C YC[ VJCV EQPUVTCKPVU QP VJG QTKI KPCN RCTCOGVGTU CTG CNYC[U OCKPVCKPGF /QTG FGVCKNGF KNNWUVTCVKQPU QH VJKU CRRTQCEJ CTG IKXGP KP NCVGT UGEVKQPU





1.5.2.2 Empirical Loss

 



   VJG GORKTKECN (QT C IKXGP VTCKPKPI FCVC UGV EQPUKUVKPI QH  UCORNGU   RTQDCDKNKV[ OGCUWTG   FGſPGF QP VJG VTCKPKPI FCVC UGV KU C FKUETGVG RTQDCDKNKV[ OGC

‹E\&5&3UHVV//&

UWTG YJKEJ CUUKIPU GSWCN OCUU CV GCEJ UCORNG 6JG GORKTKECN NQUU QP VJG QVJGT JCPF KU VJWU GZRTGUUGF CU




  



 







  

 ¾   

  






  




YJGTG  FGPQVGU VJG KPFGZ QH VJG VTCKPKPI WVVGTCPEG   KP VJG VTCKPKPI UGV QH UK\G   CPF 
KU VJG GORKTKECN OGCUWTG FGſPGF QP VJG VTCKPKPI UGV +H VJG VTCKPKPI UCORNGU CTG QDVCKPGF D[ CP KPFGRGPFGPV UCORNKPI HTQO C URCEG YKVJ C ſZGF RTQDCDKNKV[ FKUVTK DWVKQP   VJG GORKTKECN RTQDCDKNKV[ FKUVTKDWVKQP 
YKNN EQPXGTIG VQ  KP FKUVTKDWVKQP  +P QVJGT YQTFU HQT CP[ OGCUWTCDNG HWPEVKQP  CU 


















6JG GORKTKECN NQUU FGſPGF QP VJG  KPFGRGPFGPV VTCKPKPI UCORNGU YKNN EQPXGTIG VQ VJG GZRGEVGF NQUU CU VJG UCORNG UK\G  KPETGCUGU 9KVJ UWHſEKGPV VTCKPKPI UCORNGU VJG GORKTKECN NQUU KU CP GUVKOCVG QH VJG GZRGEVGF NQUU 6JG IQQFPGUU QH VJKU GUVKOCVG KU FGVGTOKPGF D[ VJG VTCKPKPI UCORNG UK\G   CPF VJG EQPXGTIGPEG TCVG QH VJG GORKTKECN RTQDCDKNKV[ OGCUWTG 
VQ VJG NKOKV FKUVTKDWVKQP   8CTKQWU WRRGT DQWPFU QP VJG EQPXGTIGPEG TCVG QH VJG GORKTKECN RTQDCDKNKV[ OGCUWTG ECP DG HQWPF KP =?

1.5.3 Other Optimization Methods +V UJQWNF DG RQKPVGF QWV VJCV CNVJQWIJ VJG )2& V[RG QH CFCRVCVKQP CNIQTKVJO KU GH HGEVKXG CPF OQUV RQRWNCT QVJGT QRVKOK\CVKQP OGVJQFU ECP CNUQ DG WUGF HQT GTTQT TCVG OKPKOK\CVKQP /%' DCUGF ENCUUKſGT FGUKIP KU XGT[ URGEKſE QP VJG HQTO CPF UVTWEVWTG QH VJG FKUETKOKPCPV HWPEVKQP CPF NQUU HWPEVKQP TGICTFKPI VJG ENCUUKſGT CPF TGNCVKXGN[ WPTGUVTKEVGF VQ YJCV RCTVKEWNCT QRVKOK\CVKQP OGVJQFU YJKEJ CTG WUGF VQ OKPKOK\G VJG NQUU /CP[ KPPQXCVKQPU CTG RQUUKDNG HQT DGVVGT QRVKOK\CVKQP TGUWNVU +P RCTVKE WNCT OGVJQFU QH NKPGCT RTQITCOOKPI =? ITCFKGPV RTQLGEVKQP =? CPF ITQYVJ VTCPUHQTOCVKQP =  ? CTG CNUQ WUGF HQT OKPKOK\CVKQP QH VJG GZRGEVGF NQUU KP /%' ENCUUKſGT FGUKIP +P VJG ITQYVJ VTCPUHQTOCVKQP DCUGF CRRTQCEJ VJG IQCN KU VQ UGGM CVTCPUHQTOCVKQP 
 UWEJ VJCV
 
 YJGTG KU C RTQDCDKNKV[ XGEVQT 

KG     CPF   6JKU CRRTQCEJ UQOGVKOGU TGHGTTGF VQ CU GZVGPFGF $CWO9GNEJ $9 CNIQTKVJO KP UQOG NKVGTCVWTG YCU QTKIKPCVGF HTQO $CWO'CIQPŏU KPGSWCNKV[ HQT
 DGKPI C RQN[PQOKCN YKVJ PQPPGICVKXG EQGHſEKGPVU CPF JQOQIG PGQWU QH FGITGG  KP KVU XCTKCDNGU +V KU GZVGPFGF VQ TCVKQPCN HWPEVKQPU CPF CRRNKGF VQ URGGEJ TGEQIPKVKQP HQT OCZKOWO OWVWCN KPHQTOCVKQP //+ VTCKPKPI YKVJ FKUETGVG RTQDCDKNKV[ EQFGDQQMU =? .CVGT KV YCU HWTVJGT IGPGTCNK\GF VQ CPCN[VKE HWPEVKQPU =? 5KPEG VJGP VJKU CRRTQCEJ YCU CFQRVGF HQT /%' VTCKPKPI QH *// DCUGF URGGEJ TGEQIPKVKQP U[UVGOU YKVJ FKUETGVG RTQDCDKNKV[ FGPUKVKGU =? +P VJG ITQYVJVTCPUHQTOCVKQP DCUGF QRVKOK\CVKQP CRRTQCEJ VJG OQFGN RCTCOGVGT YJKEJ KU C EQORQPGPV QH VJG RTQDCDKNKV[ XGEVQT  KU WRFCVGF YKVJ VJG HQNNQYKPI TG



‹E\&5&3UHVV//&



GUVKOCVKQP NKMG HQTOWNC













   ¼

 ¼


¼  



YJGTG  KU C EQPUVCPV VQ DG FGVGTOKPGF CPF VJG UWO KP VJG FGPQOKPCVQT KU VCMGP QXGT CNN RCTCOGVGTU DGNQPIKPI VQ VJG UCOG FKUVTKDWVKQP  +V KU UJQYP KP =? VJCV VJGTG KU C XCNWG  UWEJ VJCV KH    'S  KU C ITQYVJVTCPUHQTOCVKQP
   *QYGXGT 'S  ECPPQV DG FKTGEVN[ CRRNKGF VQ CPF       RCTCOGVGT GUVKOCVKQP QH EQPVKPWQWU RTQDCDKNKV[ FGPUKVKGU CPF PGY HQTOWNCVKQPU HQT EQPVKPWQWU FGPUKV[ *//U CTG PGGFGF 7UKPI C FKUETGVG CRRTQZKOCVKQP CTIWOGPV VJG ITQYVJVTCPUHQTOCVKQP OGVJQF KU GZVGPFGF VQ //+DCUGF RCTCOGVGT GUVKOCVKQP YKVJ EQPVKPWQWU FGPUKV[ *//U CPF PGY HQTOWNCVKQPU QH VJG ITQYVJVTCPUHQTOCVKQP HQT EQPVKPWQWU FGPUKV[ *//U KP //+ VTCKPKPI CTG FGTKXGF =? /QTG TGEGPVN[ CP GNGICPV RTQQH QH VJG VJGQTGVKECN RTQRGTVKGU QH VJG ITQYVJVTCPUHQTOCVKQP OGVJQF HQT //+ VTCKPKPI KU IKXGP KP =? YJKEJ GUVCDNKUJGU VJG ITQYVJVTCPUHQTOCVKQP OGVJQF KP C OQTG IGPGTCN UGVVKPI $WV OQTG YQTM TGOCKPU VQ DG FQPG KP QTFGT VQ CRRN[ VJG UKOKNCT OGVJQF HQT IGPGTCN /%' DCUGF RCTCOGVGT GUVKOCVKQP YKVJ EQPVKPWQWU RTQDCDKNKV[ FGPUKVKGU +V KU KPVGTGUVKPI VQ PQVG VJCV KH  KU NCTIG VJGP VJG EQPXGTIGPEG QH VJKU CNIQTKVJO KU UNQY CPF KH  KU VQQ NCTIG VJG CNIQTKVJO KU RTCEVKECNN[ PQV WUGHWN +P QTFGT VQ IGV HCUV EQPXGTIGPEG  PGGFU VQ DG CU UOCNN CU RQUUKDNG /QFKſECVKQPU QH VJG QTKIKPCN CNIQTKVJO CPF WUKPI UGCTEJ JGWTKUVKEU CTG CVVGORVGF VQ URGGF WR VJG EQPXGTIGPEG CPF RGTHQTOCPEG KORTQXGOGPVU QXGT VJG QTKIKPCN CRRTQCEJ CTG CNUQ TGRQTVGF KP = ?

1.5.4 HMM as a Discriminant Function (QNNQYKPI  YG JCXG UGXGTCN YC[U QH WUKPI CP *// CU VJG FKUETKOKPCPV HWPE VKQP # DCUKE EQORQPGPV KP  KU VJG LQKPV QDUGTXCVKQPUVCVG RTQDCDKNKV[      
     
¼








  ½     




YJKEJ KU PQY FGſPGF CU C EQORQPGPV HWPEVKQP      HQT ENCUU CU YGNN 6JG FKUETKOKPCPV HWPEVKQP HQT ENCUU ECP VCMG UGXGTCN RQUUKDNG HQTOU DCUGF QP     





    

       



    





             







   



YJGTG  KU VJG VQVCN PWODGT QH RQUUKDNG UVCVG UGSWGPEGU CPF  KU C RQUKVKXG PWODGT CPF

 (WPEVKQPU QH VJG CDQXG

‹E\&5&3UHVV//&

0QVG VJCV  KU GSWKXCNGPV VQ VJG NKMGNKJQQF HWPEVKQP  KU GSWKXCNGPV VQ VJG OCZKOWO LQKPV QDUGTXCVKQPUVCVG UGSWGPEG RTQDCDKNKV[ CPF  KU C IGPGTCNK\GF OKZVWTG OQFGN YJKEJ CRRTQCEJGU  YJGP
 9G WUG VJG NQICTKVJO QH

 CU CP GZCORNG KP QWT FGTKXCVKQP DGECWUG KV KU VJG OQUV RQRWNCT EJQKEG HQT *// DCUGF TGEQIPKVKQP U[UVGOU CUUQEKCVGF YKVJ 8KVGTDK FGEQFKPI 6JG CNIQTKVJO DCUGF QP  KU QHVGP ECNNGF segmental GPD =? 9G FGſPG HQT 
ܽ  Ü
     Ü
 CPF Ü
  
      ¼ YKVJ  DGKPI VJG FKOGPUKQP QH Ü  







    
    Õ 
   Õ   
  
 
    Ü     









YJGTG Õ
          
 KU VJG QRVKOCN UVCVG UGSWGPEG VJCV CEJKGXGU    Õ   9G CNUQ CUUWOG VJCV

 Ü 
YJGTG

 

  



      Ü  





 FGPQVGU C PQTOCN FKUVTKDWVKQP  CTG VJG OKZVWTG YGKIJVU 


    VJG OGCP XGEVQT CPF   VJG EQXCTKCPEG OCVTKZ YJKEJ HQT UKORNKEKV[ KU 
    CUUWOGF VQ DG FKCIQPCN KG  
  +V OC[ DG FGUKTCDNG VQ OCKPVCKP VJG QTKIKPCN EQPUVTCKPVU KP VJG *// CU RTQDCDKNKV[ OGCUWTG UWEJ CU  VJG HWPEVKQP DGKPI PQPPGICVKXG   
 HQT CNN  CPF    HQT CNN  CPF GVE #NUQ YG CUUWOG     6JG HQNNQYKPI RCTCOGVGT   



VTCPUHQTOCVKQPU CNNQY WU VQ OCKPVCKP VJGUG EQPUVTCKPVU FWTKPI RCTCOGVGT CFCRVCVKQP

  
   
 


 

  

     YJGTG 
         YJGTG 



 
 


 

 

   CPF    





  KP VJG VTCKPKPI UGV FKUETKOKPCVKXG CFLWUVOGPV QH VJG 

    

   
          

‹E\&5&3UHVV//&





    



+V ECP DG UJQYP VJCV HQT   OGCP XGEVQT HQNNQYU



 



YJGTG




  
















 


 
























   





 

Æ
 
 





Ü 

CPF












Ü



















  










 

 










 












 


Ü 



































YJGTG  KU VJG EGPVGT UNQRG QH VJG GZRQPGPVKCN UKIOQKF HWPEVKQP HQT   CU FGſPGF KP

 CPF Æ
 FGPQVGU VJG -TQPGEMGT FGNVC HWPEVKQP (KPCNN[ 
 
   



    

  








5KOKNCTN[ HQT VJG XCTKCPEG 
 
 



    

 





  





 



YJGTG












Ü 











 

  

 

Æ
 
 





Ü 







  


   


 

  

  

   



 
        





 
 
Ü  

  

 


(KPCNN[





 

     


   



5KOKNCT FGTKXCVKQPU HQT VJG VTCPUKVKQP RTQDCDKNKVKGU CPF VJG OKZVWTG YGKIJVU ECP DG GCUKN[ CEEQORNKUJGF = ?

‹E\&5&3UHVV//&

#U OGPVKQPGF GCTNKGT VJG )2& CNIQTKVJO KU C ITCFKGPV DCUGF CPF WPEQPUVTCKPGF OKP KOK\CVKQP OGVJQF +P QTFGT VQ WUG HQT FKUETKOKPCPV HWPEVKQPU HTQO EGTVCKP HCOKNKGU UWEJ CU RTQDCDKNKV[ FGPUKV[ HWPEVKQPU HTQO *//U ECTG OWUV DG VCMGP UWEJ VJCV VJQUG RTQDCDKNKUVKE EQPUVTCKPVU CTG OCKPVCKPGF 6TCPUHQTOCVKQPU KP 'SU    CTG WUGF HQT VJKU RWTRQUG #PQVJGT KORQTVCPV CPF RGTJCRU VJG OQUV FKHſEWNV KUUWG KP )2& DCUGF NQUU HWPEVKQP OKPKOK\CVKQP CRRTQCEJ KU JQY VQ FGUKIP VJG UVGR UK\G 1PG CRRCTGPV TGCUQP KU VJCV YG PGGF C IQQF UVGR UK\G VQ UVCTV YKVJ UKPEG VJG OQFGN CFCRVCVKQP YKNN DG RGTHQTOGF QPN[ C ſPKVG PWODGT QH VKOGU +H VJG UVGR UK\G KU VQQ NCTIG VJG ENCUUKſGT YKNN DG FGITCFGF CV VJG UVCTV CPF UGSWGPVKCN NGCTPKPI ECPPQV DG OCFG UWEEGUUHWN +H VJG UVGR UK\G KU VQQ UOCNN VJG EQPXGTIGPEG URGGF QH VJG CNIQTKVJO KU VQQ UNQY CPF KV KU RTCEVKECNN[ PQV WUGHWN 6JG UVGR UK\G RTQDNGO KU TGNCVGF VQ VJG RCTVKEWNCT HWPEVKQPCN HQTO QH VJG NQUU HWPEVKQP CPF VQ VJG DGUV QH QWT MPQYNGFIG VJG IGPGTCN UQNWVKQP VQ KV KU UVKNN NCEMKPI )GPGTCNN[ URGCMKPI KV UJQWNF DG TGNCVGF VQ VJG GKIGPXCNWGU QH VJG *GUUKCP OCVTKZ GXGP VJQWIJ KV KU CP KVGTCVKXG CNIQTKVJO (QT *// DCUGF U[UVGOU WUKPI OKZVWTG )CWUUKCP QDUGTXCVKQP FGPUKVKGU RCTCOGVGTU KP VJG ENCU UKſGT JCXG FKHHGTGPV UGPUKVKXKVKGU VQ VJG UVGR UK\G KP RCTCOGVGT CFCRVCVKQP 1PG UVGR UK\G ECP DG VQQ UOCNN HQT UQOG RCTCOGVGTU CPF VQQ NCTIG HQT QVJGTU +P RCTVKEWNCT VJG OCIPKVWFG QH XCTKCPEGU KP VJG OKZVWTG )CWUUKCP QDUGTXCVKQP FGPUKVKGU ECP XCT[ KP VJG TCPIG DGVYGGP ½¼¼ VQ ½¼
 +H WUKPI C EQPUVCPV UVGR UK\G HQT CNN OGCP XGEVQTU VJG CNIQTKVJO YKNN GKVJGT PQV EQPXGTIG QT YKNN DG VQQ UNQY VQ DGEQOG RTCEVKECNN[ WUGNGUU 6JG VTCPUHQTOCVKQP KP 'S  KU ETKVKECN CPF RTQXKFGU CP GHHGEVKXG UQNWVKQP VQ VJKU RTQDNGO +P UGIOGPVCN )2& CRRTQCEJ VJG GTTQT TCVG OKPKOK\CVKQP KU RGTHQTOGF QP VJG VTCPUHQTOGF OGCP XGEVQT PQTOCNK\GF D[ KVU UVCPFCTF FGXKCVKQP 6JKU VCMGU CYC[ VJG FGRGPFGPEKGU QP VJG XCTKCPEG XCTKCVKQPU &WTKPI )2& VTCKPKPI VJG VTCKPKPI FCVC ECP DG TGWUGF CPF VTCKPKPI ECP DG KVGTCVGF UGXGTCN VKOGU QP VJG UCOG FCVC VQ TGCEJ EQPXGTIGPEG /GVJQFU HTQO UVCVKUVKECN FCVC UCORNKPI VJGQT[ ECP CNUQ DG CRRNKGF JGTG +PUVGCF QH UGSWGPVKCNN[ WUKPI CNN VTCKPKPI UCORNGU DQQVUVTCR TGUCORNKPI UEJGOGU QT KORQTVCPEG QH UCORNKPI UEJGOGU ECP DG WUGF JGTG VQ GZVTCRQNCVG VJG UCORNG FCVC FKUVTKDWVKQP QT VQ CFCRV VJG ENCUUKſGT VQYCTFU UQOG URGEKſE RQRWNCVKQPU 9KVJ VJG CFXCPEG QH OKETQRTQEGUUQTU UWEJ OGVJQFU JCXG DGEQOG EQORWVCVKQPCNN[ HGCUKDNG 6JG CDQXGOGPVKQPGF UGIOGPVCN )2& CNIQTKVJO ſPFU OCP[ CRRNKECVKQPU KP URGGEJ TGEQIPKVKQP 6JG TGEQIPKVKQP RGTHQTOCPEG CFXCP VCIG QXGT VJG VTCFKVKQPCN FKUVTKDWVKQP DCUGF /. CRRTQCEJ KU TGRQTVGF HTQO XCTKQWU UKVGU CPF KP FKHHGTGPV CRRNKECVKQPU =    ? 6JG UWEEGUU QH VJKU CNIQTKVJO KP URGGEJ TGEQIPKVKQP RTQXKFGU GZRGTKOGPVCN GXKFGPEG VJCV ENCUUKſGT FGUKIP DCUGF QP GTTQT TCVG OKPKOK\CVKQP KU HGCUKDNG GXGP HQT F[PCOKE RCVVGTPU CU FKHſEWNV CU URGGEJ +ORTQXGF TGEQIPKVKQP RGTHQTOCPEG QXGT VTCFKVKQPCN FKUVTKDWVKQP DCUGF /. CRRTQCEJ CTG CNUQ TGRQTVGF KP CTGCU QWVUKFG QH URGGEJ UWEJ CU 1%4 KOCIG TGEQIPKVKQP CPF JCPFYTKVKPI TGEQIPKVKQP =?

1.5.5 Relation between MCE and MMI +P CFFKVKQP VQ VJG /%' ETKVGTKQP QVJGT ETKVGTKC CTG CNUQ WUGF KP UQECNNGF FKUETKOKPC VKXG ENCUUKſGT FGUKIP (QT *// DCUGF U[UVGOU VJG ETKVGTKC QH OCZKOWO OWVWCN KPHQTOCVKQP //+ =? EQPFKVKQPCN OCZKOWO NKMGNKJQQF GUVKOCVG %/.' =?

‹E\&5&3UHVV//&

OKPKOWO FKUETKOKPCVKQP KPHQTOCVKQP /&+ =? CPF *ETKVGTKC =? CTG QVJGT CN VGTPCVKXGU YJKEJ JCXG HQWPF VJGKT WUG KP URGGEJ TGEQIPKVKQP #OQPI VJGO //+ KU OQUV RQRWNCT CPF CRRNKGF KP OCP[ CRRNKECVKQPU YKVJ UWEEGUU 6JG //+ CRRTQCEJ KU DCUGF QP VJG OWVWCN KPHQTOCVKQP


   DGVYGGP VJG CEQWUVKE QDUGTXCVKQP : CPF KVU EQTTGEV NGZKECN U[ODQN 
 (QT VJG  ENCUU ENCUUKſECVKQP RTQDNGO VJG NQICTKVJO QH VJG OWVWCN KPHQTOCVKQP JCU VJG HQNNQYKPI HQTO


    



   


 

 


 

 

   



















 


 
 

  
 
  





YJGTG  TWPU QXGT CNN RQUUKDNG  ENCUU U[ODQNU 

    


 CPF 
    

  CTG VJG NQINKMGNKJQQF UEQTGU QH  QP VJG EQTTGEV NGZKECN U[ODQN CPF VJG VJ NGZKECN U[ODQN TGURGEVKXGN[ (TQO 'S  






   




  


 

 


     





YJKEJ TGNCVGU


   VQ VJG RQUVGTKQT RTQDCDKNKV[ 


  +P //+ VTCKPKPI VJG ETKVGTKQP QH VJG ENCUUKſGT FGUKIP CPF RCTCOGVGT GUVKOCVKQP KU VQ OCZKOK\G VJG CXGTCIG OWVWCN KPHQTOCVKQP

     QP VJG VTCKPKPI UGV 'ZRGTKOGPVCN URGGEJ TGEQIPKVKQP TGUWNVU KPFKECVG VJCV ENCUUKſGT FGUKIP DCUGF QP VJG //+ ETKVGTKQP ECP NGCF VQ DGV VGT TGEQIPKVKQP RGTHQTOCPEG VJCP VJG EQPXGPVKQPCN CRRTQCEJ WUKPI VJG /. ETKVGTKQP = ? #NVJQWIJ VJKU ETKVGTKQP KU YGNN HQWPFGF KP KPHQTOCVKQP VJGQT[ RQUUGUUKPI IQQF VJGQTGVKECN RTQRGTVKGU CPF WPKSWG KP OCP[ YC[U KV KU PQV DCUGF QP C FKTGEV OKP KOK\CVKQP QH VJG ENCUUKſECVKQP GTTQT TCVG CPF KU SWKVG FKHHGTGPV HTQO VJG /%' DCUGF CRRTQCEJ 6JG TGNCVKQP DGVYGGP //+ CPF /%' KU C XGT[ KPVGTGUVKPI VQRKE CPF UVWF KGF KP = ? +V KU HQWPF VJCV WPFGT EGTVCKP EQPFKVKQPU FKTGEV EQORCTKUQPU ECP DG OCFG DGVYGGP VJGUG VYQ CRRTQCEJGU 6JG FKUEWUUKQP DGNQY KU DCUGF QP =? 9G FGTKXG VJG GZRNKEKV TGNCVKQPU DGVYGGP /%' CPF //+ CPF HTQO VJGTG RTQRGTVKGU QH DQVJ CRRTQCEJGU ECP DG KNNWUVTCVGF .GV WU CUUWOG 
   
     C ECUG QH WUKPI C WPKHQTON[ FKUVTKDWVGF NCPIWCIG OQFGN 6JG OWVWCN KPHQTOCVKQP HTQO 'S  KU IKXGP D[

 




    

   







     




CPF VJG //+ OQFGN RCTCOGVGT GUVKOCVKQP ETKVGTKQP KU

 

   


       


   

















      




 6JG EQTTGURQPFKPI /%' CRRTQCEJ WUKPI VJG UCOG 

  CPF 
  CU KP 'S

 JCU VJG HQNNQYKPI HQTO

‹E\&5&3UHVV//&




¯ 6JG OKUENCUUKſECVKQP OGCUWTG




    

   




  

    









¯ 6JG /%' OQFGN RCTCOGVGT GUVKOCVKQP KU     



 








YJGTG





   %QPUKFGT VJG URGEKCN ECUG QH 

 

   
 


 

YKVJ 








VJG HQNNQYKPI CNIGDTCKE TGNCVKQPU ECP DG FGTKXGF




   
      

   

  



 




 









  

  


 



    CPF






   



  


   

   







(TQO 'S  VJG NQICTKVJO QH VJG OWVWCN KPHQTOCVKQP ECP DG GZRTGUUGF DCUGF QP VJG OKUENCUUKſECVKQP OGCUWTG


  QH 'S  KP VJG /%' HQTOWNCVKQP



     

    




    



   



     



   
  



 

 



   


(TQO 'S  KV ECP DG UGGP VJCV VJG NQUU HWPEVKQP KP //+ KU XGT[ FKHHGTGPV HTQO VJG QPG WUGF KP /%' CRRTQCEJ +P CFFKVKQP VQ C EQPUVCPV UJKHV    KV KU VJG NQIC TKVJO QH VJG UKIOQKF HWPEVKQP 
 QP VJG OKUENCUUKſECVKQP OGCUWTG


  PQV QP VJG UKIOQKF HWPEVKQP KVUGNH CU KP /%' CRRTQCEJ 6JG F[PCOKE TCPIG QH VJKU NQUU HWPEVKQP KU HTQO   VQ  ½ +V KU CRRCTGPV VJCV VJG //+ QDLGEVKXG HWPEVKQP KU PQV VT[KPI VQ CRRTQZKOCVG VJG TGEQIPKVKQP GTTQT TCVG HWPEVKQP CPF KVŏU QRVKOCNKV[ VQ ENCUUKſGT FGUKIP ECPPQV DG FKTGEVN[ GUVCDNKUJGF HTQO VJG GTTQT TCVG OKPKOK\CVKQP EQPUKFGTCVKQP (TQO VJG FKUETKOKPCPV HWPEVKQP RQKPV QH XKGY //+ KU VQ OKPKOK\G VJG CXGTCIG OKU ENCUUKſECVKQP OGCUWTG


  KH 

     KP 'S  KU CRRTQZKOCVGF D[


  6JKU KPVGTRTGVCVKQP KU SWKVG FKHHGTGPV HTQO VJG RQUVGTKQT RTQDCDKNKV[ DCUGF KPVGTRTGVCVKQP FGUETKDGF KP 'S  YJKEJ KU DCUGF QP FKUVTKDWVKQPCN CUUWORVKQPU +V GZRNCKPU VJG GZRGTKOGPVCN TGEQIPKVKQP RGTHQTOCPEG KORTQXGOGPVU QDVCKPGF HTQO VJG //+ CRRTQCEJ GXGP KP VJG ECUG YJGTG VJG FKUVTKDWVKQPCN CUUWORVKQP KU MPQYP PQV XCNKF

‹E\&5&3UHVV//&

0.6

0.5

0.4

0.3

0.2

0.1

0

-0.1 -10

-8

-6

-4

-2

0

2

4

6

8

10

FIGURE 1.1 A plot of the value of the derivative of the sigmoid function. $CUGF QP 'SU  CPF  VJG /%' QDLGEVKXG HWPEVKQP KU U[OOGVTKECN CTQWPF VJG OKUENCUUKſECVKQP OGCUWTG


  YJGTGCU VJG QDLGEVKXG HWPEVKQP KP //+ KU CU[O OGVTKECN (QT EQTTGEV TGEQIPKVKQP YJGTG


    CPF 

  KU JKIJGT VJCP VJG CX GTCIG KPEQTTGEV EQORGVKVKXG ECPFKFCVGU DQVJ VJG /%' CPF //+ QDLGEVKXG HWPEVKQPU CTG DQWPFGF 6JG[ FKHHGT VJQWIJ KP VJGKT UGPUKVKXKV[ VQ VJG UKIP EJCPIGU KP


  6JG QDLGEVKXG HWPEVKQP KP VJG /%' CRRTQCEJ KU FKTGEVN[ TGNCVGF VQ VJG UKIP EJCPIGU KP VJG OKUENCUUKſECVKQP OGCUWTG


  +P VJG //+ QDLGEVKXG HWPEVKQP VJG UKIP EJCPIG KP


  YKNN PQV NGCF VQ C EQTTGURQPFKPI EJCPIG KP VJG UKIP QH


   

 WPNGUU


  KU UOCNNGT VJCP


 5KPEG  KU VJG PWODGT QH ENCUUGU VJKU ECP JCRRGP KP //+ DCUGF CRRTQCEJ QPN[ KH


    9JGP


    KV KPFK ECVGU C TGEQIPKVKQP GTTQT KU EQOOKVVGF D[ VJG TGEQIPK\GT QP VJG TCPFQO KPRWV   6JG QDLGEVKXG HWPEVKQP KP VJG /%' CRRTQCEJ KU DQWPFGF PQ OCVVGT VJG XCNWG QH


  #U C EQPVTCUV VJG QDLGEVKXG HWPEVKQP KP //+ KU PQV DQWPFGF HQT


    6JKU DGJCXKQT OC[ JCXG UQOG CFXGTUG GHHGEVU KP //+ DCUGF RCTCOGVGT GUVKOCVKQP UKPEG KV KU DCUGF QP VJG OWVWCN KPHQTOCVKQP 

  CXGTCIGF QXGT VJG GPVKTG VTCKPKPI UGV (WTVJGT KPUKIJVU ECP DG ICKPGF D[ GZCOKPKPI VJG ITCFKGPV QH VJG QDLGEVKXG HWPEVKQPU CUUQEKCVGF YKVJ VJGUG VYQ CRRTQCEJGU 6JG ITCFKGPV QH VJG QDLGEVKXG HWPEVKQP KP /%' CRRTQCEJ JCU VJG HQNNQYKPI HQTO



 


  YKVJ

‹E\&5&3UHVV//&











 



 




 
 
   
 

 



    



¼

YJGTG

 KU VJG FGTKXCVKXG QH VJG UKIOQKF HWPEVKQP 1P VJG QVJGT JCPF VJG ITCFKGPV QH VJG QDLGEVKXG HWPEVKQP KP VJG //+ CRRTQCEJ DCUGF QP VJG OKUENCUUKſECVKQP OGCUWTG 

  KU 

   
 





 





   





+V ECP DG UGGP VJCV VJG ITCFKGPV HWPEVKQP KP VJG /%' CRRTQCEJ KU DCUGF QP VJG FKH HGTGPVKCVGF UKIOQKF HWPEVKQP YJKEJ KU EQPEGPVTCVGF QP VJG ENCUU FGEKUKQP DQWPFCT[ 

    6JG CDUQNWVG XCNWG QH VJG ITCFKGPV HWPEVKQP FGETGCUGU OQPQVQPKECNN[ KH VJG XCNWG QH 

  OQXGU CYC[ HTQO VJG FGEKUKQP DQWPFCT[ 1P VJG QVJGT JCPF VJG ITCFKGPV QH VJG QDLGEVKXG HWPEVKQP KP //+ CRRTQCEJ KU VJG UKIOQKF HWPEVKQP KVUGNH C HWPEVKQP YJKEJ KU OQPQVQPKE KPETGCUKPI CPF RWVU GORJCUKU QP GZVTGOG HCNUG ENCUUK ſECVKQPU 

    'ZVTGOG HCNUG ENCUUKſECVKQPU CTG V[RKECNN[ QWVNKGTU 9KVJQWV RTQRGT EQPVTQN VJG RCTCOGVGT GUVKOCVKQP ECP DG UVTQPIN[ KPƀWGPEGF D[ VJG QWVNKGTU CPF VJG GUVKOCVKQP TGUWNVU OC[ DGEQOG DKCUGF 6JKU RTQDNGO ECP DG CEWVG KP URGGEJ TGEQIPKVKQP UKPEG UWEJ QWVNKGTU CTG QHVGP HTQO YTQPI NCDGNKPI CPF GZVTGOG OKUOCVEJ KP CEQWUVKE EQPFKVKQPU 2CTCOGVGT GUVKOCVKQP KP FKUETKOKPCPV HWPEVKQP DCUGF CRRTQCEJ KU VQ ſPF CP QRVKOCN RCTVKVKQP QH VJG UCORNG URCEG UWEJ VJCV VJG TGEQIPKVKQP GTTQT TCVG ECP DG TGFWEGF 6JGTGHQTG C TQDWUV VTCKPKPI CNIQTKVJO DCUGF QP VJG /%' ETKVG TKQP UJQWNF DG OQTG UGPUKVKXG QP VJG EJCPIGU KP VJG FGEKUKQP DQWPFCTKGU UKPEG VJGUG EJCPIGU JCXG FKTGEV KORCEV QP VJG TGEQIPKVKQP GTTQT TCVG +V KU ENGCT HTQO 'S  VJCV VJKU RTQRGTV[ KU GODGFFGF KP VJG UKIOQKF HWPEVKQP WUGF KP VJG /%' DCUGF CR RTQCEJ $CUGF QP 'S  CPF (KI  VJG OQFGN RCTCOGVGT CFLWUVOGPVU KP UKIOQKF HWPEVKQP DCUGF /%' CRRTQCEJ CTG OQFWNCVGF D[ VJG FGTKXCVKXG QH VJG UKI OQKF HWPEVKQP YJQUG XCNWG JCU C RGCM CTQWPF 

    VJG UKOWNCVGF FGEKUKQP DQWPFCT[ 6JG //+ CPF /%' ETKVGTKC JCXG CNUQ DGGP UVWFKGF GZRGTKOGPVCNN[ 5RGGEJ TGEQIPK VKQP GZRGTKOGPVCN TGUWNVU QH EQORCTKPI VJGUG VYQ CRRTQCEJGU CTG TGRQTVGF D[ UGXGTCN UKVGU +P RCTVKEWNCT C UKFG D[ UKFG UVWF[ YCU IKXGP KP =? +P VJG UVWF[ //+ CPF /%' ENCUUKſGT VTCKPKPI YGTG RGTHQTOGF DCUGF QP KFGPVKECN GZRGTKOGPVCN UGVWRU CPF WUKPI VJG UCOG ITQYVJVTCPUHQTOCVKQP DCUGF QRVKOK\CVKQP OGVJQF HQT RCTCOGVGT GU VKOCVKQP +V YCU HQWPF VJCV DQVJ //+ CPF /%' ECP NGCF VQ URGGEJ TGEQIPKVKQP RGTHQTOCPEG KORTQXGOGPVU QXGT VJG /. DCUGF CRRTQCEJ CPF VJG CDUQNWVG GTTQT TCVG TGFWEVKQP KP VJG /%' CRRTQCEJ KU   XGTUWU  KP VJG //+ CRRTQCEJ +V UJQWNF DG RQKPVGF QWV VJCV CNVJQWIJ GZRNKEKV TGNCVKQP DGVYGGP VJG //+ CPF /%' CRRTQCEJGU ECP DG GUVCDNKUJGF WUKPI VJG OKUENCUUKſECVKQP OGCUWTG CPF WPFGT UQOG URGEKCN EQP FKVKQPU VJG VJGQTGVKECN UVWF[ CU YGNN CU OQTG GZVGPUKXG GZRGTKOGPVCN UVWFKGU DGVYGGP VJGUG VYQ ETKVGTKC CTG HCT HTQO EQORNGVG CPF OCP[ SWGUVKQPU TGOCKP VQ DG CPUYGTGF

1.5.6 Discussions and Comments 6JG /%' CRRTQCEJ FGUETKDGF KP VJKU UGEVKQP KU C FKUETKOKPCPV HWPEVKQP DCUGF CR RTQCEJ VQ RCVVGTP ENCUUKſECVKQP 6JG FGEKUKQP TWNG QH VJG ENCUUKſGT KU VTGCVGF CU C FKU ETKOKPCPV HWPEVKQP CPF VJG RCTCOGVGT GUVKOCVKQP KPXQNXGU OKPKOK\KPI VJG GZRGEVGF NQUU KPEWTTGF YJGP VJGUG FGEKUKQP TWNGU CTG CRRNKGF KP VJG ENCUUKſGT 6JG HQTO QH VJG

‹E\&5&3UHVV//&

NQUU HWPEVKQP KU ETKVKECN KP FKUETKOKPCPV HWPEVKQP DCUGF ENCUUKGT FGUKIP +P VJG /%' CRRTQCEJ VJG NQUU HWPEVKQP KU EQPUVTWEVGF KP UWEJ C YC[ VJCV VJG TGEQIPKVKQP GTTQT TCVG QH VJG ENCUUKſGT KU GODGFFGF KP C UOQQVJ HWPEVKQPCN HQTO CPF OKPKOK\KPI VJG GZRGEVGF NQUU QH VJG ENCUUKſGT JCU C FKTGEV TGNCVKQP VQ VJG ENCUUKſGT GTTQT TCVG TGFWE VKQP 6JKU FKTGEV TGNCVKQP VQ TGEQIPKVKQP GTTQT TCVG KP VJG /%' CRRTQCEJ JCU UGXGTCN CFXCPVCIGU KP ENCUUKſGT FGUKIP


+V KU OGCPKPIHWN KP VJG UGPUG QH OKPKOK\KPI VJG GORKTKECN TGEQIPKVKQP GTTQT TCVG QH VJG ENCUUKſGT CPF VJKU RTQRGTV[ KU PQV FGRGPFGPV QP VJG RCTCOGVTKE HQTO QH VJG FKUETKOKPCPV HWPEVKQP PQT KVU TGNCVKQP VQ VJG HQTO QH VJG VTWG ENCUU RQUVGTKQT FKUVTKDWVKQP
+H VJG VTWG ENCUU RQUVGTKQT FKUVTKDWVKQPU CTG WUGF CU FKUETKOKPCPV HWPEVKQPU VJG CU[ORVQVKE DGJCXKQT QH VJG ENCUUKſGT YKNN CRRTQZKOCVG VJG OKPKOWO $C[GU TKUM 6JG FKUETKOKPCPV HWPEVKQP HQTOWNCVKQP KP VJG /%' CRRTQCEJ OCMGU KV CRRNKECDNG VQ XCTKQWU HWPEVKQPCN FGEKUKQP TWNGU KPENWFKPI VJQUG FGEKUKQP TWNGU YJKEJ CTG PQV DCUGF QP RTQDCDKNKV[ HWPEVKQPU UWEJ CU IGPGTCNK\GF NKPGCT FKUETKOKPCPV HWPEVKQPU GVE +V CNUQ CRRNKGU VQ ECUGU YJGTG VJG RCTCOGVTKE HQTO QH VJG FKUETKOKPCPV HWPEVKQP KU MPQYP VQ DG FKHHGTGPV HTQO VJG VTWG ENCUU RQUVGTKQT FKUVTKDWVKQPU QT VJG FKUETKOKPCPV HWPEVKQP KU UGNGEVGF HTQO QVJGT EQPUKFGTCVKQPU UWEJ CU OCVJGOCVKECN VTCEVCDKNKV[ CPF CNIQTKVJOKE EQORNGZKV[ 6JG /. DCUGF FKUVTKDWVKQP GUVKOCVKQP CRRTQCEJ VQ RCVVGTP TGEQIPKVKQP KU HTQO C FKH HGTGPV RGTURGEVKXG 6JG RTQDCDKNKV[ FKUVTKDWVKQP 2& QH VJG TCPFQO UQWTEG HQT TGEQI PKVKQP KU GUVKOCVGF D[ CVVTKDWVKPI VQ VJG UQWTEG C RCTCOGVTKE OQFGN 2& CPF GUVKOCVKPI RCTCOGVGTU QH VJKU 2& HTQO IKXGP VTCKPKPI FCVC 6JG QRVKOCN /#2 FGEKUKQP TWNG KU CR RNKGF VQ VJG GUVKOCVGF OQFGN 2&U CU KH VJG[ YGTG VJG VTWG RTQDCDKNKV[ OGCUWTGU 5WEJ CP CRRTQCEJ KU TGHGTTGF VQ CU VJG őRNWIKPŒ OGVJQF KP UVCVKUVKECN NKVGTCVWTG &GPQVG VJG FKUVTKDWVKQPU HTQO VJG UQWTEG CU
CPF VJG FKUVTKDWVKQPU HTQO VJG RCTCOGVTKE OQFGN CU   6JG UVCVKUVKEU QH VJG UQWTEG IGPGTCVKPI VJG VTCKPKPI FCVC CTG PQV PGEGUUCTKN[ VJQUG QH VJG OQFGNU CPF VJG QRVKOCNKV[ TGUWNVU QH VJG $C[GU ENCUUKſGT ECPPQV DG CRRNKGF FKTGEVN[ +V KU UVWFKGF KP =? VJCV CRRTQCEJGU QH /. //+ ECP DG HQTOWNCVGF CU ECUGU KP VJG /&+ CRRTQCEJ CPF FKHHGTGPV CUUWORVKQPU CTG OCFG CDQWV VJG VTWG 2&U QH VJG UQWTEG VQ DG OQFGNGF CPF VJG 2&U VJCV CTG WUGF VQ OQFGN VJG UQWTEG 6JG /&+ KPVGT RTGVCVKQP QH VJG /. CRRTQCEJ KU VJCV /. GUVKOCVKQP QH RCTCOGVGTU KP VJG OQFGN 2&U HQT C IKXGP UQWTEG KU GSWKXCNGPV VQ CRRTQZKOCVKPI VJG GORKTKECN FKUVTKDWVKQP QH VJG UQWTEG QP VJG VTCKPKPI FCVC D[ 2&U QH VJG OQFGN KP VJG /&+ KG -WNNDCEM.GKDNGT FKUVCPEG QT TGNCVKXG GPVTQR[ UGPUG +P VJG /. DCUGF CRRTQCEJ VJG /&+ OGCUWTG VQ DG OKPKOK\GF JCU VJG HQNNQYKPI HQTO



  


   


















  


 



YJGTG 
 KU VJG RTKQT RTQDCDKNKV[ QH VJG OVJ ENCUU CPF 




  


  KU VJG -WNNDCEM.GKDNGT FKUVCPEG DGVYGGP VJG GORKTKECN FKUVTKDWVKQP QH VJG UQWTEG

‹E\&5&3UHVV//&

CPF VJG FKUVTKDWVKQP QH VJG RCTCOGVTKE OQFGN GUVKOCVGF HTQO VJG VTCKPKPI FCVC HQT VJG IKXGP VTCKPKPI UCORNG EQPFKVKQPGF QP VJG ENCUU NCDGN =? 6JWU C IQQFPGUU ETKVGTKQP HQT VJG /. GUVKOCVG KU KPVTQFWEGF +H VJG 2&U QH VJG OQFGN KPENWFG VJG VTWG 2&U QH VJG UQWTEG CU[ORVQVKECNN[ KV YKNN NGCF VQ C $C[GU ENCUUKſGT *QYGXGT UWEJ RTQRGTVKGU OC[ GZKUV QPN[ WPFGT C OQFGN EQTTGEVPGUU CUUWORVKQP VJCV VJG VTWG 2&U QH VJG UQWTEG CTG EQXGTGF KP VJG 2&U QH VJG RCTCOGVTKE OQFGN +H VJG 2&U QH VJG OQFGN CTG TKEJ GPQWIJ VQ RTQXKFG C IQQF CRRTQZKOCVKQP VQ VJG VTWG 2&U QH VJG UQWTEG YKVJ UWHſEKGPV VTCKPKPI FCVC VJG TGEQIPKVKQP RGTHQTOCPEG OC[ KORTQXG CU KV TGUWNVU KP C DGVVGT CRRTQZKOCVKQP VQ VJG VTWG 2&U QH VJG UQWTEG VJTQWIJ VJG GORKTKECN FKUVTKDWVKQP DCUGF QP VJG VTCKPKPI FCVC $WV KH VJG 2&U QH VJG OQFGN CTG SWKVG FKHHGTGPV EQORCTKPI VQ 2&U QH VJG UQWTEG VJG CEJKGXCDNG TGEQIPK VKQP RGTHQTOCPEG OC[ DG XGT[ NKOKVGF WUKPI VJG FKUVTKDWVKQP GUVKOCVKQP CRRTQCEJ CPF FKUETKOKPCPV HWPEVKQP DCUGF ENCUUKſGT FGUKIP UJQWNF DG OQTG CRRTQRTKCVG +P URGGEJ TGEQIPKVKQP OCP[ CUUWORVKQPU QP VJG OQFGN 2&U CTG OCFG TGICTFKPI VJG URGGEJ IGPGTCVKQP RTQEGUU 5RGGEJ CU C UKIPCN UQWTEG HQT RCVVGTP TGEQIPKVKQP OC[ PQV DG /CTMQXKCP PQT UJQWNF KV DG VJG ECUG VJCV EQPFKVKQPGF QP C IKXGP UVCVG VJG QDUGTXCVKQP 2&U UJQWNF DG CP KKF RTQEGUU 6JG UWEEGUU QH *//U KP URGGEJ TGEQI PKVKQP UJQWNF PQV DG EQPUVTWGF CU VJCV VJG 2&U QH VJG OQFGN EQXGT VJG VTWG 2&U QH VJG UQWTEG +P HCEV 2&U HTQO *//U CTG SWKVG NKOKVGF EQORCTKPI VQ VJG UQWTEG 2&U QH URGGEJ #NVJQWIJ VJG PCVWTG QH UQWTEG 2&U CTG WPMPQYP CPF PQ HWPFCOGPVCN CEJKGX CDNG TGEQIPKVKQP DQWPFU UKOKNCT VQ 5JCPPQP DQWPFU KP EQFKPI VJGQT[ CTG CXCKNCDNG GZRGTKOGPVCN TGUWNVU KPFKECVGF VJCV FKUETKOKPCPV HWPEVKQP DCUGF /%' CRRTQCEJ ECP NGCF VQ UKIPKſECPV KORTQXGOGPVU KP TGEQIPKVKQP RGTHQTOCPEG QXGT VJG /. DCUGF CR RTQCEJ 6JG UKIPKſECPEG QH VJG /%' CRRTQCEJ KP URGGEJ TGEQIPKVKQP KU VYQHQNF (KTUV C ENCUUKſGT FGUKIP DCUGF QP FKTGEV OKPKOK\CVKQP QH VJG TGEQIPKVKQP GTTQT TCVG KU C OGCPKPIHWN CNVGTPCVKXG VQ FKUVTKDWVKQP GUVKOCVKQP DCUGF CRRTQCEJ 5GEQPF VJG 2&U WUGF KP RCTCOGVTKE OQFGNKPI QH URGGEJ CTG XGT[ NKOKVGF EQORCTGF VQ VJG VTWG 2&U KP VJG UQWTEG CPF VJG FGEKUKQP TWNG DCUGF QP FKUETKOKPCPV HWPEVKQP CRRTQCEJ KU C TGCUQPCDNG CNVGTPCVKXG VQ VJG őRNWIKPŒ /#2 TWNG YJKEJ KU DCUGF QP VJG OQFGN EQTTGEVPGUU CUUWORVKQP






1.6 Embedded String Model Based MCE Training +P VJG CDQXG OGPVKQPGF FGXGNQROGPV QH VJG /%' VTCKPKPI HQTOCNKUO VJG WVVGTCPEG ENCUUGU (QT TGEQIPKVKQP QH EQP QDUGTXCVKQP KU CUUWOGF VQ DG HTQO QPG QH VJG VKPWQWU URGGEJ QT HQT URGGEJ TGEQIPKVKQP WUKPI UWDYQTF OQFGN WPKVU KU C EQPECVG PCVGF UVTKPI QH QDUGTXCVKQPU DGNQPIKPI VQ FKHHGTGPV ENCUUGU (QT GZCORNG C UGPVGPEG KU C UGSWGPEG QH YQTFU GCEJ QH YJKEJ KU VQ DG OQFGNGF D[ C FKUVTKDWVKQP 6JG FGEQF KPI RTQEGUU KP EQPVKPWQWU URGGEJ TGEQIPKVKQP KU VQ EQORCTG KORNKEKVN[ CNN RQUUKDNG

YQTF QT UWDYQTF UVTKPI OQFGNU CPF VJG YQTF UVTKPI YJQUG UVTKPI OQFGN JCU VJG JKIJGUV NKMGNKJQQF UEQTG KU EJQUGP CU VJG FGEQFGF UVTKPI 6JG NKMGNKJQQF UEQTG QH




‹E\&5&3UHVV//&






FIGURE 1.2 A structure diagram of a context dependent head-body-tail digit model in speech recognition. VJG YQTF UVTKPI KU V[RKECNN[ C EQODKPCVKQP QH UEQTGU HTQO XCTKQWU OQFGNU KPENWFKPI VJG UEQTG HTQO VJG CEQWUVKE OQFGN NCPIWCIG OQFGN FWTCVKQP OQFGN GVE 6JG OCKP TGCUQP QH CFQRVKPI VJKU V[RG QH UVTKPI OQFGN KU UKORN[ VJCV VJG DCUKE URGGEJ TGEQI PKVKQP OQFGN WPKVU YJKEJ CTG WUGF VQ HQTO UVTKPI OQFGNU ECP DG GUVKOCVGF HTQO C ſPKVG COQWPV QH CXCKNCDNG VTCKPKPI FCVC # RTQJKDKVKXG PWODGT QH YQTF UVTKPIU ECP DG IGPGTCVGF GXGP HTQO C XGT[ NKOKVGF XQECDWNCT[ CPF VQ KPFKXKFWCNN[ OQFGN GCEJ YQTF UVTKPI KU PQV RTCEVKECN VQ KORNGOGPV HQT UVTKPIU YKVJ WPMPQYP NGPIVJ 1P VJG QVJGT JCPF NQPI VGTO NCPIWCIG OQFGNU CPF EQPVGZV FGRGPFGPV CEQWUVKE OQFGNU CTG WUGF GZVGPUKXGN[ KP URGGEJ TGEQIPKVKQP CPF RTQXKFG OWEJ JKIJGT TGUQNWVKQP HQT ENCUUKH[KPI CNNQRJQPKE CEQWUVKE CPF NKPIWKUVKE GXGPVU 6JG WUG QH VJGUG FGVCKNGF CPF NQPI VGTO MPQYNGFIG UQWTEGU KP URGGEJ TGEQIPKVKQP JCU GZVGPFGF VJG OQFGNKPI FGRGPFGPEKGU DG[QPF VJG NGXGN QH KPFKXKFWCN YQTFU VQ RJTCUG ITQWRU QT CV VJG YJQNG WVVGTCPEG NGXGN 6JGTGHQTG PGY HQTOWNCVKQPU CTG PGGFGF VQ GZVGPF VJG /%' CRRTQCEJ VQ ENCUUKſGT FGUKIP KP EQPVKPWQWU URGGEJ TGEQIPKVKQP +P VJKU UGEVKQP YG ſTUV FGUETKDG C IGPGTCN GODGFFGF UVTKPI OQFGN DCUGF /%' RCTCFKIO HQT EQPVKPWQWU URGGEJ TGEQIPKVKQP CPF HTQO VJGTG VJG /%' VTCKPKPI HQT GCEJ EQORQPGPV KP VJG WVVGTCPEG DCUGF UVTKPI OQFGN ECP DG CEJKGXGF WPFGT VJKU WPKſGF HTCOGYQTM

1.6.1 String Model Based MCE Approach &KUETKOKPCPV HWPEVKQPU DCUGF QP UVTKPI NGXGN OQFGNKPI CTG PGEGUUCT[ KP EQPVKPWQWU URGGEJ TGEQIPKVKQP DGECWUG VJG ENCUUKſGT FGEKUKQP TWNGU CTG DCUGF QP VJG YJQNG WV VGTCPEG NGXGN INQDCN OCVEJKPI 6JG UVTKPI OQFGN YJKEJ FGUETKDGU VJG IKXGP YQTF

‹E\&5&3UHVV//&

UVTKPI VJCV DGUV OCVEJGU VJG KPRWV URGGEJ WVVGTCPEG JCU VQ DG FGVGTOKPGF D[ 8KVGTDK CNKIPOGPV RTQEGUU DGVYGGP CNN RQUUKDNG UVTKPI OQFGNU CPF VJG KPRWV URGGEJ WVVGTCPEG

½
 (QT GCUG QH TGRTGUGPVCVKQP YG FTQR VJG PQPCEQWUVKE RCTVU KP VJG UVTKPI OQFGN ſTUV CPF EQPUKFGT VJGO UGRCTCVGN[ NCVGT 6JG UVTKPI OQFGN HQT C IKXGP YQTF UVTKPI KP CP *// DCUGF URGGEJ TGEQIPKVKQP U[UVGO WUKPI EQPVKPWQWU QDUGTXCVKQP FGPUKVKGU KU IKXGP D[




 




   
    







YJGTG  KU C RQUUKDNG UVTKPI OQFGN HQT YQTF UVTKPI 5   KU VJG QRVKOCN UVCVG UGSWGPEG KP VJG UVTKPI OQFGN QH   KU VJG OQFGN UGV QH CNN TGEQIPKVKQP OQFGN WPKVU CPF      KU VJG NQINKMGNKJQQF UEQTG CNQPI VJG QRVKOCN UVCVG UGSWGPEG   +P VJG GODGFFGF UVTKPI OQFGN DCUGF /%' VTCKPKPI FGUETKDGF KP =? VJG FKUETKOKPCPV HWPEVKQP CV VJG UVTKPI NGXGN KU DCUGF QP VJG UVTKPI OQFGN HQT VJG EQTTGEV YQTF UVTKPI CPF VJG UVTKPI OQFGNU QH VJG OQUV EQPHWUCDNG YQTF UVTKPIU QDVCKPGF WUKPI C HCUV VTGGVTGNNKU DGUV UGCTEJ =? .GV
½    
DG CP CTDKVTCT[ YQTF UVTKPI )KXGP VJG OQFGN UGV  VJG QRVKOCN UVCVG UGSWGPEG   KU C HWPEVKQP QH VJG QDUGTXCVKQP  CPF VJG YQTF UVTKPI KU QHVGP FGVGTOKPGF D[ C 8KVGTDK FGEQFKPI RTQEGUU 6JG VQR DGUV UVTKPI J[RQVJGUGU
½      ECP DG FGſPGF KPFWEVKXGN[ CU HQNNQYU

















 







½
   
       
   
      
½ 









½


      CTG


  
  
     

6JG FKUETKOKPCPV HWPEVKQPU HQT



YJGTG  KU VJG VJ DGUV UVTKPI KU VJG *// UGV WUGF KP VJG  DGUV FGEQFKPI  KU VJG QRVKOCN RCVJ UVCVG UGSWGPEG QH VJG VJ UVTKPI IKXGP VJG OQFGN UGV  CPF   
     KU VJG TGNCVGF NQINKMGNKJQQF UEQTG QP VJG QRVKOCN RCVJ QH VJG



VJ UVTKPI (QT VJG EQTTGEV UVTKPI   VJG FKUETKOKPCPV HWPEVKQP KU IKXGP D[




  
  
     



YJGTG  KU VJG EQTTGEV UVTKPI   KU VJG QRVKOCN CNKIPOGPV RCVJ CPF   
      KU VJG EQTTGURQPFKPI NQINKMGNKJQQF UEQTG 6JGUG FKUETKOKPCPV HWPEVKQPU CTG GODGFFGF KP VJG /%' DCUGF NQUU HWPEVKQP VJTQWIJ VJG HQNNQYKPI UVGRU

 6JG OKUENCUUKſECVKQP OGCUWTG KP GODGFFGF UVTKPI OQFGN DCUGF /%' VTCKPKPI KU FGſPGF CU






     
  

‹E\&5&3UHVV//&




 


   ½ 



5RGGEJ (GCVWTG

0$GUV5VTKPI *[RQVJGUGU&GEQFKPI

5EQTG1RVKOCN2CVJ QH0$GUV5VTKPI/QFGN

7RFCVGF *//U

5GIOGPVCN )2&6TCKPGT

*//U

FIGURE 1.3 A diagram of the embedded string model based MCE training process.  6JG NQUU HWPEVKQP KP OKPKOWO UVTKPI GTTQT TCVG VTCKPKPI KU FGſPGF CU

 



    




YJGTG  KU C RQUKVKXG EQPUVCPV YJKEJ EQPVTQNU VJG UNQRG QH VJG UKIOQKF HWPE VKQP  6JG GZRGEVGF NQUU YJKEJ KU CUUQEKCVGF YKVJ VJG UVTKPI GTTQT TCVG KU IKXGP D[ 




 

 



+V UJQWNF DG PQVGF VJCV VJG UKVWCVKQP KP EQPVKPWQWU URGGEJ TGEQIPKVKQP KU SWKVG FKHHGT GPV HTQO VJG ſPKVG ENCUU ENCUUKſECVKQP RTQDNGO YJGTG C ſZGF UGV QH FKUETKOKPCPV HWPEVKQPU ECP DG RTGURGEKſGF 6JG FKUETKOKPCPV HWPEVKQPU KP VJG GODGFFGF UVTKPI OQFGN DCUGF /%' CRRTQCEJ CTG F[PCOKE FGRGPFKPI QP VJG RCTVKEWNCT NGZKECN YQTF UVTKPI  TCPFQO KPRWV   CPF C NKUV QH VJG  OQUV EQORGVKVKXG UVTKPI OQFGNU 6JG  OQUV EQORGVKVKXG UVTKPI OQFGNU CNUQ FGRGPF QP VJG UVTKPI NGXGN OQFGN OCVEJKPI QH VJG WVVGTCPEG  CICKPUV VJG EWTTGPV OQFGN UGV  +P VJG /. DCUGF FKUVTKDWVKQP GUVKOCVKQP CRRTQCEJ VJG OQFGN RCTCOGVGTU CTG GUVKOCVGF QPN[ HTQO VJG VTCKPKPI FCVC YKVJ VJG EQTTGEV UVTKPI OQFGN 6JG FKUETKOKPCVKXG KPHQTOCVKQP GZKUVKPI KP VJG EQORGV KPI UVTKPI OQFGNU KU IGPGTCNN[ PQV WUGF 6JG WUG QH VJG UGSWGPVKCN VTCKPKPI RTQEGFWTG DCUGF QP )2& CNIQTKVJO HQT RCTCOGVGT CFCRVCVKQP CNUQ OCMGU VJKU VTCKPKPI RTQEG FWTG őUGIOGPVCNŒ KP VJG UGPUG VJCV VJG UVCVG UGIOGPVCVKQP QH VJG URGGEJ WVVGTCPEG KU WUGF VQ WRFCVG VJG EWTTGPV OQFGN CPF VJG WRFCVGF OQFGN KU WUGF VQ KPVTQFWEG PGY UGIOGPVCVKQP HQT VJG PGZV VTCKPKPI UCORNG

‹E\&5&3UHVV//&

1PG QH VJG KUUWGU KP CEQWUVKE OQFGNKPI KU JQY VQ OQFGN VJG YQTF UVTKPIU VJCV CTG PQV KP VJG VTCKPKPI UGV +P EQPVKPWQWU URGGEJ TGEQIPKVKQP VJG EQXGTCIG QH VJG VTCKPKPI OCVGTKCN QP VJG RQUUKDNG YQTF UVTKPIU KU CNYC[U NKOKVGF IKXGP VJG HCEV VJCV C JWIG PWODGT QH YQTF UVTKPIU ECP QEEWT KP VJG NCPIWCIG 6JGUG WPUGGP UVTKPIU CTG KP IGP GTCN XGT[ JCTF VQ OQFGN CPF /. GUVKOCVKQP KU DCUGF QP VJG UGGP VTCKPKPI FCVC CPF ECPPQV EQXGT VJG ECUGU YJKEJ CTG WPUGGP 6JG WUG QH EQORGVKPI UVTKPI OQFGNU KP /%' VTCKPKPI RTQXKFGU C DGVVGT EQXGTCIG QH YQTF UVTKPIU UKPEG OCP[ QH VJGO OC[ PQV CEVWCNN[ QEEWT KP VJG VTCKPKPI FCVC 6JQUG EQPHWUCDNG YQTF UVTKPIU CTG UGNGEVGF DCUGF QP VJGKT EQPHWUKDKNKV[ YKVJ VJG EQTTGEV NGZKECN UVTKPI IKXGP VJG EWTTGPV OQFGN UGV 6JG[ CTG WUGF VQ HQTO VJG UVTKPI OQFGN DCUGF FKUETKOKPCPV OGCUWTG CPF OQFGNGF KP VJG UOQQVJ /%' DCUGF NQUU HWPEVKQP YJKEJ TGNCVGU VQ UVTKPI GTTQT TCVG # FKCITCO QH VJG GODGFFGF UVTKPI OQFGN DCUGF /%' VTCKPKPI KU IKXGP KP (KI  6JG GODGFFGF UVTKPI OQFGN DCUGF /%' CRRTQCEJ KU YGNN UWKVGF HQT CEQWUVKE OQFGN KPI WUKPI FGVCKNGF EQPVGZV FGRGPFGPV OQFGNU YJGTG UGRCTCVG CEQWUVKE OQFGN WPKV KU WUGF VQ OQFGN RJQPGOG YKVJ FKHHGTGPV NGHV CPF TKIJV EQPVGZV +V ECP FGUETKDG XCTKQWU NQPI URCP NGHV CPF TKIJV EQPVGZV FGRGPFGPEKGU UWEJ CU VTKRJQPG SWKPRJQPG GVE 1PG GZCORNG QH C ETQUUYQTF EQPVGZV FGRGPFGPV OQFGN WUGF KP EQPPGEVGF FKIKV TGEQIPK VKQP KU FGRKEVGF KP (KI  YJKEJ JCU C HWNN GZRCPUKQP HQT CNN RQUUKDNG NGHV CPF TKIJV EQPVGZVU CV VJG YQTF DQWPFCTKGU 6JG KPVTQFWEVKQP QH VJG GODGFFGF UVTKPI OQFGN KP /%' VTCKPKPI JCU VYQ CFXCPVCIGU

¯ +V GZVGPFU VJG /%' DCUGF FKUETKOKPCPV HWPEVKQP CRRTQCEJ VQ EQPVKPWQWU URGGEJ TGEQIPKVKQP YJGTG OQFGNKPI GCEJ KPFKXKFWCN YQTF UVTKPI ENCUU KU PQV HGCUKDNG ¯ +V RTQXKFGU CP GZCEV GOWNCVKQP QH VJG ENCUUKſGT KP EQPVKPWQWU URGGEJ TGEQIPK VKQP CPF GODGFU VJG WVVGTCPEG NGXGN URGGEJ OCPKHGUVCVKQP KP VJG DCUKE TGEQIPK VKQP OQFGN WPKVU +P GODGFFGF UVTKPI OQFGN DCUGF /%' CRRTQCEJ VJG NQPI VGTO FGRGPFGPEKGU CTG GODGFFGF KP VJG DCUKE URGGEJ TGEQIPKVKQP OQFGN WPKVU GXGP KH VJGKT QTKIKPCN EQPVGZV FGRGPFGPE[ FGſPKVKQPU CTG PQV +V KU QDUGTXGF KP VJG GZRGTKOGPVU VJCV OCP[ OQPQ RJQPG DCUGF EQPVGZV KPFGRGPFGPV OQFGN WPKVU QDVCKPGF HTQO VJG /%' CRRTQCEJ GZ JKDKV URGGEJ TGEQIPKVKQP RGTHQTOCPEG QH EQPVGZV FGRGPFGPV OQFGN WPKVU =? 6JG GODGFFGF UVTKPI OQFGN DCUGF /%' CRRTQCEJ HQWPF CRRNKECVKQPU KP XCTKQWU TGEQIPK VKQP VCUMU CPF UKIPKſECPV GTTQT TCVG TGFWEVKQP YGTG QDUGTXGF =     ? #NVJQWIJ VJG UVTKPI OQFGN DCUGF CRRTQCEJ KU VJG PCVWTCN EJQKEG HQT UVTKPI GTTQT TCVG OKPKOK\CVKQP KV KU RQUUKDNG VQ KPENWFG YQTF NGXGN GTTQT GHHGEVU KP /%' VTCKPKPI 1PG OQFKſECVKQP RTQRQUGF KP =? WUGU YQTF GTTQT EQWPVU CU VJG YGKIJVU DGVYGGP VJG EQTTGEV NGZKECN UVTKPI OQFGN CPF VJG OQUV EQPHWUCDNG UVTKPI OQFGN KP VJG OKUENCUUKſ ECVKQP OGCUWTG .GV VJG NGZKECN UVTKPI OQFGN DG

 CPF VJG OQUV EQPHWUCDNG UVTKPI OQFGN
½  VJG OKUENCUUKſECVKQP OGCUWTG YKVJ VJG YQTF GTTQT EQWPV YGKIJVKPI JCU VJG HQNNQYKPI HQTO


   LD


 
½ 
 


    

½  

 YJGTG LD


 
½  KU VJG UQ ECNNGF .GXGPUJVGKPFKUVCPEG DGVYGGP VJG EQTTGEV TGHGT GPEG YQTF UVTKPI

 CPF VJG TKXCN YQTF UVTKPI
½  KG VJG PWODGT QH GTTQTU EQP VCKPGF KP
½  6JG TGUV QH VJKU /%' HQTOWNCVKQP HQNNQYU VJG GODGFFGF UVTKPI GTTQT

‹E\&5&3UHVV//&

DCUGF /%' CRRTQCEJ *QYGXGT KV UJQWNF DG PQVGF VJCV OWNVKRN[KPI C RQUKVKXG EQP UVCPV QP VJG OKUENCUUKſECVKQP OGCUWTG

  FQGU PQV EJCPIG KVU UKIP PQT VJG UVTKPI GTTQT DCUGF /%' HQTOWNCVKQP 9JGP   VJG NQUU HWPEVKQP YKNN UVKNN EQPXGTIG VQ VJG UVTKPI GTTQT EQWPV HWPEVKQP CNVJQWIJ C YQTF GTTQT DCUGF YGKIJVKPI KU CRRNKGF




1.6.2 Combined String Model Based MCE Approach #U OGPVKQPGF CV VJG DGIKPPKPI QH VJKU UGEVKQP VJG ſPCN FGEKUKQP KP URGGEJ TGEQIPK VKQP KU DCUGF QP VJG EQODKPCVKQP QH UEQTGU HTQO XCTKQWU MPQYNGFIG UQWTEGU TGRTG UGPVGF D[ FKHHGTGPV OQFGNU #UUWOKPI KPFGRGPFGPEG QH GCEJ OQFGN VJG ſPCN UEQTG KP VJG NQICTKVJO FQOCKP DGEQOGU C UWO QH NQINKMGNKJQQF UEQTGU HTQO GCEJ KPFKXKF WCN OQFGN +P RCTVKEWNCT KP CFFKVKQP VQ VJG CEQWUVKE OQFGN KH C NCPIWCIG OQFGN KU WUGF CPF KVU UEQTG KU YGKIJVGF D[ C YGKIJVKPI HCEVQT    VJG ſPCN NKMGNKJQQF UEQTG QH C ECPFKFCVG UVTKPI KU  




   




    
 



+H VJG OQFGN EQTTGEVPGUU CUUWORVKQP KU XCNKF VJG NQINKMGNKJQQF UEQTG UJQWNF UVTKEVN[ HQNNQY 'S  CPF VJG UEQTG YGKIJVKPI HCEVQT     *QYGXGT KP URGGEJ TGEQI PKVKQP GZRGTKOGPVU CPF CRRNKECVKQPU KV KU HQWPF VJCV C XCNWG QH   YKVJ   FGOQPUVTCVGU OWEJ DGVVGT TGEQIPKVKQP RGTHQTOCPEG =? CP KPFKECVKQP VJCV VJG VTWG FKUVTKDWVKQP QH VJG UKIPCN UQWTEG FGRCTVU HTQO VJG CUUWORVKQP OCFG D[ VJG OQFGN 6JG NCPIWCIG OQFGN HCEVQT   KP URGGEJ TGEQIPKVKQP KU QHVGP VWPGF CPF CFLWUVGF DCUGF QP VJG TGEQIPKVKQP TGUWNVU QP VJG VTCKPKPI CPF FGXGNQROGPV FCVC 6JG CEVWCN XCNWG QH VJG NCPIWCIG OQFGN HCEVQT WUGF KP TGEQIPKVKQP KU SWKVG FKHHGTGPV HTQO VJG QPG FGTKXGF HTQO VJG OQFGN EQTTGEVPGUU CUUWORVKQP 6JG VWPKPI RTQEGFWTG KVUGNH DGKPI GORKTK ECN KU C FGRCTVWTG HTQO VJG FKUVTKDWVKQP DCUGF RCVVGTP TGEQIPKVKQP CRRTQCEJ CPF VJG UEQTG EQODKPCVKQP ECP DG EQPUKFGTGF CU C RTQDNGO QH UGNGEVKPI FKUETKOKPCPV HWPE VKQPU KP RCVVGTP ENCUUKſECVKQP 6JG KPVTQFWEVKQP QH GODGFFGF UVTKPI OQFGN OCMGU KV RQUUKDNG VQ GZVGPF VJG FKUETKOKPCPV HWPEVKQP DCUGF CRRTQCEJ VQ VJG NGXGN QH JCPFNKPI OWNVKOQFGN EQODKPCVKQPU CPF VQ VJG RCTCFKIO QH EQODKPGF UVTKPI OQFGN VTCKPKPI 6JG EQODKPGF UVTKPI OQFGN ECP DG FQPG KP VJG EQODKPCVKQP QH VJG HQNNQYKPI VYQ FKTGEVKQPU 1PG KU JQTK\QPVCN UEQTGU HTQO OWNVKRNG OQFGNU CPF FKHHGTGPV MPQYNGFIG UQWTEGU CTG EQODKPGF VQ HQTO VJG ſPCN UEQTG YJGTG GCEJ KPFKXKFWCN OQFGN OC[ DG GUVKOCVGF UGRCTCVGN[ DCUGF QP FKHHGTGPV GUVKOCVKQP OGVJQFU KPENWFKPI WUKPI FKHHGTGPV VTCKPKPI FCVC CPF EQPUVTCKPVU &KUETKOKPCVKXG OQFGN EQODKPCVKQP =? HCNNU KP VJKU ECV GIQT[ #PQVJGT KORQTVCPV FKTGEVKQP HQT OQFGN EQODKPCVKQP KU VQ GUVKOCVG VJG KPFKXKF WCN OQFGN RCTCOGVGTU KP VJG EQODKPGF UVTKPI OQFGN CU CP KPVGITCVGF EQORQPGPV QH VJG ſPCN EQODKPGF UVTKPI OQFGN &KUETKOKPCVKXG HGCVWTG GZVTCEVKQP =? FKUETKOKPCVKXG NCPIWCIG OQFGN GUVKOCVKQP = ? CPF GODGFFGF UVTKPI OQFGN DCUGF GUVKOCVKQP WUKPI OWNVKRNG MPQYNGFIG UQWTEGU =? CTG UWEJ CRRTQCEJGU KP YJKEJ VJG FKUETKOK PCPV HWPEVKQP KU EQPUVTWEVGF CV VJG EQODKPGF UVTKPI OQFGN NGXGN CPF VJG GUVKOCVKQP QH RCTCOGVGTU CV GCEJ KPFKXKFWCN OQFGN KU CEJKGXGF D[ VTCEKPI FQYP VJG OQFGN EQODKPC VKQP VTGG VQ GCEJ QH KVU NGCH PQFGU HQNNQYKPI C EJCKP TWNG NKMG TGNCVKQPUJKR #NVJQWIJ KV ECP DG EQORWVCVKQPCNN[ FGOCPFKPI VQ GUVKOCVG CNN OQFGN RCTCOGVGTU KP UWEJ C INQDCN



‹E\&5&3UHVV//&

OCPPGT VJG EQODKPGF UVTKPI OQFGN DCUGF CRRTQCEJ PGXGTVJGNGUU RTQXKFGU CP GZCEV EJCTCEVGTK\CVKQP QH VJG FGEKUKQP RTQEGUU HQT GXGP VJG OQUV UQRJKUVKECVGF TGEQIPKVKQP CRRNKECVKQPU CPF KV KU CRRNKGF UWEEGUUHWNN[ KP OCP[ URGGEJ TGEQIPKVKQP U[UVGOU +P QTFGT VQ TGFWEG VJG EQORWVCVKQPCN EQORNGZKV[ OQFGN VTCKPKPI ECP DG FQPG KP C UGNGE VKXG YC[ YJGTG UQOG RQTVKQP QH VJG EQODKPGF UVTKPI OQFGN KU CUUWOGF ſZGF YJKNG GUVKOCVKPI RCTCOGVGTU KP QVJGT UGNGEVGF EQORQPGPVU QH VJG EQODKPGF UVTKPI OQFGN 6JG VTCKPKPI RTQEGUU KU QHVGP KVGTCVGF UGXGTCN VKOGU QP VJG VTCKPKPI FCVC YJGTG FKHHGT GPV OQFGN EQORQPGPVU CTG UGNGEVGF CV GCEJ KVGTCVKQP =? 6JKU KPVGITCVGF CRRTQCEJ YKNN DG HWTVJGT GZGORNKſGF KP VJG HQNNQYKPI UWDUGEVKQPU 1.6.2.1 Discriminative Model Combination 6JG CDKNKV[ VQ EQODKPG OWNVKRNG OQFGNU HTQO XCTKQWU MPQYNGFIG UQWTEGU KP URGGEJ TGEQIPKVKQP KU KORQTVCPV 6JKU KU DGECWUG URGGEJ KU C EQORNKECVGF UQWTEG CPF ECP DG CHHGEVGF D[ OCP[ HCEVQTU UWEJ CU EQPVGZV RTQUQFKEU XQECN VTCEV NGPIVJ CODK GPV GPXKTQPOGPV URGCMKPI UV[NG OQFG QH VJG URGCMGT CEEGPV GVE /WNVKRNG UKIPCN UQWTEGU HTQO OWNVKRNG UKIPCN DCPFU CTG CNUQ WUGF KP URGGEJ TGEQIPKVKQP =? /CP[ QH VJGUG OQFGNU QT MPQYNGFIG UQWTEGU OC[ PQV DG DCUGF QP RTQDCDKNKVKGU CPF C FKU ETKOKPCPV HWPEVKQP DCUGF CRRTQCEJ KU C UWKVCDNG EJQKEG HQT OQFGN EQODKPCVKQP .GV

½
  


 DG VJG KPFKXKFWCN OQFGN EQORQPGPVU KP VJG OQFGN EQODKPCVKQP 9G WUG VJG PQVCVKQP  
½
  


 VQ FGPQVG VJG EQODKPGF UVTKPI OQFGN IKXGP TCPFQO KPRWV   YJGTG  KU VJG HWPEVKQP UGNGEVGF HQT OQFGN EQODKPCVKQP +H ) KU NKPGCT



 


½
  


 



 




  


 




YJGTG   
  KU VJG UEQTG HTQO VJG  VJ OQFGN CPF   KU VJG OQFGN EQODKPCVKQP YGKIJVU &KUETKOKPCVKXG OQFGN EQODKPCVKQP DCUGF QP VJG /%' CRRTQCEJ KU VQ GODGF VJG EQODKPGF UVTKPI OQFGN DCUGF FKUETKOKPCPV HWPEVKQP  

  


 KP VJG NQUU HWPEVKQP CPF GUVKOCVG VJG OQFGN EQODKPCVKQP HCEVQT  
  

CU RCTCOGVGTU KP VJG EQODKPGF UVTKPI OQFGN +P RCTVKEWNCT VJG OKUENCUUKſECVKQP OGCUWTG KP VJG EQODKPGF UVTKPI OQFGN DCUGF /%' CRRTQCEJ KU


   

 


 






  
 









    ½













CPF VJG NQUU HWPEVKQP KU FGſPGF CU


 






    






YJGTG KU C RQUKVKXG EQPUVCPV YJKEJ EQPVTQNU VJG UNQRG QH VJG UKIOQKF HWPEVKQP 6Q FGVGTOKPG OQFGN EQODKPCVKQP EQGHſEKGPVU OCP[ QRVKOK\CVKQP OGVJQFU ECP DG CRRNKGF VQ GUVKOCVG 
  

YJKEJ OKPKOK\G VJG GZRGEVGF NQUU 6JG RQRWNCT )2& CNIQTKVJO JCU C XGT[ UKORNG HQTO KP VJKU ECUG =? %QPUVTCKPVU QP VJG XCNWG QH OQFGN EQODKPCVKQP EQGHſEKGPVU ECP CNUQ DG CRRNKGF FWTKPI RCTCOGVGT QRVKOK\C VKQP FGRGPFKPI QP VJG PCVWTG QH VJG MPQYNGFIG UQWTEGU WUGF KP VJG EQODKPGF UVTKPI

‹E\&5&3UHVV//&

OQFGN 5KPEG GUVKOCVKPI VJG OQFGN EQODKPCVKQP YGKIJVU KU C TGNCVKXGN[ UKORNG EQP UVTCKPGF QRVKOK\CVKQP RTQDNGO OGVJQFU QH NKPGCT RTQITCOOKPI EQPLWICVG ITCFKGPV UGCTEJ GVE DGEQOG EQORWVCVKQPCNN[ CRRNKECDNG &KUETKOKPCVKXG OQFGN EQODKPCVKQP KU CRRNKGF KP OCP[ CRRNKECVKQPU WPFGT VJG PCOG QH EQODKPGF UVTKPI OQFGN = ? FKUETKOKPCVKXG OQFGN EQODKPCVKQP =? CPF WPKXGTUCN UVQEJCUVKE GPIKPG =? 6JG /%' DCUGF FKUETKOKPCPV HWPEVKQP DCUGF CRRTQCEJ RTQXKFGU C IQQFPGUU ETKVGTKQP HQT GUVKOCVKPI CPF CFLWUVKPI VJQUG őVWPKPI RCTCOGVGTUŒ KP URGGEJ TGEQIPKVKQP GURGEKCNN[ YJGP GKVJGT C OQFGN EQTTGEVPGUU CUUWORVKQP KU PQV XCNKF QT C WPKſGF HTCOGYQTM KU PGGFGF VQ EQODKPG MPQYNGFIG UQWTEGU VJCV CTG FKHHGTGPV KP QTKIKP QT PCVWTG +V UJQWNF DG RQKPVGF QWV VJCV FKUETKOKPCVKXG OQFGN EQODKPCVKQP KU XGT[ FKHHGTGPV HTQO OGVJ QFU WUGF VQ EQODKPG TGUWNVU HTQO OWNVKRNG TGEQIPK\GTU UWEJ CU 418'4 =? +P 418'4 KV KU DCUGF QP C XQVKPI UEJGOG CPF KV WVKNK\GU VJG FKXGTUKV[ QH VJG TGEQIPK VKQP GTTQTU HTQO KPFGRGPFGPV TGEQIPKVKQP U[UVGOU VQ KORTQXG VJG TGEQIPKVKQP RGTHQT OCPEG &KUETKOKPCVKXG OQFGN EQODKPCVKQP EQODKPGU FKHHGTGPV MPQYNGFIG UQWTEGU KPVQ QPG FKUETKOKPCPV HWPEVKQP 6JG EQORQPGPV KP VJG EQODKPGF UVTKPI OQFGN OC[ PQV DG CP KPFGRGPFGPV TGEQIPK\GT CPF KV ECP DG CP[ MPQYNGFIG UQWTEG TGNCVGF VQ VJG TCPFQO KPRWV : *QYGXGT KH GCEJ EQORQPGPV KP VJG EQODKPGF UVTKPI OQFGN KU C TGEQIPK\GT DQVJ OGVJQFU CRRN[ CPF KV KU CP KPVGTGUVKPI TGUGCTEJ VQRKE VQ UGG JQY VQ KPVGITCVG VJGO VQIGVJGT VQYCTFU C OQTG FKUETKOKPCVKXG EQODKPCVKQP DCUGF QP QWVRWVU HTQO OWNVKRNG TGEQIPKVKQP U[UVGOU 1.6.2.2 Discriminative Language Model Estimation .CPIWCIG OQFGNKPI KU C ETKVKECN EQORQPGPV KP URGGEJ TGEQIPKVKQP CPF HTQO VJG UQWTEG CPF OQFGN RQKPV QH XKGY KV RTQXKFGU VJG NCPIWCIG NGXGN OQFGNKPI QH VJG UQWTEG /QTGQXGT C NQV QH YQTFU KP URGGEJ CTG CEQWUVKECNN[ UKOKNCT CPF UQOG QH VJGO JQOQRJQPGU CTG GXGP KFGPVKECN UWEJ CU “too” CPF “two” +H QPN[ DCUGF QP CEQWUVKE KPHQTOCVKQP KFGPVKſECVKQP QH VJGUG YQTFU CPF RJTCUGU KP EQPVKPWQWU URGGEJ ECP DG XGT[ FKHſEWNV CPF QVJGT MPQYNGFIG UQWTEGU KP RCTVKEWNCT C NCPIWCIG OQFGN CTG PGGFGF # UVCVKUVKECN DCUGF
ITCO NCPIWCIG OQFGN KU C RQRWNCT EJQKEG KP URGGEJ TGEQIPKVKQP CPF KV JCU VJG HQNNQYKPI HQTO  ´


½      

·½ µ YJKEJ KU VJG GUVKOCVGF RTQDCDKNKV[ QH QDUGTXKPI YQTF   IKXGP VJG RCUV
 ½ YQTF JKUVQT[ $GECWUG VJG PWODGT QH RQUUKDNG
ITCO RTQDCDKNKVKGU ITQYU GZRQPGPVKCNN[ YKVJ VJG QTFGT
 NQYGT QTFGT NCPIWCIG OQFGNU UWEJ CU WPKITCO DKITCO VTKITCO CPF HQWT ITCO CTG WUGF KP XCTKQWU URGGEJ TGEQIPKVKQP VCUMU 6JG
ITCO UVCVKUVKECN NCPIWCIG OQFGN KU V[RKECNN[ GUVKOCVGF HTQO C NCTIG VGZV EQTRWU KPFGRGPFGPV QH QVJGT MPQYNGFIG UQWTEGU KP URGGEJ TGEQIPKVKQP 'XGP YKVJ C XGT[ NCTIG VGZV EQNNGEVKQP QPN[ RQTVKQPU QH WPKSWG VTKITCO CPF HQWTITCO GPVTKGU ECP DG GUVKOCVGF FWG VQ URCTUGPGUU QH VJG VTCKPKPI FCVC /CP[ NCPIWCIG OQFGN GPVTKGU TCTGN[ QEEWT KP VJG EQTRWU CPF DCEMQHH UEJGOGU GI -CV\ DCEMQHHU =? CTG WUGF VQ UWDUVKVWVG VJG WPUGGP NCPIWCIG OQFGN GPVTKGU YKVJ VJGKT NQYGT QTFGT DCEMQHH EQWPVGTRCTVU 6JG URCTUG FCVC RTQDNGO KU C UGTKQWU KUUWG KP NCPIWCIG OQFGN GUVKOCVKQP CPF KP QTFGT VQ IGPGTCVG OQTG GPVTKGU VJG UCORNG EQWPV EWVQHH VJTGUJQNF KP NCPIWCIG OQFGN GUVKOCVKQP KU WUWCNN[ UGV XGT[ NQY OCMKPI VJG GUVKOCVG HCT HTQO DGKPI TGNKCDNG +V KU CNUQ QDXKQWU HTQO VJG NCPIWCIG RQKPV QH XKGY VJCV CNVJQWIJ VJG UVCVKUVKECN
ITCO NCPIWCIG OQFGN KU SWKVG UWEEGUUHWN

‹E\&5&3UHVV//&

KP URGGEJ TGEQIPKVKQP KV KU PQV VJG őVTWGŒ UQWTEG OQFGN HQT URGGEJ +V KU QDXKQWU VJCV JWOCP NCPIWCIG JCU C OWEJ OQTG EQORNKECVGF UVTWEVWTG VJCV QHVGP ECPPQV DG EQXGTGF D[ VJG UKORNG UVTWEVWTG QH C ſZGF ITCO NCPIWCIG OQFGN 1PG FKTGEVKQP QH VJG EQODKPGF UVTKPI OQFGN DCUGF /%' CRRTQCEJ KU FKUETKOKPCVKXG NCPIWCIG OQFGN GUVKOCVKQP +PUVGCF QH VTGCVKPI NCPIWCIG OQFGN GUVKOCVKQP CU C FKU VTKDWVKQP GUVKOCVKQP RTQDNGO KV KU RTQRQUGF KP =? VQ HQTOWNCVG VJG NCPIWCIG OQFGN GUVKOCVKQP CU C RTQDNGO QH /%' VTCKPKPI CEEQTFKPI VQ VJG EQODKPGF UVTKPI OQFGN $[ GZRCPFKPI VJG NCPIWCIG OQFGN RCTV KP 'S  RCTCOGVGTU KP VJG NCPIWCIG OQFGN ECP DG GODGFFGF KP VJG EQODKPGF UVTKPI OQFGN DCUGF /%' HQTOWNCVKQP CPF VJGKT XCNWGU ECP DG GUVKOCVGF DCUGF QP VJG OKPKOK\CVKQP QH VJG GZRGEVGF NQUU 6JKU CR RTQCEJ FKHHGTU HTQO VJG EQPXGPVKQPCN FKUVTKDWVKQP DCUGF /. GUVKOCVKQP HQT NCPIWCIG OQFGNU (KTUV KV KU C FKUETKOKPCPV HWPEVKQP DCUGF CRRTQCEJ CPF DCUGF QP OKPKOK\C VKQP QH C URGEKCN NQUU HWPEVKQP YJKEJ TGNCVGU VQ TGEQIPKVKQP GTTQT TCVG 5GEQPF VJG NCPIWCIG OQFGN RCTCOGVGTU DGEQOG CP KPVGITCN RCTV QH VJG EQODKPGF UVTKPI OQFGN PQV KUQNCVG EQORQPGPVU FGTKXGF HTQO YQTF QEEWTTGPEG HTGSWGPEKGU +V KU EQPFKVKQPGF QP QVJGT MPQYNGFIG UQWTEGU UWEJ CU VJG CEQWUVKE OQFGN FWTCVKQP OQFGN GVE /CP[ EQPUVTCKPGF QRVKOK\CVKQP OGVJQFU ECP DG CRRNKGF VQ NCPIWCIG OQFGN GUVKOC VKQP KP VJG EQODKPGF UVTKPI OQFGN DCUGF CRRTQCEJ VQ OCKPVCKP RTQDCDKNKV[ EQPUVTCKPVU KPENWFKPI VTCPUHQTO DCUGF )2& CFCRVCVKQP ITQYVJ VTCPUHQTO DCUGF QRVKOK\CVKQP GVE +P VJG EQODKPGF UVTKPI OQFGN DCUGF CRRTQCEJ NCPIWCIG OQFGN GUVKOCVKQP FG RGPFU QP WVVGTCPEG NGXGN UVTKPI OQFGN OCVEJKPI QH CNN OQFGN EQORQPGPVU 6JG  OQUV EQPHWUKPI UVTKPI OQFGNU CTG FGVGTOKPGF WPFGT VJG INQDCN EQODKPGF UVTKPI OQFGN HTCOGYQTM (CUV YQTF ITCRJ UGCTEJ ECP DG WUGF VQ URGGF WR VJG UGCTEJ RTQEGUU QH RTGRCTKPI EQORGVKPI UVTKPI OQFGNU =? 5RCTUG VTCKPKPI FCVC KU CP KUUWG YJKEJ FGUGTXGU URGEKCN CVVGPVKQP +P QTFGT VQ CNNGXKCVG VJG URCTUG FCVC RTQDNGO FKUETKOKPC VKXG NCPIWCIG OQFGN VTCKPKPI ECP DG HQEWUGF QP VGTOU YKVJ UKIPKſECPV QEEWTTGPEGU QT NCPIWCIG OQFGN RCTCOGVGTU YJKEJ QEEWT QHVGP UWEJ CU NCPIWCIG OQFGN DCEMQHH YGKIJVU GVE = ? &KUETKOKPCVKXG NCPIWCIG OQFGN GUVKOCVKQP ECP CNUQ DG WUGF KP CRRNKECVKQPU YJGTG NCPIWCIG OQFGNU CTG WUGF CU ENCUUKſGTU UWEJ CU ECNN TQWVKPI =? CPF FKCNQIWG UVCVG KFGPVKſECVKQP =? 'ZRGTKOGPVCN TGUWNVU KPFKECVG VJCV FKU ETKOKPCVKXG NCPIWCIG OQFGN GUVKOCVKQP ECP KORTQXG NCPIWCIG OQFGN DCUGF ENCUUKſGTU EQORCTKPI VQ /. DCUGF NCPIWCIG OQFGN GUVKOCVKQP = ?

1.6.3 Discriminative Feature Extraction /QUV URGGEJ TGEQIPKVKQP U[UVGOU WUG UQOG V[RG QH URGEVTCN CPCN[UKU QP VJG YKPFQYGF TCY URGGEJ YCXGHQTO 5RGGEJ KU TGRTGUGPVGF CU C UGSWGPEG QH UJQTVVKOG RQYGT URGE VTC QT TGNCVGF TGEQIPKVKQP HGCVWTG XGEVQTU 6JG VYQ V[RGU QH URGEVTCN CPCN[UKU OGVJQFU OQUV HTGSWGPVN[ GORNQ[GF CTG ſNVGT DCPM CPCN[UKU CPF NKPGCT RTGFKECVKQP (KNVGT DCPM CRRTQCEJGU V[RKECNN[ WUG C DCPM QH DCPFRCUU ſNVGTU 6JG HTGSWGPE[ URCEKPI QH VJG ſNVGTU CTG GKVJGT WPKHQTO URCEGF QT ETKVKECNDCPFURCEGF HQNNQYKPI $CTM UECNG QT /GN UECNG 6JGUG ſNVGTU CTG IGPGTCNN[ JKIJN[ QXGTNCRRGF CPF EQXGT VJG TGNGXCPV HTGSWGPE[ TCPIG QH VJG KPRWV UKIPCN 6KOG CPF HTGSWGPE[ TGUQNWVKQP KU CP KORQTVCPV HCEVQT KP VJG ſNVGT DCPM FGUKIP 5RGEVTWO KPVGPUKV[ KU QHVGP UECNGF NQICTKVJOKECNN[ CPF VJG KFGC QH URGEVTWO YGKIJVKPI KU CNUQ WUGF VQ EQPVTQN VJG HGCVWTG UGPUKVKXKV[ *QYGXGT OQUV

‹E\&5&3UHVV//&

HGCVWTG GZVTCEVKQPU CTG DCUGF QP VJG CPCN[UKU QH JWOCP JGCTKPI ECRCDKNKVKGU CPF VJWU CTG PQV PGEGUUCTKN[ CRRNKECDNG VQ UVCVKUVKECNN[ DCUGF OCEJKPG TGEQIPKVKQP 6JG IQCN QH FKUETKOKPCVKXG HGCVWTG GZVTCEVKQP KU VQ CEEQORNKUJ VJG URGGEJ TGEQIPKVKQP HGCVWTG GZVTCEVKQP HTQO VJG UVCPFRQKPV QH OKPKOK\KPI VJG TGEQIPKVKQP GTTQT TCVG HQT ENCUUKſECVKQP D[ OCEJKPGU +P RNCEG QH VJG $CTM UECNG HTQO JGCTKPI C PGY HTGSWGPE[ UECNKPI =? ECP DG FGTKXGF DCUGF QP VJG ENCUUKſGT KORNGOGPVGF D[ OCEJKPGU CPF QVJGT QRGTCVKQPU KP HGCVWTG GZVTCEVKQP ECP DG OCFG FKUETKOKPCVKXG DCUGF QP VJG EQO DKPGF UVTKPI OQFGN /%' RCTCFKIO 5KPEG VJGUG URGGEJ TGEQIPKVKQP HGCVWTG XGEVQTU CTG RCTV QH VJG EQODKPGF UVTKPI OQFGN C IQQFPGUU ETKVGTKQP QH VJKU CRRTQCEJ ECP DG FGTKXGF HTQO VJG TGNCVKQP QH OKPKOK\CVKQP QH VJG GZRGEVGF NQUU CPF TGEQIPKVKQP GTTQT TCVG KP VJG /%' HQTOWNCVKQP 1PG CRRNKECVKQP QH FKUETKOKPCVKXG HGCVWTG GZVTCEVKQP KU KP VJG FGUKIP QH EGRUVTCN NKHVGTU =? %QPUKFGT C UGSWGPEG QH URGGEJ TGEQIPKVKQP HGCVWTG XGEVQTU DCUGF QP VJG EGRUVTCN XGEVQTU GZVTCEVGF HTQO C UJQTVVKOG URGEVTCN CPCN[UKU QH VJG URGGEJ YCXGHQTO +V KU YGNN MPQYP VJCV RJQPGOG ENCUU KFGPVKV[ WUGHWN VQ URGGEJ TGEQIPKVKQP GZKUVU NQECNN[ KP VJG NQY HTGSWGPE[ EGRUVTCN EQGHſEKGPVU 6JGTGHQTG URGGEJ TGEQIPK\GTU UGNGEVKXGN[ WUG VJKU PCTTQY TGIKQP QH EGRUVTCN EQORQPGPVU D[ CRRN[KPI EGRUVTCN YGKIJVKPI QT NKVVGTKPI DCUGF QP C YKPFQYKPI HWPEVKQP QT C NKHVGT

 6JG FGUKIP QH EGRUVTCN NKHVGT KU VQ EQPVTQN VJG PQPKPHQTOCVKQP DGCTKPI EGRUVTCN XCTKCDKNKVKGU KP QTFGT VQ RGTHQTO TGNKCDNG FKUETKOKPCVKQP QH UQWPFU 1PG RQRWNCT V[RG QH VJG NKHVGT HQT EGRUVTWO HGCVWTGU KU DCUGF QP C TCKUGF UKPG HWPEVKQP QH VJG HQTO



 






 

 HQT          HQT 
   

YJGTG  KU WUWCNN[ EJQUGP CU   CPF  KU V[RKECNN[    HQT URGGEJ QH M*\ DCPFYKFVJ 6JG YGKIJVGF HGCVWTG UGSWGPEG


HTQO VJG EGRUVTCN NKHVGT EQTTG URQPFU VQ C UOQQVJGF NQI RQYGT URGEVTWO 6JG LWUVKſECVKQP QH VJKU V[RG QH NKHVGT HQT EGRUVTCN HGCVWTG XGEVQTU KU IKXGP KP =? +P FKUETKOKPCVKXG HGCVWTG GZVTCEVKQP NKHVGT FGUKIP ECP DG FQPG CEEQTFKPI VQ VJG EQODKPGF UVTKPI OQFGN HQTOWNCVKQP CPF KPUVGCF QH TGN[KPI QP JWOCP JGCTKPI ECRCDKNKVKGU HQT EJQQUKPI VJG TKIJV NKHVGT VJG NKHVGT RC TCOGVGTU ECP DG GUVKOCVGF WUKPI VJG /%' ETKVGTKQP VQ OKPKOK\G VJG TGEQIPKVKQP GTTQT TCVG 5RGGEJ GZRGTKOGPVU WUKPI FKUETKOKPCVKXG HGCVWTG GZVTCEVKQP DCUGF NKHVGT YGTG RGTHQTOGF KP UGXGTCN VCUMU = ?

1.7 Verification and Identification 5RGCMGT XGTKſECVKQP CPF KFGPVKſECVKQP DCUGF QP XQKEG KU CP KORQTVCPV CTGC KP URGGEJ TGUGCTEJ CPF JCU DGGP UVWFKGF HQT UGXGTCN FGECFGU 6JG IGPGTCN RTQDNGO QH RCVVGTP XGTKſECVKQP ECP DG HQTOWNCVGF CU HQNNQYU IKXGP C TCPFQO KPRWV UKIPCN   YG YCPV VQ XGTKH[ KH VJG UKIPCN  KU HTQO C UKIPCN UQWTEG ¼  +P OCMKPI C FGEKUKQP TGICTFKPI VJG QTKIKP QH VJG UKIPCN UQWTEG VYQ V[RGU QH GTTQTU ECP QEEWT 1PG EQWNF OKUVCMGPN[

‹E\&5&3UHVV//&






FGEKFG VJCV KU PQV HTQO VJG UKIPCN UQWTEG ¼  YJKNG VJG VTWG UQWTEG QH VJG UKIPCN KU ¼  6JKU V[RG QH GTTQT KP XGTKſECVKQP KU TGHGTTGF VQ CU C V[RG + GTTQT VJG GTTQT QH HCNUG TGLGEVKQP QT OKUUGF FGVGEVKQP 6JG UGEQPF V[RG QH GTTQT KU VJCV KU CEEGRVGF CU EQOKPI HTQO VJG UKIPCN UQWTEG ¼ YJKNG VJG VTWG UQWTEG QH UKIPCN KU PQV ¼  6JKU V[RG QH GTTQT KU QHVGP TGHGTTGF VQ CU C V[RG ++ GTTQT QT VJG GTTQT QH HCNUG CEEGRVCPEG 6JG RGTHQTOCPEG QH C XGTKſECVKQP U[UVGO KU V[RKECNN[ GXCNWCVGF DCUGF QP VJG EQODK PCVKQP QH V[RG + CPF V[RG ++ GTTQTU 6JG RTQDNGO QH XGTKſECVKQP ECP DG EQPXGPKGPVN[ HQTOWNCVGF KPVQ C UVCVKUVKECN J[RQVJGUKU VGUVKPI RTQDNGO IKXGP VJG VGUV UKIPCN  YG YCPV VQ VGUV VJG PWNN J[RQVJGUKU ¼  CICKPUV VJG CNVGTPCVKXG J[RQVJGUKU ½  YJGTG KU HTQO VJG UQWTEG ¼  CPF ½ CUUWOGU VJCV KU IGPGTCVGF D[ ¼ CUUWOGU VJCV CPQVJGT UQWTEG ½  +P OCP[ CRRNKECVKQPU VJG CNVGTPCVKXG J[RQVJGUKU ½ CUUWOGU QPN[ KU PQV IGPGTCVGF HTQO VJG MPQYP UQWTEG ¼  CPF KP UWEJ UKVWCVKQPU ½ KU C VJCV EQORQUKVG J[RQVJGUKU CU QRRQUGF VQ DGKPI C UKORNG J[RQVJGUKU +P IGPGTCN C VGUV RTQEGFWTG FKXKFGU VJG UKIPCN URCEG
KPVQ VYQ TGIKQPU
CPF




 CPF YG TGLGEV ¼ KH 
CPF CEEGRV ¼ KH 

KU QHVGP TGHGTTGF VQ CU VJG ETKVKECN TGIKQP QH VJG VGUV 6JG RTQDCDKNKVKGU QH VJGUG VYQ V[RGU QH GTTQTU ECP DG GZRTGUUGF CU































 











  
 


 ½ 
 
 
 ¼ CPF






 ¾
 
 
 ½

 
 
 ½



6JG RQYGT QH VJG VGUV YJKEJ KU CP KORQTVCPV SWCPVKV[ VQ EJCTCEVGTK\G VJG VGUV KU IKXGP D[
 
 ½




  

+P UVCVKUVKECN J[RQVJGUKU VGUVKPI QPG KU QHVGP KPVGTGUVGF KP ſPFKPI VJG ETKVKECN TGIKQP


UWEJ VJCV VJG RQYGT QH VJG VGUV KU OCZKOK\GF QT KP QVJGT YQTFU VJG V[RG ++ GTTQT

KU OKPKOK\GF CV C IKXGP NGXGN QH V[RG + GTTQT # VGUV YJKEJ KU QRVKOCN KP VJKU UGPUG KU QHVGP TGHGTTGF VQ CU VJG OQUV RQYGTHWN VGUV 6JGTG CTG RNGPV[ QH UVWFKGU CXCKNCDNG KP VJG UVCVKUVKECN NKVGTCVWTG TGICTFKPI VJG FGUKIP QH VJG QRVKOCN VGUVU KH   ¼  CPF   ½  CTG MPQYP CPF HCNN KPVQ UQOG URGEKſE FKUVTKDWVKQPU UWEJ CU VJG GZRQPGPVKCN HCOKN[ =? +P RTCEVKEG VJG VGUV RTQEGFWTG KU QHVGP DCUGF QP C VGUV UVCVKUVKEU   UWEJ VJCV ¼ KU TGLGEVGF KH     #EEQTFKPI VQ VJG 0G[OCP2GCTUQP NGOOC   ECP DG DCUGF QP RTQDCDKNKV[ TCVKQ VGUV  
  ½    ¼  QT NKMGNKJQQF TCVKQ VGUV  
  ½    ¼  CPF KU UGNGEVGF UWEJ VJCV VJG NGXGN QH VJG V[RG + GTTQT      ¼ 
 *QYGXGT KP OQUV RTCEVKECN XGTKſECVKQP RTQDNGOU YG JCXG PQ GZCEV MPQYNGFIG TGICTFKPI VJG FKUVTKDWVKQPU QH PWNN CPF CNVGTPCVKXG J[RQVJGUGU 6JKU RTQDNGO KU GXGP OQTG CEWVG HQT VJG URGGEJ UKIPCN YJKEJ KU PQPUVCVKQPCT[ CPF VJG GZCEV PCVWTG QH URGGEJ IGPGTCVKQP RTQEGUU KU UVKNN NCTIGN[ WPMPQYP /QTGQXGT RCTCOGVGTU QH VJG URGGEJ OQFGN CTG GUVKOCVGF HTQO XGT[ URCTUG FCVC RQKPVU EQNNGEVGF HTQO MPQYP UQWTEGU 9KVJ VJG EQTTGEVPGUU QH VJG OQFGN KP SWGUVKQP CPF VJG GUVKOCVKQP GTTQTU FWG VQ URCTUG VTCKPKPI UCORNGU HQT VJG RCTCOGVGTU QH VJG OQFGN VJG QRVKOCNKV[ QH VJG VGUV KP VJG ENCUUKECN UGPUG ECPPQV DG TGCNK\GF CPF FKUETKOKPCPV HWPEVKQP DCUGF OGVJQFU ECP DG WUGF VQ KORTQXG VJG






















 




 
  
  

‹E\&5&3UHVV//&

5RGGEJ (GCVWTG

5RGCMGT6GUV5EQTG 'XCNWCVKQP

+FGPVKV[ %NCKO

#EEGRV4GLGEV

*[RQVJGUKU 6GUVKPI

5RGCMGT 6JGUJQNFU

5RGCMGT /QFGN

FIGURE 1.4 Block diagram of a speaker verification system XGTKſECVKQP RGTHQTOCPEG =  ? +P VJG HQNNQYKPI UWDUGEVKQPU OQTG FGVCKNGF FKUEWUUKQPU CTG IKXGP VQ VJGUG CRRTQCEJGU

1.7.1 Speaker Verification and Identification #WVJGPVKECVKQP D[ XQKEG JCU XCTKQWU CRRNKECVKQPU KP JWOCPOCEJKPG EQOOWPKEC VKQP &GRGPFKPI QP CRRNKECVKQP TGSWKTGOGPVU KV ECP DG ENCUUKſGF KPVQ VYQ ECVGIQTKGU PCOGN[ URGCMGT XGTKſECVKQP CPF URGCMGT KFGPVKſECVKQP 5RGCMGT XGTKſECVKQP KPXQNXGU XGTKH[KPI VJG KFGPVKV[ QH C ENCKOGF URGCMGT HTQO C MPQYP URGCMGT RQRWNCVKQP CPF URGCMGT KFGPVKſECVKQP KPXQNXGU KFGPVKH[KPI CP WPMPQYP URGCMGT HTQO C MPQYP RQRW NCVKQP # V[RKECN URGCMGT XGTKſECVKQP U[UVGO KU UJQYP KP (KI  )KXGP C UGSWGPEG CPF VJG ENCKOGF URGCMGT KFGPVKV[  VJG VGUV UVCVKUVKEU QH URGGEJ HGCVWTG XGEVQTU UEQTG 

 ECP DG EQORWVGF HTQO VJG EQTTGURQPFKPI URGCMGT OQFGN 
 6JG VGUV UEQTG KU VJGP EQORCTGF YKVJ C VJTGUJQNF
CUUQEKCVGF YKVJ VJG ENCKOGF URGCMGT VQ FGEKFG KH VJG ENCKOGF KFGPVKV[ UJQWNF DG CEEGRVGF QT TGLGEVGF 5RGCMGT OQFGNKPI KU VJG OQUV ETKVKECN RCTV QH C URGCMGT XGTKſECVKQP U[UVGO #NVJQWIJ VJG URGCMGTŏU URGGEJ RTQFWEVKQP RTQEGUU ECP DG OQFGNGF FKTGEVN[ DCUGF QP VJG RJ[U KQNQIKECN UVTWEVWTG QH VJG URGCMGTŏU CTVKEWNCVQT[ CRRCTCVWU UWEJ CU VJG UJCRG QH VJG XQECN VTCEV GVE KV KU SWKVG FKHſEWNV KP RTCEVKEG VQ WPKSWGN[ GZVTCEV UWEJ UVTWEVWTCN RC TCOGVGTU HTQO URGGEJ UCORNGU +PUVGCF KPFKTGEV URGCMGT OQFGNKPI KU QHVGP WUGF KP YJKEJ C UGV QH URGGEJ OQFGNU KU ETGCVGF HQT GCEJ URGCMGT DCUGF QP C EQNNGEVKQP QH URGCMGT URGEKſE URGGEJ VTCKPKPI FCVC 6JGUG URGGEJ OQFGNU EJCTCEVGTK\G VJG CEQWUVKE OCPKHGUVCVKQP QH URGGEJ HQT C IKXGP URGCMGT YJKEJ ECP DG FQPG DCUGF QP XCTKQWU ETKVGTKC CPF FGRGPF QP VJG V[RG QH VJG XGTKſECVKQP UVTCVGIKGU +P QTFGT VQ OQFGN VJG VGORQTCN UVTWEVWTG KP URGGEJ *//U CTG VJG OQUV RQRWNCT EJQKEG HQT URGCMGT OQF




‹E\&5&3UHVV//&








GNKPI CNVJQWIJ QVJGT OQFGNKPI VGEJPKSWGU UWEJ CU 83 EQFGDQQM PGWTCN PGVYQTMU GVE ECP CNUQ DG WUGF *//U CTG WUGF KP URGCMGT XGTKſECVKQP VQ OQFGN VJG IGP GTCN VTGPF QH VJG URGGEJ HTQO C MPQYP URGCMGT CPF FGRGPFKPI QP CRRNKECVKQPU CPF VJG CXCKNCDKNKV[ QH CPPQVCVGF VTCKPKPI FCVC KV ECP DG DCUGF QP ſZGF RJTCUGU DTQCF RJQPGVKE ENCUUGU YJQNG YQTFU QT GXGP UWDYQTFU +P URGCMGT XGTKſECVKQP VJG FGEKUKQP KU DCUGF QP VJG UEQTG QH VJG VGUV UVCVKUVKEU

  (QT URGCMGT KFGPVKſECVKQP RTQDNGO VJG URGCMGT  KU KFGPVKſGF CU VJG VTWG URGCMGT KH



 

  




(QT URGCMGT XGTKſECVKQP YG WUWCNN[ CEEGRV VJG ENCKOGF URGCMGT KFGPVKV[  KH VJG VGUV UVCVKUVKEU HQT  VJ URGCMGT

     

 6JG VGUV UVCVKUVKEU

   ECP DG FKTGEVN[ DCUGF QP VJG NKMGNKJQQF UEQTG QH VJG URGGEJ  KP  VJ URGCMGTŏU OQFGN


 *QYGXGT VJG PQTOCNK\GF UEQTG HWPEVKQP YJKEJ KU C HQTO QH VJG IGPGTCNK\GF NKMGNKJQQF TCVKQ VGUV






 IKXGU C OWEJ DGVVGT URGCMGT XGTKſECVKQP RGTHQTOCPEG =? 6JG NKMGNKJQQF



 OQFGNU VJG CEQWUVKE URCEG YJKEJ FQGU PQV DGNQPI VQ URGCMGT   CPF KU QHVGP QDVCKPGF HTQO VJG URGCMGT EQJQTV OQFGNKPI (QT GCEJ URGCMGT   C UGV QH URGCMGTU YJQ CTG OQUV ENQUG VQ URGCMGT   KU ECNNGF C EQJQTV UGV 
 CPF ECP DG KFGPVKſGF HTQO VJG VTCKPKPI FCVC 6JG NKMGNKJQQF



 KU OQFGNGF CU C HWPEVKQP QH VJG NKMGNKJQQF HTQO EQORGVKPI

  KP VJG EQJQTV UGV 
 )QQF URGCMGT XGTKſECVKQP RGTHQTOCPEG KU QDUGTXGF YJGP


 KU CRRTQZKOCVGF D[





  

   ¾



6JG FKUETKOKPCVKXG HWPEVKQP DCUGF /%' CRRTQCEJ ECP DG CRRNKGF WUKPI C UWKVCDNG OKUENCUUKſECVKQP OGCUWTG CEEQTFKPI VQ VJG VGUV UVCVKUVKEU =? (QT URGCMGT XGTKſEC VKQP VJG XGTKſECVKQP GTTQT ECP CNUQ DG EJCTCEVGTK\GF D[ C OKUXGTKſECVKQP OGCUWTG DCUGF QP VJG VGUV UVCVKUVKEU

   KP C YC[ UKOKNCT VQ VJG OKUENCUUKſECVKQP OGCUWTG KP TGEQIPKVKQP CPF VJG /%' DCUGF FKUETKOKPCPV HWPEVKQP CRRTQCEJ ECP DG FKTGEVN[ CFCRVGF VQ OKPKOK\G VJG XGTKſECVKQP GTTQT = ? +P HCEV VJG EQJQTV OQFGNKPI HQT OWNCVKQP JCU C ENQUG TGNCVKQP VQ VJG OKUENCUUKſECVKQP OGCUWTG KP /%' CRRTQCEJ 'Z RGTKOGPVCN TGUWNVU KPFKECVGF VJCV VJG FKUETKOKPCPV HWPEVKQP DCUGF CRRTQCEJ VQ URGCMGT KFGPVKſECVKQP CPF XGTKſECVKQP ECP NGCF VQ C UKIPKſECPV TGFWEVKQP QH VJG QXGTCNN XGTKſ ECVKQP GTTQT 6JG URGCMGT VGUV UVCVKUVKEU CTG OWEJ DGVVGT UGRCTCVGF VJCP VJG /. DCUGF FKUVTKDWVKQP GUVKOCVKQP CRRTQCEJ =? 6JG U[UVGO UGPUKVKXKV[ VQ VJTGUJQNF UGNGEVKQP KU OWEJ TGFWEGF CPF TQDWUVPGUU QH VJG U[UVGO KU KORTQXGF = ? +P CFFKVKQP VQ CFFTGUUKPI VJG KFGPVKſECVKQP CPF XGTKſECVKQP RTQDNGO CU C URGEKCN ENCU UKſECVKQP RTQDNGO OGVJQFU QH KPVTQFWEKPI VJG UVTWEVWTG QH UVCVKUVKECN VGUV KP FKUETKOK PCPV HWPEVKQP DCUGF CRRTQCEJ CTG CNUQ CVVGORVGF /KPKOWO XGTKſECVKQP GTTQT /8' VTCKPKPI KU UWEJ CP CRRTQCEJ = ? 6JG OCKP FKHHGTGPEG VQ /%' CRRTQCEJ KU VJG WUG QH VYQ UGRCTCVG NQUU HWPEVKQPU VQ OQFGN VYQ V[RGU QH GTTQTU KP J[RQVJG UKU VGUVKPI 6JG FGVCKNU QH /8' CRRTQCEJ CTG FGUETKDGF DGNQY CPF KV GZGORNKſGU

‹E\&5&3UHVV//&

VJG FKUETKOKPCPV HWPEVKQP CRRTQCEJ VQ URGCMGT XGTKſECVKQP CPF KFGPVKſECVKQP 6JG OKUXGTKſECVKQP OGCUWTG CU QRRQUGF VQ VJG OKUENCUUKſECVKQP OGCUWTG HQT VJG
VJ URGCMGT KU FGſPGF CU






  




   

 


   

 

KH  HTQO VJG ENCKOGF
VJ URGCMGT KH  KU PQV HTQO VJG ENCKOGF
VJ URGCMGT

YJGTG 

  KU VJG NQINKMGNKJQQF UEQTG HTQO VJG ENCKOGF
VJ URGCMGT OQFGN CPF 

  KU VJG UEQTG HTQO VJG EQJQTV URGCMGT ITQWR 

 6JG OKUXGTKſECVKQP OGCUWTG KU GODGFFGF KP C UOQQVJ UKIOQKF DCUGF NQUU HWPEVKQP 

  CU KP VJG /%' HQTOWNCVKQP 6YQ UGRCTCVG NQUU HWPEVKQPU CTG WUGF VQ FGUETKDG VJG V[RG + CPF V[RG ++ GTTQTU 6JG CXGTCIG NQUU HQT GCEJ V[RG QH GTTQTU CRRTQZKOCVGU VJG GORKTKECN V[RG + CPF V[RG ++ GTTQT TCVG QP VJG VTCKPKPI UCORNGU ½
  

CPF 
  







½


½











  


¾








  


 %NCKOGF 5RGCMGT

 %NCKOGF 5RGCMGT





YJGTG 
 KU VJG KPFKECVQT HWPEVKQP 6JG QXGTCNN GZRGEVGF NQUU QH VJG /8' KU IKXGP D[
    
    
 

 YJGTG  CPF  CTG FGUKIP RCTCOGVGTU YJKEJ EQPVTQN VJG KPƀWGPEG QH V[RG + CPF V[RG ++ GTTQTU KP VJG QXGTCNN NQUU HWPEVKQP 6JG OQFGN RCTCOGVGT GUVKOCVKQP KP /8' VTCKP KPI KU VQ OKPKOK\G VJG GZRGEVGF NQUU QH 'S  YJKEJ TGNCVGU VQ VJG OKPKOK\CVKQP QH GORKTKECN GTTQT TCVG QH V[RG + CPF V[RG ++ GTTQTU 6JG IQQFPGUU QH VJKU ETKVGTKQP KU LWUVKſGF HTQO VJG FKUETKOKPCPV HWPEVKQP CRRTQCEJ CPF KV KU OGCPKPIHWN GXGP YJGP VJG OQFGN EQTTGEVPGUU CUUWORVKQP ECPPQV DG GUVCDNKUJGF 8CTKQWU URGCMGT XGTKſECVKQP GZ RGTKOGPVU CTG EQPFWEVGF CPF VJG FKUETKOKPCPV HWPEVKQP DCUGF CRRTQCEJ FGOQPUVTCVGU UKIPKſECPV RGTHQTOCPEG CFXCPVCIGU QXGT VJG FKUVTKDWVKQP GUVKOCVKQP DCUGF CRRTQCEJ =  ?

1.7.2 Utterance Verification 7VVGTCPEG XGTKſECVKQP KU VQ XGTKH[ VJG EQPVGPV QH VJG URGGEJ WVVGTCPEG CICKPUV C ENCKOGF

QT J[RQVJGUK\GF VGZV UVTKPI +V ECP DG WUGF VQ XGTKH[ VJG URGGEJ TGEQIPKVKQP TGUWNV CPF FGEKFG YJGVJGT VJG FGEQFGF YQTF UVTKPI KU TGNKCDNG CPF UJQWNF DG CEEGRVGF YKVJ EQPſFGPEG 1PG PGY CRRTQCEJ KP WVVGTCPEG XGTKſECVKQP KU VQ XGTKH[ VJG KPHQTOCVKQP EQPVGPV QH VJG URGGEJ WVVGTCPEG CICKPUV UQOG MPQYP FCVC UVQTGF KP VJG WUGT RGTUQPCN RTQſNG UWEJ CU DKTVJ FCVG QT CICKPUV C FCVCDCUG VQ YJKEJ VJG KPHQTOCVKQP EQPVGPV QH VJG WVVGTCPEG RTQXKFGF D[ VJG WUGT YKVJ ENCKOGF KFGPVKV[ UJQWNF OCVEJ 6JKU CRRTQCEJ QH XGTKH[KPI VJG EQPVGPV QH VJG WVVGTCPEG CICKPUV C MPQYP FCVCDCUG KU ECNNGF XGTDCN KP HQTOCVKQP XGTKſECVKQP 8+8 =? 8GTDCN KPHQTOCVKQP XGTKſECVKQP ECP DG CEJKGXGF

‹E\&5&3UHVV//&

YKVJQWV VJG PGGF QH EQNNGEVKPI URGCMGT URGEKſE VTCKPKPI FCVC 7VVGTCPEG XGTKſECVKQP CU C UVCVKUVKECN J[RQVJGUKU VGUVKPI RTQDNGO JCU C ENQUG TGNCVKQP VQ VJG RTQDNGO QH URGCMGT XGTKſECVKQP 6JG FKUETKOKPCPV HWPEVKQP CRRTQCEJ DCUGF QP /%' QT /8' ECP DG CRRNKGF VQ WVVGTCPEG XGTKſECVKQP YKVJ VJG UCOG HCUJKQP CU KV KU CRRNKGF VQ URGCMGT XGT KſECVKQP *QYGXGT VJG RWTRQUG QH WVVGTCPEG XGTKſECVKQP KU VQ XGTKH[ VJG KPHQTOCVKQP EQPVGPV QH VJG WVVGTCPEG PQV VJG KFGPVKV[ QH VJG URGCMGT #U C EQPUGSWGPEG KPUVGCF QH WUKPI VJG URGCMGT OQFGN VJG EQPſFGPEG UEQTGU TGICTFKPI VJG YQTF EQPVGPV KP VJG WVVGTCPEG CTG WUGF KP WVVGTCPEG XGTKſECVKQP 6JGTG CTG OCP[ YC[U VQ HQTO YQTF CPF UVTKPI NGXGN EQPſFGPEG OGCUWTG HQT WVVGTCPEG XGTKſECVKQP +P VJG CRRTQCEJ FGUETKDGF KP =? VJG YQTF NGXGN EQPſFGPEG UEQTG KU DCUGF QP VJG YQTF NGXGN NKMGNKJQQF TCVKQ FGſPGF CU HQNNQYU



 
 



¼

 

½





YJGTG ¼

 CPF ½

 CTG PWNN CPF CNVGTPCVKXG J[RQVJGUGU TGURGEVKXGN[ HQT XGT KH[KPI TCPFQO KPRWV  JCU YQTF EQPVGPV 
 6JG NKMGNKJQQF QH 

 ¼

  

  KU HTQO VJG OQFGN HQT 
 6JG NKMGNKJQQF HQT VJG CNVGTPCVKXG J[RQVJGUKU 

½

 KU HTQO C FKHHGTGPV OQFGN   YJKEJ KU OQFGNGF D[ VYQ UGRCTCVG *//U
     KU CP *// VTCKPGF QP CNN VJG VTCKPKPI FCVC KP VJG EQ JQTV UGV QH 
 +V KU WUGF VQ OQFGN VJG EQORQUKVG CEQWUVKE URCEG EQPUKUVU QH QVJGT YQTFU GZEGRV 
 CPF KV KU UQOGVKOGU ECNNGF CPVKOQFGN VQ YQTF 
  KU CP QVJGT *// YJKEJ KU C ſNNGT OQFGN VQ OQFGN PQPMG[YQTF GXGPV 6JG NKMGNKJQQF QH 

½

 DCUGF QP VJGUG VYQ V[RGU QH *//U KU IKXGP CU



½

  

   


   

  ½  

YJGTG  KU C RQUKVKXG EQPUVCPV +P FKUETKOKPCPV HWPEVKQP DCUGF WVVGTCPEG XGTKſECVKQP VJG OKUXGTKſECVKQP OGCUWTG KU


 
 
 
      
   



6JG OKUXGTKſECVKQP OGCUWTG KU GODGFFGF KP C UKIOQKF V[RG HWPEVKQP QH VJG HQTO



 
 



 


 
 




YJGTG KU C RQUKVKXG EQPUVCPV EQPVTQNNKPI VJG UNQRG QH VJG UKIOQKF HWPEVKQP CPF  VCMGU QP VJG XCNWG QH  CPF
CU HQNNQYU


   



KH 
KH 
KH 


     

YJGTG  TGHGTU VQ VJG ECUGU VJCV 
KU EQTTGEVN[ TGEQIPK\GF /4 TGHGTU VQ VJG ECUGU VJCV 
KU OKUTGEQIPK\GF CPF  TGHGTU VQ VJG ECUGU VJCV VJG KPRWV URGGEJ EQP VCKPU PQ MG[YQTF $CUGF QP 'S  VJG FKUETKOKPCPV HWPEVKQP DCUGF /%' CPF /8' CRRTQCEJ ECP DG CRRNKGF CU KP VJG URGCMGT XGTKſECVKQP 6JG CDQXG CRRTQCEJ

‹E\&5&3UHVV//&

ECP DG GZVGPFGF VQ UVTKPI NGXGN XGTKſECVKQP CPF DCUGF QP UWDYQTF WPKVU GPCDNKPI VJG XGTKſECVKQP RTQEGUU XQECDWNCT[ KPFGRGPFGPV =? 5VTKPI NGXGN XGTKſECVKQP OCMGU VJG ſPCN TGLGEVKQPCEEGRVCPEG FGEKUKQP QP VJG MG[YQTF J[RQVJGUKU OCFG D[ VJG TGE QIPK\GT #UUWOKPI KPFGRGPFGPEG VJG UVTKPI NGXGN NKMGNKJQQF TCVKQ ECP DG YTKVVGP CU C RTQFWEV QH UWDYQTFNGXGN NKMGNKJQQF TCVKQ



 ½ 
     
 



 








  


 





$[ EQNNGEVKPI VGTOU QH VJG PWOGTCVQT CPF FGPQOKPCVQT VJG OKUXGTKſECVKQP OGCUWTG ECP DG IKXGP CU HQNNQYU


       
 


  







 
  
  




YJGTG       
ECP DG DCUGF QP VJG YQTFU QT UWDYQTFU CEEQTFKPI VQ VJG OQFGNKPI UVTCVGI[ WUGF KP CRRNKECVKQP 6JG FKUETKOKPCPV HWPEVKQP DCUGF CRRTQCEJ ECP VJGP DG CRRNKGF VQ XGTKH[KPI YQTF UVTKPIU 6JG EQPUVTWEVKQP QH UVTKPI NGXGN EQPſFGPEG OGCUWTG KU CP CEVKXG TGUGCTEJ CTGC CPF XCTKQWU UVCVKUVKECN ETKVGTKC ECP CRRN[ 1PG ETKVGTKQP QHVGP WUGF KP UVTKPI NGXGN EQPſFGPEG OGCUWTG KU VJG OKPKOCZ RTKPEKRNG YJKEJ KU VQ OKPKOK\G VJG OCZKOWO TKUM QH CEEGRVKPI GCEJ KPFKXKFWCN ENCUU QT YQTF KP VJG UVTKPI 9QTF ENCUU FGRGPFGPV YGKIJVKPI ECP CNUQ DG WUGF YJKEJ IKXGU OQTG GORJCUKU VQ XGTKH[ UCNKGPV YQTFU CPF KPHQTOCVKQP DCTKPI ENCUUGU &KUEWUUKQPU QH HQTOKPI XCTKQWU V[RGU QH UVTKPI NGXGN EQPſFGPEG OGCUWTGU CTG IKXGP KP = ? CPF VJG TGHGTGPEGU EKVGF VJGTG 6JG KPVTQFWEVKQP QH XGTKſECVKQP RTQEGUU KP URGGEJ TGEQIPKVKQP QRGPU C RCTCFKIO QH CRRN[KPI VJG FKUETKOKPCPV HWPEVKQP DCUGF CRRTQCEJ KP URGGEJ RTQEGUUKPI +PUVGCF QH WUKPI OQFGN NKMGNKJQQF UEQTGU CU KU V[RKECN QPG CRRTQCEJ RTQRQUGF KP =? KU VQ WUG IGPGTCNK\GF EQPſFGPEG UEQTG HQT FGEQFKPI 6JG IGPGTCNK\GF EQPſFGPEG UEQTGU KU HQTOGF D[ KPVGITCVKPI XCTKQWU MPQYNGFIG UQWTEGU 'ZCORNGU QH UWEJ MPQYNGFIG UQWTEGU CTG HTCOG CEQWUVKE NKMGNKJQQF HTCOG CEQWUVKE NKMGNKJQQF TCVKQ RJQPG CPF YQTF FWTCVKQP RGPCNVKGU YQTF NCPIWCIG RTQDCDKNKVKGU YQTF KPUGTVKQP RGPCNVKGU HTCOG GPGTI[ RGPCNVKGU RTQUQFKE EQPſFGPEG UEQTG GVE # EQPſFGPEG UEQTG RTGRTQEGUUQT KU WUGF VQ EQPXGTV VJG EQPſFGPEG UEQTG HTQO GCEJ EQORQPGPV KPVQ C UWKVCDNG HQTO HQT EQODKPCVKQP +P RCTVKEWNCT VJG PQPNKMGNKJQQF TCVKQ DCUGF MPQYNGFIG UQWTEG KU EQP XGTVGF KPVQ VJG NQICTKVJO FQOCKP CPF KPVGITCVGF WUKPI C NKPGCT EQODKPCVKQP 6JG NKMGNKJQQF TCVKQ DCUGF MPQYNGFIG UQWTEG KU ſTUV GODGFFGF KP C UKIOQKF HWPEVKQP VQ EQPVTQN KVU F[PCOKE TCPIG CPF VJGP KV KU NKPGCT EQODKPGF KP VJG NQICTKVJO FQOCKP YKVJ QVJGT EQORQPGPVU $QVJ FKUETKOKPCVKXG OQFGN EQODKPCVKQP CPF FKUETKOKPCVKXG WVVGT CPEG XGTKſECVKQP CTG CRRNKGF VQ KPVGITCVG EQPſFGPEG UEQTGU HTQO FKHHGTGPV MPQYNGFIG UQWTEGU CPF VQ GUVKOCVG OQFGN RCTCOGVGTU CPF EQODKPCVKQP YGKIJVU KP VJG IGPGTCN K\GF EQPſFGPEG UEQTG DGECWUG VJG IGPGTCNK\GF EQPſFGPEG UEQTG OC[ PQV DG C RTQD CDKNKV[ DCUGF NKMGNKJQQF QT C RTQDCDKNKV[ FKUVTKDWVKQP /QTGQXGT VJG EQPſFGPEG UEQTG EQORQPGPVU KP VJG IGPGTCNK\GF EQPſFGPEG UEQTG ECP DG DCUGF QP FKHHGTGPV NGXGN QH KPHQTOCVKQP UWEJ CU HTCOG NGXGN UVCVG NGXGN RJQPG NGXGN YQTF NGXGN GVE &WTKPI

‹E\&5&3UHVV//&

VJG FGEQFKPI RTQEGUU VJQUG EQPſFGPEG UEQTG EQORQPGPVU CTG CRRNKGF CV FKHHGTGPV VKOGU CPF CV FKHHGTGPV NC[GTU QH VJG FGEQFKPI PGVYQTM CEEQTFKPI VQ VJGKT URGEKſEC VKQPU 6JGTGHQTG VJG XGTKſECVKQP QH VJG FGEQFGF RCTVKCN YQTF UVTKPI ECP DG FQPG CV FKHHGTGPV RJQPG YQTF CPF RJTCUG ITQWR LWPEVKQPU CEEQTFKPI VQ VJG XGTKſECVKQP DCUGF NKMGNKJQQF TCVKQ UEQTG EQORQPGPVU KP VJG IGPGTCNK\GF EQPſFGPEG UEQTG 6JG FGEQF KPI RTQEGUU QH VJKU CRRTQCEJ ECP DG RGTHQTOGF KP C VYQ RCUU HCUJKQP DCUGF QP C YQTF ITCRJ QT VJG DGUV NKUV QDVCKPGF HTQO VJG ſTUV RCUU UGCTEJ QT KP C QPG RCUU UGCTEJ YJKEJ CRRNKGU VJG IGPGTCNK\GF EQPſFGPEG UEQTG KP UGCTEJ FKTGEVN[ = ? #PQVJGT CRRTQCEJ QH WVKNK\KPI XGTKſECVKQP KP URGGEJ TGEQIPKVKQP CPF WPFGTUVCPFKPI KU DCUGF QP C FGVGEVKQP CPF XGTKſECVKQP UVTCVGI[ =? +P VJKU CRRTQCEJ C MG[ RJTCUG DCUGF FGVGEVKQP WUKPI C IGPGTCN CEQWUVKE RJQPGVKE OQFGN KU RGTHQTOGF ſTUV 6JG FGVGEVGF MG[ RJTCUGU CTG RTQEGUUGF KP C XGTKſECVKQP OQFWNG VQ XGTKH[ VJG FGVGEVGF MG[ RJTCUGU CPF GNKOKPCVG HCNUG CNCTOU 'CEJ MG[ RJTCUG KU VCIIGF YKVJ C UGOCPVKE VCI CPF VJG XGTKſGF MG[ RJTCUGU CTG EQPPGEVGF KPVQ UGPVGPEG J[RQVJGUKU WUKPI VCUM URGEKſE UGOCPVKE MPQYNGFIG # UVCEM FGEQFGT KU VJGP WUGF VQ UGCTEJ HQT VJG QRVKOCN J[RQVJGUKU VJCV UCVKUH[ VJG UGOCPVKE EQPUVTCKPVU 6JG QRVKOCN J[RQVJGUKU HTQO VJG UVCEM FGEQFGT KU HWTVJGT XGTKſGF CV VJG UGPVGPEG NGXGN DCUGF QP DQVJ CEQWUVKE CPF UG OCPVKE KPHQTOCVKQP HQT VJG ſPCN QWVRWV 6JG FKUETKOKPCPV HWPEVKQP DCUGF CRRTQCEJ KU CRRNKGF KP DQVJ MG[ RJTCUG FGVGEVKQP CPF XGTKſECVKQP VQ KORTQXG VJG U[UVGO RGT HQTOCPEG &GVCKNU QH VJKU CRRTQCEJ CTG IKXGP KP = ? CPF VJG TGHGTGPEGU EKVGF VJGTG



1.8 Summary +P VJKU EJCRVGT YG GZCOKPGF VJG ENCUUKECN $C[GU FGEKUKQP VJGQT[ CRRTQCEJ VQ VJG RTQDNGO QH RCVVGTP ENCUUKſECVKQP CPF FKUEWUUGF VJG KORNKGF CUUWORVKQPU CPF KUUWGU CU KV KU CRRNKGF VQ VJG URGGEJ TGEQIPKVKQP RTQDNGO 6JG ENCUUKECN $C[GU FGEKUKQP VJG QT[ CRRTQCEJ VTCPUHQTOU VJG TGEQIPK\GT FGUKIP RTQDNGO VQ C RTQDNGO QH RTQDCDKNKV[ FKUVTKDWVKQP GUVKOCVKQP 6JG NKOKVCVKQP QH VJG CRRTQCEJ JQYGXGT EQOGU HTQO VJG HCEV VJCV VJG VTWG HQTO QH VJG RTQDCDKNKV[ FKUVTKDWVKQPU QH URGGEJ UKIPCN KU TGCNKUVKECNN[ WPMPQYP CPF CP[ CUUWOGF FKUVTKDWVKQP HQTO WUGF KP VJG OQFGN YKNN FGXKCVG HTQO VJG VTWG QPG QH VJG UQWTEG TGUWNVKPI KP UWDQRVKOCN TGEQIPKVKQP RGTHQTOCPEG CPF OCMKPI VJG OKPKOWO GTTQT RTQDCDKNKV[ CU UWIIGUVGF D[ VJG $C[GU CRRTQCEJ WPCVVCKPCDNG +P NKIJV QH VJKU NKOKVCVKQP VJG FKUETKOKPCPV HWPEVKQP DCUGF /%' CRRTQCEJ YCU KP VTQFWEGF CU CP CNVGTPCVKXG VQ VJG FKUVTKDWVKQP GUVKOCVKQP DCUGF CRRTQCEJ KP RCVVGTP TGEQIPKVKQP +V VCMGU C FKUETKOKPCPV HWPEVKQP DCUGF UVCVKUVKECN RCVVGTP ENCUUKſECVKQP CRRTQCEJ VQ ENCUUKſGT FGUKIP (QT C IKXGP UGV QH FKUETKOKPCPV HWPEVKQPU VJG ENCUUKſGT FGUKIP KU VQ ſPF C UGV QH RCTCOGVGTU YJKEJ OKPKOK\G VJG GORKTKECN TGEQIPKVKQP GTTQT TCVG 6JKU KU CEJKGXGF VJTQWIJ C URGEKCN NQUU HWPEVKQP YJGTG OKPKOK\KPI VJG GZRGEVGF NQUU TGNCVGU VQ VJG TGFWEVKQP QH VJG TGEQIPKVKQP GTTQT TCVG 6JG FKUETKOKPCPV HWPEVKQP DCUGF /%' CRRTQCEJ CRRNKGU VQ ECUGU YJGTG VJG VTCFKVKQPCN FKUVTKDWVKQP GUVKOCVKQP

‹E\&5&3UHVV//&

DCUGF CRRTQCEJ FQGU PQV CRRN[ GURGEKCNN[ YJGP VJG HCOKN[ QH VJG FKUETKOKPCPV HWPE VKQPU GPEQWPVGTGF KP VJG ENCUUKſGT CTG PQV DCUGF QP RTQDCDKNKV[ FKUVTKDWVKQPU 6JG IQQFPGUU QH VJKU CRRTQCEJ KU LWUVKſGF YKVJQWV VJG OQFGN EQTTGEVPGUU CUUWORVKQP CPF KV CRRNKGU VQ ECUGU YJGTG VJG OQFGN EQTTGEVPGUU CUUWORVKQP KU MPQYP VQ DG KPXCNKF 9G HQTOWNCVGF VJG DCUKE VJGQTGVKECN HTCOGYQTM QH VJKU CRRTQCEJ CPF FKUEWUUGF KVU TGNCVKQP VQ QVJGT ETKVGTKC WUGF KP VJG ENCUUKſGT FGUKIP 6JG FGXGNQROGPV QH /%' CR RTQCEJ JCU NGF VQ C PGY RCTCFKIO KP RCVVGTP TGEQIPKVKQP CPF KV NGCFU VQ TGEQIPKVKQP RGTHQTOCPEG CFXCPVCIGU QXGT VJG EQPXGPVKQPCN CRRTQCEJ KP OCP[ CRRNKECVKQPU 9G UVWFKGF XCTKQWU GZVGPUKQPU QH VJG /%' CRRTQCEJ CPF RTQXKFGF VJGQTGVKECN LWUVKſEC VKQPU CU YGNN CU KORNGOGPVCVKQP FGVCKNU YJGP KV YCU CRRNKGF VQ FKHHGTGPV ENCUUKſECVKQP RTQDNGOU KP URGGEJ CPF NCPIWCIG RTQEGUUKPI 6JKU EJCRVGT KU DCUGF QP PGY FGXGN QROGPVU KP FKUETKOKPCPV HWPEVKQP DCUGF /%' CRRTQCEJ FWTKPI VJG RCUV VGP [GCTU #NVJQWIJ CVVGORVU YGTG OCFG VQ RTQXKFG C UPCRUJQV QH TGUGCTEJ KP VJKU CTGC VJG OC VGTKCN EQXGTGF KP VJKU EJCRVGT KU D[ PQ OGCPU GZJCWUVKXG 4GUGCTEJ QP FKUETKOKPCVKXG OGVJQFU KP RCVVGTP ENCUUKſECVKQP KU C HCUV OQXKPI ſGNF YKVJ PGY RTQDNGOU CPF CRRNK ECVKQPU HTQO XCTKQWU FKTGEVKQPU CPF YG CTG LWUV CV VJG DGIKPPKPI QH TGCNK\KPI VJG PGY RQVGPVKCN QH VJKU CRRTQCEJ KP RCVVGTP TGEQIPKVKQP

Acknowledgement 6JG CWVJQT YQWNF NKMG VQ CEMPQYNGFIG VJG EQPVTKDWVKQPU QH JKU RCUV CPF RTGUGPV EQN NCDQTCVQTU /QUV OCVGTKCNU RTGUGPVGF KP VJKU EJCRVGT CTG DCUGF QP LQKPV RWDNKECVKQPU CPF FKUEWUUKQPU

References =? 5+ #OCTK ő# VJGQT[ QH CFCRVKXG RCVVGTP ENCUUKſGTUŒ IEEE Trans. on Electronic Computers 8QN  0Q  RR Ō  =? 5+ #OCTK ő.GCTPKPI RCVVGTPU CPF RCVVGTP UGSWGPEGU D[ UGNHQTICPK\KPI PGVU QH VJTGUJQNF GNGOGPVUŒ IEEE Transactions on Computers 8QN % 0Q  RR Ō 0QXGODGT  =? # )WPCYCTFCPC ő/CZKOWO OWVWCN KPHQTOCVKQP GUVKOCVKQP QH CEQWUVKE *// GOKUUKQP FGPUKVKGUŒ CLSP Research Note No. 40 %GPVGT HQT .CPIWCIG CPF 5RGGEJ 2TQEGUUKPI ,QJPU *QRMKPU 7PKXGTUKV[ ,WPG   =? .4 $CJN 2( $TQYP 28 FG5QW\C CPF 4 . /GTEGT ő/CZKOWO OWVWCN KPHQTOCVKQP GUVKOCVKQP QH *// RCTCOGVGTU HQT URGGEJ TGEQIPKVKQPŒ Proceedings of ICASSP-86 RR Ō 

‹E\&5&3UHVV//&

=? .4 $CJN 2( $TQYP 28 FG5QW\C CPF 4. /GTEGT ő'UVKOCVKPI JKFFGP /CTMQX OQFGN RCTCOGVGTU UQ CU VQ OCZKOK\G URGGEJ TGEQIPKVKQP CEEWTCE[Œ IEEE Trans. Speech and Audio Processing 8QN  0Q  RR Ō  =? .4 $CJN ( ,GNKPGM CPF 4. /GTEGT ő# OCZKOWO NKMGNKJQQF CRRTQCEJ VQ EQPVKPWQWU URGGEJ TGEQIPKVKQPŒ IEEE Transactions on Pattern and Machine Intelligence 8QN 2#/+ RR Ō  =? .' $CWO 6 2GVTKG ) 5QWNGU CPF 0 9GKUU ő# OCZKOK\CVKQP VGEJPKSWG QE EWTTKPI KP VJG UVCVKUVKECN CPCN[UKU QH RTQDCDKNKUVKE HWPEVKQPU QH /CTMQX EJCKPUŒ Ann. Math. Stat. 8QN  RR Ō  =? .' $CWO ő#P KPGSWCNKV[ CPF CUUQEKCVGF OCZKOK\CVKQP VGEJPKSWGU KP UVCVKU VKECN GUVKOCVKQP HQT RTQDCDKNKUVKE HWPEVKQPU QH /CTMQX RTQEGUUŒ Inequalities 8QN  RR Ō  =? .' $CWO CPF ,# 'CIQP ő#P KPGSWCNKV[ YKVJ CRRNKECVKQPU VQ UVCVKUVKECN RTGF KECVKQP HQT HWPEVKQPU QH /CTMQX RTQEGUU CPF VQ OQFGN QH GEQNQI[Œ Bull. Amer. Math Soc., 8QN  RR Ō  =? .' $CWO CPF ) 5GNN ő)TQYVJ VTCPUHQTOCVKQPU HQT HWPEVKQPU QP OCPKHQNFUŒ Pacific J. Math. 8QN  0Q  RR Ō  =? $KEMGN CPF &QMUWO Mathematical Statistics 2TGPVKEG*CNN  =? # $GPXGPKUVG / /GVKXKGT CPF 2 2TKQWGV Adaptive Algorithms and Stochastic Approximations 5RTKPIGT8GTNCI  =? 2 $G[GTNGKP ő&KUETKOKPCVKXG OQFGN EQODKPCVKQPŒ Proc. 1997 Workshop on Automatic Speech Recognition and Understanding Proceedings RR Ō  =? # $KGO 5 -CVCIKTK CPF $* ,WCPI ő2CVVGTP TGEQIPKVKQP WUKPI FKUETKOKPC VKXG HGCVWTG GZVTCEVKQPŒ IEEE Trans. Signal Processing 8QN  RR Ō  =? ,4 $NWO ő/WNVKFKOGPUKQPCN UVQEJCUVKE CRRTQZKOCVKQP OGVJQFUŒ #PP /CVJ 5VCV 8QN  RR Ō  =? 2% %JCPI CPF $* ,WCPI ő&KUETKOKPCVKXG VGORNCVG VTCKPKPI HQT F[PCOKE RTQITCOOKPI URGGEJ TGEQIPKVKQPŒ Proc. ICASSP92 8QN  RR  Ō   =? 2% %JCPI CPF $* ,WCPI ő&KUETKOKPCVKXG VTCKPKPI HQT F[PCOKE RTQITCO OKPI DCUGF URGGEJ TGEQIPK\GTUŒ IEEE Trans. Speech and Audio Processing 5#2   =? 9 %JQW %* .GG CPF $ * ,WCPI ő5GIOGPVCN )2& VTCKPKPI QH CP JKFFGP /CTMQX OQFGN DCUGF URGGEJ TGEQIPK\GTŒ IEEE Proc. ICASSP-92 RR Ō  #RTKN 

‹E\&5&3UHVV//&

=? 9 %JQW %* .GG CPF $* ,WCPI ő/KPKOWO GTTQT TCVG VTCKPKPI DCUGF QP 0DGUV UVTKPI OQFGNUŒ IEEE Proc. ICASSP-93 8QN ++ RRŌ  =? 9 %JQW %* .GG CPF $* ,WCPI ő/KPKOWO GTTQT TCVG VTCKPKPI QH KPVGT YQTF EQPVGZV FGRGPFGPV CEQWUVKE OQFGN WPKVU KP URGGEJ TGEQIPKVKQPŒ Proc. ICSLP-94 RR Ō ;QMQJCOC  =? 9 %JQW $* ,WCPI CPF %* .GG ő/KPKOWO GTTQT TCVG VTCKPKPI QH EQO DKPGF UVTKPI OQFGNUŒ 75 2CVGPV   =? 9 %JQW %* .GG CPF $* ,WCPI ő5RGGEJ TGEQIPKVKQP DCUGF QP EQODKPGF UVTKPI OQFGNUŒ Proc. DARPA ANN Tech. Program CSR Mtg. RR Ō  =? 9 %JQW ő&KUETKOKPCPVHWPEVKQPDCUGF OKPKOWO TGEQIPKVKQP GTTQT TCVG RCVVGTPTGEQIPKVKQP CRRTQCEJ VQ URGGEJ TGEQIPKVKQPŒ Proceedings of The IEEE 8QN  0Q  RR  #WIWUV  =? 9 %JQW %* .GG $* ,WCPI CPF ( - 5QQPI ő# OKPKOWO GTTQT TCVG RCVVGTP TGEQIPKVKQP CRRTQCEJ VQ URGGEJ TGEQIPKVKQPŒ International Journal of Pattern Recognition and Artificial Intelligence 8QN  0Q  RR Ō  =? 9 %JQW CPF $* ,WCPI “Adaptive discriminative learning in pattern recognition,” 6GEJPKECN 4GRQTV QH #66 $GNN .CDQTCVQTKGU  =? 6 %QXGT CPF , 6JQOCU Elements of Information Theory ,QJP 9KNG[  5QPU  =? 5/ %JW CPF ;