言語モデル入門 (第二版)

  • Published on
    01-Jul-2015

  • View
    2.085

  • Download
    0

Embed Size (px)

DESCRIPTION

Introduction to language models in Japanese

Transcript

  • 1. 20141019@akkikiki

2. 3. 4. CourseraIME 5. P( > P(IME210 6. BrownGoogle n-gramBCCWJ 7. +=Michael)Li+ ACL 2004TwitterTopic tracking(Lin+ KDD 2011) Danescu-Niculescu-Mizil+ WWW 2013 8. 9. I want to go to ______P(w | I want to go to) 10. n-gramn-1n-gramn-gramn-gram1-gram = unigram, 2-gram = bigrambigramP(w_i | w_i-1) = c(w_i, w_i-1) / c(w_i-1)P(/ / / / / /)=P(| ) * P(|) * P(|) * P(|) * P(|) * P( | ) 11. ///////P( | ) = 0 12. n-1, n-2, n-gramngramGood TuringDiscountinterpolatedInterpolated Kneser-ney 13. n(w):C:kP(w) = (n(w) + k) / (C + kV) 14. Good Turing N_c: , , N_1 = 3, N_2 = 1, N_3 = 11 15. Good Turing P_gt() = N_1 / NP_gt = 3/8c*() =2 * N_2 / N_1 = 2/3P_gt() = 2/3 / 8 = 1/12 16. Kneser-ney DiscountingGood turing 0.75 discountAP corpus(Church & Gale 1991)discount cC*00.00002702610.445721.26032.23743.23654.2365.1976.2187.24 17. Kneser-ney overviewInterpolated Kneser-ney (Kneser & Ney 1995)Modified Kneser-ney(Chen & Goodman 1999) 18. Kneser-neyBigramI want to go to Toyama FransiscoFransiscoSanunigramP(Toyama Fransisco) P(Fransisco)P(Toyama Fransisco)Kneser-neyP_continuation: w 19. Kneser-ney 20. Kneser-ney w_i 21. Kneser-ney w_i 22. 23. D: N: N 24. 25. 1 (Bengio+ 2003)AP newsModified Kneser-ney PerplexitymodelnPerplexityNeural LM6109N-gram (Kneser-ney)3127N-gram (Kneser-ney)4119N-gram (Kneser-ney)5117Neural LMinterpolated trigram 26. 2 (Brants+ EMNLP 2007)Stupid back offstupidDiscouting = 0.4Unigram 27. 2 (Brants+ EMNLP 2007)BLEU score stupid backoff 28. 2 (Brants+ EMNLP 2007)BLEU score stupid backoff 29. 30. CMU-Cambridge language model toolkitKneser-neySRI language model toolkit (SRILM)Kneser-neyBengio 31. N-gramKneser-ney SRILM 32. StanfordNLPhttps://class.coursera.org/nlp/lecturehttp://nlp.stanford.edu/~wcmac/papers/20050421-smoothing- tutorial.pdfNYUhttp://www.cs.nyu.edu/~petrov/lecture2.pdfUniv. of Marylandhttp://www.umiacs.umd.edu/~jimmylin/cloud-2010- Spring/session9-slides.pdfNeural Language Modelhttp://www.inf.ed.ac.uk/teaching/courses/asr/2013-14/asr09- nnlm.pdf 33. SRILM# build a 5-gram modelngram-count -order 5 -text hogehoge.txt - unk -lm hogehoge_lm# calculate perplexityngram -order 5 -lm hogehoge_lm -ppl test.txt

Recommended

View more >