/h8SrSSKrSSKJrJrJrJr SSKJ r J r SSK J r SSK Jr "SS5r"S S \5r"S S \5r"S S\5rSSjr\S:XaESSKrSSKJr \"S\R.S-5r\"S\R.S-5r\"\\5 /SQrg!\a SrN1f=f!\a SrN&f=f)a Tools to identify collocations --- words that often appear consecutively --- within corpora. They may also be used to find other associations between word occurrences. See Manning and Schutze ch. 5 at https://nlp.stanford.edu/fsnlp/promo/colloc.pdf and the Text::NSP Perl package at http://ngram.sourceforge.net Finding collocations requires first calculating the frequencies of words and their appearance in the context of other words. Often the collection of words will then requiring filtering to only retain useful content terms. Each ngram of words may then be scored according to some association measure, in order to determine the relative likelihood of each ngram being a collocation. The ``BigramCollocationFinder`` and ``TrigramCollocationFinder`` classes provide these functionalities, dependent on being provided a function which scores a ngram given appropriate frequency counts. A number of standard association measures are provided in bigram_measures and trigram_measures. N)BigramAssocMeasuresContingencyMeasuresQuadgramAssocMeasuresTrigramAssocMeasures)ranks_from_scoresspearman_correlation)FreqDist)ngramsc\rSrSrSrSr\SSj5r\S5r\ S5r S4S jr S r S r S rS rSrSrSrSrg)AbstractCollocationFinder/ad An abstract base class for collocation finders whose purpose is to collect collocation candidate frequencies, filter and rank them. As a minimum, collocation finders require the frequencies of each word in a corpus, and the joint frequency of word tuples. This data should be provided through nltk.probability.FreqDist objects or an identical interface. cFXlUR5UlX lgN)word_fdNngram_fd)selfrrs I/var/www/auris/envauris/lib/python3.13/site-packages/nltk/collocations.py__init__"AbstractCollocationFinder.__init__:s  Nc^U4US- -mU(a)[RRU4SjU55$U(a)[RRU4SjU55$g)zE Pad the document with the place holder according to the window_size c3R># UHn[R"UT5v M g7fr _itertoolschain.0docpaddings r AAbstractCollocationFinder._build_new_documents..Hs$2:C3   g..)$'c3R># UHn[R"TU5v M g7frrrs rr"r#Ls$2:C3   #..)r$N)rr from_iterable)cls documents window_sizepad_left pad_right pad_symbolr!s @r_build_new_documents.AbstractCollocationFinder._build_new_documents?sn-;?3 ##112:C2  ##112:C2  rcTURURXRSS95$)zrConstructs a collocation finder given a collection of documents, each of which is a list (or iterable) of tokens. Tr+) from_wordsr- default_ws)r'r(s rfrom_documents(AbstractCollocationFinder.from_documentsPs, ~~  $ $Y$ $ O  rc\^^[UU4Sj[[T5S- 555$)Nc3D># UHn[TXT-5v M g7fr)tuple)rinwordss rr".\s$O9NAeAA.//9Ns r)r rangelen)r:r9s``r_ngram_freqdist)AbstractCollocationFinder._ngram_freqdistZs!Os5zA~9NOOOrcg)NF)ngramfreqs r"AbstractCollocationFinder.^s5rc[5nURR5Hup4U"X45(aMXBU'M X lg)zwGeneric filter removes ngrams from the frequency distribution if the function returns True when passed an ngram tuple. N)r ritems)rfn tmp_ngramrBrCs r _apply_filter'AbstractCollocationFinder._apply_filter^s<J ==..0KEe??#'% 1" rc0^URU4Sj5 g)zARemoves candidate ngrams which have frequency less than min_freq.c>UT:$rrA)ngrCmin_freqs rrD=AbstractCollocationFinder.apply_freq_filter..js D8OrNrJ)rrOs `rapply_freq_filter+AbstractCollocationFinder.apply_freq_filterhs ;T"U6$rrArNfrHs rrD>AbstractCollocationFinder.apply_ngram_filter..ps RrNrQrrHs `rapply_ngram_filter,AbstractCollocationFinder.apply_ngram_filterls 01rc0^URU4Sj5 g)z]Removes candidate ngrams (w1, w2, ...) where any of (fn(w1), fn(w2), ...) evaluates to True. c.>[U4SjU55$)Nc34># UH nT"U5v M g7frrA)rwrHs rr"PAbstractCollocationFinder.apply_word_filter....vs,?BqRUUBs)anyrVs rrD=AbstractCollocationFinder.apply_word_filter..vs,?B,?)?rNrQrYs `rapply_word_filter+AbstractCollocationFinder.apply_word_filterrs ?@rc#l# URH nUR"U/UQ76nUcMX#4v M" g7f)zRGenerates of (ngram, score) pairs as determined by the scoring function provided. N)r score_ngram)rscore_fntupscores r _score_ngrams'AbstractCollocationFinder._score_ngramsxs8==C$$X44E j !s$4 4c6[URU5SS9$)zReturns a sequence of (ngram, score) pairs ordered from highest to lowest score, as determined by the scoring function provided. cUS*US4$)NrrrA)ts rrD8AbstractCollocationFinder.score_ngrams..sAaD5!A$-r)key)sortedrj)rrgs r score_ngrams&AbstractCollocationFinder.score_ngramssd((28OPPrc`URU5SUVVs/sHup4UPM snn$s snnf)z;Returns the top n ngrams when scored by the given function.Nrr)rrgr9pss rnbestAbstractCollocationFinder.nbests0"//9"1=>=da=>>>s*c#V# URU5Hup4XB:aUv M g g7f)zmReturns a sequence of ngrams, ordered by decreasing score, whose scores each exceed the given minimum score. Nru)rrg min_scorerBris r above_score%AbstractCollocationFinder.above_scores,!--h7LE   8s'))rrr)FFN)__name__ __module__ __qualname____firstlineno____doc__r classmethodr-r3 staticmethodr>rJrRrZrcrjrrrxr|__static_attributes__rArrr r /s|! QU   PP 9"=2 A !Q ?rr c@\rSrSrSrSrS Sjr\S Sj5rSr Sr g) BigramCollocationFinderzA tool for the finding and ranking of bigram collocations or other association measures. It is often useful to use from_words() rather than constructing an instance directly. c<[RXU5 X0lg)zuConstruct a BigramCollocationFinder, given FreqDists for appearances of words and (possibly non-contiguous) bigrams. N)r rr))rr bigram_fdr)s rr BigramCollocationFinder.__init__s "**4)D&rc[5n[5nUS:a [S5e[XSS9H8nUSnUcM X6==S- ss'USSHnUcMXFU4==S- ss'M M: U"X4US9$) zConstruct a BigramCollocationFinder for all bigrams in the given sequence. When window_size > 2, count non-contiguous bigrams, in the style of Church and Hanks's (1990) association ratio. rzSpecify window_size at least 2Tr0rNrr))r ValueErrorr )r'r:r)wfdbfdwindoww1w2s rr1"BigramCollocationFinder.from_wordss jj ?=> >U4@FBz GqLGQRj>RMQ&M! A355rcURnURX#4URS- - nU(dgURUnURUnU"XVU4U5$)zReturns the score for a given bigram using the given scoring function. Following Church and Hanks (1990), counts are scaled by a factor of 1/(window_size - 1). g?N)rrr)r)rrgrrn_alln_iin_ixn_xis rrf#BigramCollocationFinder.score_ngrams` }}bX&$*:*:S*@A ||B||BTlE22rrN)r r~rrrrr2rrr1rfrrArrrrs, J'66* 3rrcB\rSrSrSrSrSr\S Sj5rSr Sr Sr g ) TrigramCollocationFinderzA tool for the finding and ranking of trigram collocations or other association measures. It is often useful to use from_words() rather than constructing an instance directly. cH[RXU5 X0lX lg)zConstruct a TrigramCollocationFinder, given FreqDists for appearances of words, bigrams, two words with any word between them, and trigrams. N)r r wildcard_fdr)rrrr trigram_fds rr!TrigramCollocationFinder.__init__s "**4*E&"rcUS:a [S5e[5n[5n[5n[5n[XSS9HpnUSnUcM [R"USSS5HEupX8==S- ss'U cMXXU 4==S- ss'U cM)XHU 4==S- ss'XhX4==S- ss'MG Mr U"X5XF5$) zMConstruct a TrigramCollocationFinder for all trigrams in the given sequence. rzSpecify window_size at least 3Tr0rNrrrr r r combinations) r'r:r)rwildfdrtfdrrrw3s rr1#TrigramCollocationFinder.from_wordss ?=> >jjjU4@FBz$11&*a@1 :H " :Bx A% L!Q&!A A3V))rcB[URUR5$)zConstructs a bigram collocation finder with the bigram and unigram data from this finder. Note that this does not include any filtering applied to this finder. )rrr)rs r bigram_finder&TrigramCollocationFinder.bigram_finders 't||T^^DDrc"URnURX#U4nU(dgURX#4nURX$4nURX44n URUn URUn URUn U"XgX4XU 4U5$)zHReturns the score for a given trigram using the given scoring function. N)rrrrr) rrgrrrrn_iiin_iixn_ixin_xiin_ixxn_xixn_xxis rrf$TrigramCollocationFinder.score_ngrams rrl+ x(  "*x( R  R  R u4uU6KUSSr)rrN)r) r~rrrrr2rrr1rrfrrArrrrs3 J#**4ETrrc<\rSrSrSrSrSr\S Sj5rSr Sr g) QuadgramCollocationFinderi zA tool for the finding and ranking of quadgram collocations or other association measures. It is often useful to use from_words() rather than constructing an instance directly. c x[RXU5 X@lX0lXPlX`lXplXlg)zConstruct a QuadgramCollocationFinder, given FreqDists for appearances of words, bigrams, trigrams, two words with one word and two words between them, three words with a word between them in both variations. N)r riiiiiixiixxiiixiixii) rr quadgram_fdrrrrrrs rr"QuadgramCollocationFinder.__init__s2 "**4+F   rc TUS:a [S5e[5n[5n[5n[5n[5n[5n[5n [5n [XSS9Hn U Sn U cM [R"U SSS5HupnX<==S- ss'U cMX\U 4==S- ss'UcM*XlX4==S- ss'X|U4==S- ss'UcMKXLXU4==S- ss'XU4==S- ss'XX4==S- ss'XX4==S- ss'M M U"X4XVXxX5$)NrzSpecify window_size at least 4Tr0rrrr)r'r:r)ixxxiiiirrrrrrrrrrw4s rr1$QuadgramCollocationFinder.from_words!sC ?=> >zz ZjjzzzU4@FBz(55fQRj!D A :8 ! :L!Q&!H " :""%&!+&"X!#"\"a'""\"a'"E A(4r4>>rc 8URnURX#XE4nU(dgURX#U4nURX4U4n URX#U4n URX$U4n UR X#4n UR XE4n UR X44nUR X$4nURX%4nUR X54nURUnURUnURUnURUnU"UXX4XUUX4UUUU4U5$r) rrrrrrrrr)rrgrrrrrn_iiiin_iiixn_xiiin_iixin_ixiin_iixxn_xxiin_xiixn_ixixn_ixxin_xixin_ixxxn_xixxn_xxixn_xxxis rrf%QuadgramCollocationFinder.score_ngramDs7/0 22,'22,'BB<(BB<(""""""2(#B8$2(#b!b!b!b!  V , VVV < VVV ,    r)rrrrrrN)rrrArrrr s-J  ? ?D rrc ^SSKJnJnJn Uc URnUc UR nSSKJnJn URS5mU4SjnUR5HnURU5V s/sHoR5PM n n [RU 5n U RS5 U RU5 U"U"U R!U55U"U R!U555n [#U5 [#SU R%US 5V s/sHn S R'U 5PM sn 5 [#S UR(S U S 35 GM gs sn fs sn f)z=Finds bigram collocations in the files of the WebText corpus.r)rrrN) stopwordswebtextenglishcT>[U5S:=(d UR5T;$)Nr)r=lower)r_ ignored_wordss rrDdemo..rs CFQJD!'')}*DDrr  z Correlation to z: z0.4f) nltk.metricsrrrlikelihood_ratioraw_freq nltk.corpusrrr:fileidsrrr1rRrcrrprintrxjoinr~)scorercompare_scorerrrrrr word_filterfilewordr:cfcorrrhrs @rdemorbs1 ~$55,55.OOI.MDK!*1--*=>*=$*=> $ / / 6 Q [)# boof5 6 boon= >  d  dbhhvr.BC.BsSXXc].BCD ">#:#:";2d4[IJ">Ds /EE" __main__)rzBigramAssocMeasures.rr)rrr)NN)r itertoolsrrrrrrnltk.metrics.spearmanrrnltk.probabilityr nltk.utilr r rrrrr~sysevalargvr IndexErrorr__all__rArrrs2 J%ddN03703fAT8ATHR 9R jKL z0,sxx{:;4sxx{BC     s$(B'B5'B21B25C?C