o rZh>@sdZddlZddlmZmZddlmZddZej Z ddZ dZ zdd l mZWn ey6d d ZYnwdZ d Z d Z GdddedZGdddeZGdddeZGdddeZGdddZdS)z Provides scoring functions for a number of association measures through a generic, abstract implementation in ``NgramAssocMeasures``, and n-specific ``BigramAssocMeasures`` and ``TrigramAssocMeasures``. N)ABCMetaabstractmethodreducecCs t|SN)_mathlog2)xr G/var/www/auris/lib/python3.10/site-packages/nltk/metrics/association.pys r cCstdd|S)NcSs||Srr )r yr r r r sz..r)sr r r r sg#B ;) fisher_exactcOstrNotImplementedError)_args_kwargsr r r rsrc@seZdZdZdZeeddZeeddZe ddZ ed d Z e d d Z e d dZ eddZe ddZe ddZe ddZe ddZdS)NgramAssocMeasuresa An abstract class defining a collection of generic association measures. Each public method returns a score, taking the following arguments:: score_fn(count_of_ngram, (count_of_n-1gram_1, ..., count_of_n-1gram_j), (count_of_n-2gram_1, ..., count_of_n-2gram_k), ..., (count_of_1gram_1, ..., count_of_1gram_n), count_of_total_words) See ``BigramAssocMeasures`` and ``TrigramAssocMeasures`` Inheriting classes should define a property _n, and a method _contingency which calculates contingency values from marginals in order for all association measures defined here to be usable. rcGtd)z>Calculates values of a contingency table from marginal values.?The contingency table is not availablein the general ngram caser marginalsr r r _contingencyBzNgramAssocMeasures._contingencycGr)ACalculates values of contingency table marginals from its values.rr contingencyr r r _marginalsJrzNgramAssocMeasures._marginalsc#s^t}ddtjD}ttD]tfdd|D|jdVqdS)3Calculates expected values for a contingency table.cSsg|]}d|>qS)r ).0ir r r Vsz7NgramAssocMeasures._expected_values..c3s4|]tfddtdjDVqdS)c3s(|]}|@@kr|VqdSrr )r#r )contr$jr r ]&z@NgramAssocMeasures._expected_values...N)sumrange_n)r#clsr&r$)r'r r(\s " z6NgramAssocMeasures._expected_values..r"N)r+r,r-len_product)r/r&n_allbitsr r.r _expected_valuesRs z#NgramAssocMeasures._expected_valuescGs|t|tS)z Scores ngrams by their frequency)NGRAMTOTALrr r r raw_freqcszNgramAssocMeasures.raw_freqcGs6|tt|t|t|jd|ttdS)zScores ngrams using Student's t test with independence hypothesis for unigrams, as in Manning and Schutze 5.3.1. r"g?)r5r1UNIGRAMSr6r-_SMALLr/rr r r student_ths zNgramAssocMeasures.student_tcGs,|j|}||}tddt||DS)zZScores ngrams using Pearson's chi-square as in Manning and Schutze 5.3.3. css(|]\}}||d|tVqdS)r*N)r9r#Zobsexpr r r r(yr)z,NgramAssocMeasures.chi_sq..)rr4r+zip)r/rr&Zexpsr r r chi_sqrs  zNgramAssocMeasures.chi_sqcOs |t|ddt|tS)zScores ngrams using a variant of mutual information. The keyword argument power sets an exponent (default 3) for the numerator. No logarithm of the result is calculated. power)r5getr1r8)rkwargsr r r mi_like{szNgramAssocMeasures.mi_likecGs.t|t|t|jdtt|tS)z^Scores ngrams by pointwise mutual information, as in Manning and Schutze 5.4. r")_log2r5r6r-r1r8r:r r r pmis zNgramAssocMeasures.pmicGs,|j|}dtddt|||DS)zFScores ngrams using likelihood ratios as in Manning and Schutze 5.3.4.r*css,|]\}}|t||ttVqdSr)_lnr9r<r r r r(s  z6NgramAssocMeasures.likelihood_ratio..)rr+r>r4r/rr&r r r likelihood_ratios   z#NgramAssocMeasures.likelihood_ratiocGs:t|t|t|jd}|tt|t|dS)z1Scores ngrams using the Poisson-Stirling measure.r")r1r8r6r-r5rE)r/rr=r r r poisson_stirlingsz#NgramAssocMeasures.poisson_stirlingcGs"|j|}|dt|ddS)z&Scores ngrams using the Jaccard index.rNr)rr+rHr r r jaccards zNgramAssocMeasures.jaccardN)__name__ __module__ __qualname____doc__r- staticmethodrrr classmethodr4r7r;r?rDrFrIrJrKr r r r r-s6          r) metaclassc@sheZdZdZdZeddZeddZeddZe d d Z e d d Z e d dZ eddZ dS)BigramAssocMeasuresa A collection of bigram association measures. Each association measure is provided as a function with three arguments:: bigram_score_fn(n_ii, (n_ix, n_xi), n_xx) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: - n_ii counts ``(w1, w2)``, i.e. the bigram being scored - n_ix counts ``(w1, *)`` - n_xi counts ``(*, w2)`` - n_xx counts ``(*, *)``, i.e. any bigram This may be shown with respect to a contingency table:: w1 ~w1 ------ ------ w2 | n_ii | n_oi | = n_xi ------ ------ ~w2 | n_io | n_oo | ------ ------ = n_ix TOTAL = n_xx r*cCs0|\}}||}||}|||||||fS)zECalculates values of a bigram contingency table from marginal values.r )n_ii n_ix_xi_tuplen_xxn_ixn_xin_oin_ior r r rsz BigramAssocMeasures._contingencycCs"|||||f||||fS)rr )rTrYrZn_oor r r r s"zBigramAssocMeasures._marginalsccsJt|}tdD]}||||dA||||dA|Vq dS)r!r"r*N)r+r,)r&rVr$r r r r4s  0z$BigramAssocMeasures._expected_valuescGsF|j|\}}}}||||d||||||||S)zdScores bigrams using phi-square, the square of the Pearson correlation coefficient. r*)r)r/rrTrZrYr[r r r phi_sqszBigramAssocMeasures.phi_sqcCs|\}}|||||f|S)zScores bigrams using chi-square, i.e. phi-sq multiplied by the number of bigrams, as in Manning and Schutze 5.3.3. )r])r/rTrUrVrWrXr r r r?szBigramAssocMeasures.chi_sqcGs2|j|\}}}}t||g||ggdd\}}|S)zScores bigrams using Fisher's Exact Test (Pedersen 1996). Less sensitive to small counts than PMI or Chi Sq, but also more expensive to compute. Requires scipy. less) alternative)rr)r/rrTrZrYr[ZoddsZpvaluer r r fisherszBigramAssocMeasures.fishercCs|\}}d|||S)z(Scores bigrams using Dice's coefficient.r*r )rTrUrVrWrXr r r diceszBigramAssocMeasures.diceN)rLrMrNrOr-rPrr r4rQr]r?r`rar r r r rSs"       rSc@,eZdZdZdZeddZeddZdS)TrigramAssocMeasuresa A collection of trigram association measures. Each association measure is provided as a function with four arguments:: trigram_score_fn(n_iii, (n_iix, n_ixi, n_xii), (n_ixx, n_xix, n_xxi), n_xxx) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: - n_iii counts ``(w1, w2, w3)``, i.e. the trigram being scored - n_ixx counts ``(w1, *, *)`` - n_xxx counts ``(*, *, *)``, i.e. any trigram rAcCs|\}}}|\}}} ||} ||} ||} | || | } ||| | }||| | }||| | | | ||}|| | | | |||fS)zCalculates values of a trigram contingency table (or cube) from marginal values. >>> TrigramAssocMeasures._contingency(1, (1, 1, 1), (1, 73, 1), 2000) (1, 0, 0, 0, 0, 72, 0, 1927) r )n_iiiZ n_iix_tupleZ n_ixx_tupleZn_xxxZn_iixZn_ixiZn_xiiZn_ixxZn_xixZn_xxin_oiin_ioin_iion_ooin_oion_ioon_ooor r r rs   z!TrigramAssocMeasures._contingencyc Gs`|\}}}}}}}}|||||||f||||||||||||ft|fS)zCalculates values of contingency table marginals from its values. >>> TrigramAssocMeasures._marginals(1, 0, 0, 0, 0, 72, 0, 1927) (1, (1, 1, 1), (1, 73, 1), 2000) r+) rrdrerfrhrgrirjrkr r r r &szTrigramAssocMeasures._marginalsNrLrMrNrOr-rPrr r r r r rcs rcc@rb)QuadgramAssocMeasuresaF A collection of quadgram association measures. Each association measure is provided as a function with five arguments:: trigram_score_fn(n_iiii, (n_iiix, n_iixi, n_ixii, n_xiii), (n_iixx, n_ixix, n_ixxi, n_xixi, n_xxii, n_xiix), (n_ixxx, n_xixx, n_xxix, n_xxxi), n_all) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: - n_iiii counts ``(w1, w2, w3, w4)``, i.e. the quadgram being scored - n_ixxi counts ``(w1, *, *, w4)`` - n_xxxx counts ``(*, *, *, *)``, i.e. any quadgram r\c"Cs|\}}}}|\} } } } } }|\}}}}||}||}||}| |||}| |||}| |||}||||||||}||}||||}| |||}||||||||}| |||}||||||||}||||||||} |||||||||||||||| }!||||||||||||||| |!fS)zXCalculates values of a quadgram contingency table from marginal values. r )"n_iiiiZ n_iiix_tupleZ n_iixx_tupleZ n_ixxx_tupleZn_xxxxn_iiixn_iixin_ixiin_xiiin_iixxn_ixixn_ixxin_xixin_xxiin_xiixn_ixxxn_xixxn_xxixn_xxxin_oiiin_ioiin_iioin_ooiin_oioin_iooin_oooin_iiion_oiion_ioion_ooion_iioon_oioon_iooon_oooor r r rPs           z"QuadgramAssocMeasures._contingencyc GsV|\}}}}}}}}} } } } } }}}|| }||}||}||}||| | }||| | }||||}||||}||||}||| | }|||| || | |}|||| || | |}|||| || | | }||||||||}t|}|||||f||||||f||||f|fS)aCalculates values of contingency table marginals from its values. QuadgramAssocMeasures._marginals(1, 0, 2, 46, 552, 825, 2577, 34967, 1, 0, 2, 48, 7250, 9031, 28585, 356653) (1, (2, 553, 3, 1), (7804, 6, 3132, 1378, 49, 2), (38970, 17660, 100, 38970), 440540) rl) rror~rrrrrrrrrrrrrrrprqrrrsrtrurvrwrxryrzr{r|r}r2r r r r sN      z QuadgramAssocMeasures._marginalsNrmr r r r rn9s ;rnc@s$eZdZdZddZeddZdS)ContingencyMeasureszWraps NgramAssocMeasures classes such that the arguments of association measures are contingency table values rather than marginals. cCsZd|jj|j_t|D]}|drq t||}|ds$|||}t|||q dS)zAConstructs a ContingencyMeasures given a NgramAssocMeasures classZ Contingency___N) __class__rLdir startswithgetattr_make_contingency_fnsetattr)selfmeasureskvr r r __init__s     zContingencyMeasures.__init__cs"fdd}j|_j|_|S)zFrom an association measure function, produces a new function which accepts contingency table values as its arguments. csj|Sr)r rrold_fnr r ressz5ContingencyMeasures._make_contingency_fn..res)rOrL)rrrr rr rsz(ContingencyMeasures._make_contingency_fnN)rLrMrNrOrrPrr r r r rs  r)rOmathrabcrr functoolsrrElogrGr1r9Z scipy.statsr ImportErrorr5r8r6rrSrcrnrr r r r s2   wY<