/h>SrSSKrSSKJrJr SSKJr Sr\Rr Sr Sr SSK Jr SrS rS r"S S \S9r"SS\5r"SS\5r"SS\5r"SS5rg!\a S rNIf=f)z Provides scoring functions for a number of association measures through a generic, abstract implementation in ``NgramAssocMeasures``, and n-specific ``BigramAssocMeasures`` and ``TrigramAssocMeasures``. N)ABCMetaabstractmethodreducec.[R"U5$N)_mathlog2)xs P/var/www/auris/envauris/lib/python3.13/site-packages/nltk/metrics/association.pyr s %**Q-c[SU5$)Nc X-$r)r ys r r ..srr)ss r r r s V.2rg#B ;) fisher_exactc[erNotImplementedError)_args_kwargss r rrs!!rc\rSrSrSrSr\\S55r\\S55r \ S5r \S5r \ S5r \ S 5r\S 5r\ S 5r\ S 5r\ S 5r\ S5rSrg)NgramAssocMeasures-a An abstract class defining a collection of generic association measures. Each public method returns a score, taking the following arguments:: score_fn(count_of_ngram, (count_of_n-1gram_1, ..., count_of_n-1gram_j), (count_of_n-2gram_1, ..., count_of_n-2gram_k), ..., (count_of_1gram_1, ..., count_of_1gram_n), count_of_total_words) See ``BigramAssocMeasures`` and ``TrigramAssocMeasures`` Inheriting classes should define a property _n, and a method _contingency which calculates contingency values from marginals in order for all association measures defined here to be usable. rc[S5e)z>Calculates values of a contingency table from marginal values.?The contingency table is not availablein the general ngram caser marginalss r _contingencyNgramAssocMeasures._contingencyB" P  rc[S5e)ACalculates values of contingency table marginals from its values.r!r) contingencys r _marginalsNgramAssocMeasures._marginalsJr&rc#^^^# [T5n[TR5Vs/sHnSU-PM nn[[T55H/m[ UUU4SjU55UTRS- -- v M1 gs snf7f)3Calculates expected values for a contingency table.c3>^# UH2m[UUU4Sj[STR-555v M4 g7f)c3H># UHoT-TT-:XdMTUv M g7frr).0r contijs r @NgramAssocMeasures._expected_values...]s&P)9A!eQ=OQ)9s" "N)sumrange_n)r1r4clsr2r3s @r r56NgramAssocMeasures._expected_values..\s3!Pq#&&y)9PPP!s:>N)r8r9r:len_product)r;r2n_allr3bitss`` ` r _expected_values#NgramAssocMeasures._expected_valuesRs~D  %cff . 1Q .s4y!A!SVVaZ( * "/s#BBABc(U[U[- $)z Scores ngrams by their frequency)NGRAMTOTALr"s r raw_freqNgramAssocMeasures.raw_freqcs)E"222rcU[[U[5U[URS- -- - U[[ -S-- $)zqScores ngrams using Student's t test with independence hypothesis for unigrams, as in Manning and Schutze 5.3.1. r.g?)rDr>UNIGRAMSrEr:_SMALLr;r#s r student_tNgramAssocMeasures.student_thsR e y*+y/?CFFQJ/OP Q u  &3 ./ /rcxUR"U6nURU5n[S[X#555$)zJScores ngrams using Pearson's chi-square as in Manning and Schutze 5.3.3. c3H# UHupX- S-U[-- v M g7f)r7N)rJr1obsexps r r5,NgramAssocMeasures.chi_sq..ys"U_CI!#sV|4_s ")r$rAr8zip)r;r#r2expss r chi_sqNgramAssocMeasures.chi_sqrs9 +##D)US_UUUrc`U[URSS5-[U[5- $)zScores ngrams using a variant of mutual information. The keyword argument power sets an exponent (default 3) for the numerator. No logarithm of the result is calculated. power)rDgetr>rI)r#kwargss r mi_likeNgramAssocMeasures.mi_like{s5 6::gq#99H h =   rc[U[U[URS- --5[[ U[ 55- $)zNScores ngrams by pointwise mutual information, as in Manning and Schutze 5.4. r.)_log2rDrEr:r>rIrKs r pmiNgramAssocMeasures.pmisG Yu% %(8SVVaZ(HHIE Yx( )M   rc zUR"U6nS[S[X RU5555-$)zFScores ngrams using likelihood ratios as in Manning and Schutze 5.3.4.r7c3b# UH%upU[X[-- [-5-v M' g7fr)_lnrJrPs r r56NgramAssocMeasures.likelihood_ratio..s/ A #c6\*V34 4As-/)r$r8rTrAr;r#r2s r likelihood_ratio#NgramAssocMeasures.likelihood_ratiosE+3 &:&:4&@A    rc[U[5U[URS- -- nU[[ U[U- 5S- -$)z1Scores ngrams using the Poisson-Stirling measure.r.)r>rIrEr:rDr`)r;r#rRs r poisson_stirling#NgramAssocMeasures.poisson_stirlingsNy*+y/?CFFQJ/OP55)9C)?#@1#DEErcHUR"U6nUS[USS5- $)z&Scores ngrams using the Jaccard index.rNr)r$r8rgs r jaccardNgramAssocMeasures.jaccards-+AwT#2Y''rrN)__name__ __module__ __qualname____firstlineno____doc__r: staticmethodrr$r* classmethodrArFrLrVr]rarhrkrn__static_attributes__rrr rr-s$ B     33//VV      FF ((rr) metaclassc\rSrSrSrSr\S5r\S5r\S5r \ S5r \ S5r \ S 5r \S 5rS rg ) BigramAssocMeasuresa8 A collection of bigram association measures. Each association measure is provided as a function with three arguments:: bigram_score_fn(n_ii, (n_ix, n_xi), n_xx) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: - n_ii counts ``(w1, w2)``, i.e. the bigram being scored - n_ix counts ``(w1, *)`` - n_xi counts ``(*, w2)`` - n_xx counts ``(*, *)``, i.e. any bigram This may be shown with respect to a contingency table:: w1 ~w1 ------ ------ w2 | n_ii | n_oi | = n_xi ------ ------ ~w2 | n_io | n_oo | ------ ------ = n_ix TOTAL = n_xx r7c4Uup4X@- nX0- nXXbU- U- U- 4$)zECalculates values of a bigram contingency table from marginal values.r)n_ii n_ix_xi_tuplen_xxn_ixn_xin_oin_ios r r$ BigramAssocMeasures._contingencys2% {{D+"4t";<>> TrigramAssocMeasures._contingency(1, (1, 1, 1), (1, 73, 1), 2000) (1, 0, 0, 0, 0, 72, 0, 1927) r)n_iii n_iix_tuple n_ixx_tuplen_xxxn_iixn_ixin_xiin_ixxn_xixn_xxin_oiin_ioin_iion_ooin_oion_ioon_ooos r r$!TrigramAssocMeasures._contingencys!,u +u    %- %- %- %-5=EMeE%GGrcvUupp4pVpxUX-X-X-4X-U-U-X-U-U-X-U-U-4[U54$)zCalculates values of contingency table marginals from its values. >>> TrigramAssocMeasures._marginals(1, 0, 0, 0, 0, 72, 0, 1927) (1, (1, 1, 1), (1, 73, 1), 2000) r8) r)rrrrrrrrs r r*TrigramAssocMeasures._marginals&sm BM>eE%  ]EM5= 9 %- %- %-     rrN rprqrrrsrtr:rur$r*rwrrr rrs6& BHH$  rrc<\rSrSrSrSr\S5r\S5rSr g)QuadgramAssocMeasuresi9a  A collection of quadgram association measures. Each association measure is provided as a function with five arguments:: trigram_score_fn(n_iiii, (n_iiix, n_iixi, n_ixii, n_xiii), (n_iixx, n_ixix, n_ixxi, n_xixi, n_xxii, n_xiix), (n_ixxx, n_xixx, n_xxix, n_xxxi), n_all) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: - n_iiii counts ``(w1, w2, w3, w4)``, i.e. the quadgram being scored - n_ixxi counts ``(w1, *, *, w4)`` - n_xxxx counts ``(*, *, *, *)``, i.e. any quadgram rcUupVpxUupppUunnnnX- nXp- nX`- nX- U- U- nX- U- U- nX- U- U- nUU- U- U- U- U- U- U- nXP- nX- U- U- nX- U- U- nUU- U- U- U- U- U- U- nX- U- U- nUU- U- U- U- U- U- U- nX- U- U- U- U- U- U- n UU- U- U- U- U- U- U- U- U- U- U- U- U- U- U - n!UUUUUUUUUUUUUUU U!4$)zHCalculates values of a quadgram contingency table from marginal values. r)"n_iiii n_iiix_tuple n_iixx_tuple n_ixxx_tuplen_xxxxn_iiixn_iixin_ixiin_xiiin_iixxn_ixixn_ixxin_xixin_xxiin_xiixn_ixxxn_xixxn_xxixn_xxxin_oiiin_ioiin_iioin_ooiin_oioin_iooin_oooin_iiion_oiion_ioion_ooion_iioon_oioon_iooon_oooos" r r$"QuadgramAssocMeasures._contingencyPs ,8(;G8+7(6)F26)F26)F2&6)F2V;fDvMPVV6)F26)F2&6)F2V;fDvMPVV6)F2&6)F2V;fDvMPVV6)F2V;fDvMPVV                      (                !  rcUunnnnnnnnn n n n n nnnX-nX-nX-nX-nX-U -U -nX-U -U -nX-U-U-nX-U-U-nX-U-U-nX-U -U -nX-U-U -U-U -U -U-nX-U-U -U-U -U -U-nX-U-U -U-U -U -U -nX-U-U-U-U-U-U-n[U5nUUUUU4UUUUUU4UUUU4U4$)aCalculates values of contingency table marginals from its values. QuadgramAssocMeasures._marginals(1, 0, 2, 46, 552, 825, 2577, 34967, 1, 0, 2, 48, 7250, 9031, 28585, 356653) (1, (2, 553, 3, 1), (7804, 6, 3132, 1378, 49, 2), (38970, 17660, 100, 38970), 440540) r) r)rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr?s r r* QuadgramAssocMeasures._marginalss. #                6)F26)F26)F26)F26)F26)F26)F2V;fDvMPVV6)F2V;fDvMPVV6)F2V;fDvMPVV6)F2V;fDvMPVVK   VVV , VVVVV < VVV ,    rrNrrrr rr9s5( B9 9 v1 1 rrc.\rSrSrSrSr\S5rSrg)ContingencyMeasuresizWraps NgramAssocMeasures classes such that the arguments of association measures are contingency table values rather than marginals. c$SURR-URl[U5HYnURS5(aM[ X5nURS5(dUR X5n[ XU5 M[ g)zAConstructs a ContingencyMeasures given a NgramAssocMeasures class Contingency___N) __class__rpdir startswithgetattr_make_contingency_fnsetattr)selfmeasureskvs r __init__ContingencyMeasures.__init__sr"/(2D2D2M2M"MXA||D!!$A<<$$--h: DQ  rc\^^UU4SjnTRUlTRUlU$)zwFrom an association measure function, produces a new function which accepts contingency table values as its arguments. c(>T"TR"U66$r)r*)r)rold_fns r res5ContingencyMeasures._make_contingency_fn..ress8.. <= =r)rtrp)rrrs`` r r(ContingencyMeasures._make_contingency_fns%  >nn   rrN) rprqrrrsrtrrurrwrrr rrs     rr)rtmathr abcrr functoolsrr`logrer>rJ scipy.statsr ImportErrorrDrIrErrzrrrrrr rs ' ii 2 "( ) 7 9t(7t(nV(,V(r9 -9 xE .E PM """sA.. A:9A: