o rZhn@s,dZddlZddlZddlZddlmZmZmZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZmZmZedgdZGdddZ GdddZ!GdddZ"GdddZ#Gddde#Z$ddZ%e&dkre%gdZ'dS)a  This module brings together a variety of NLTK functionality for text analysis, and provides simple, interactive interfaces. Functionality includes: concordancing, collocation discovery, regular expression search over tokenized strings, and distributional similarity. N)Counter defaultdict namedtuple)reduce)log)BigramCollocationFinder)MLE)padded_everygram_pipeline)BigramAssocMeasures f_measure)ConditionalFreqDist)FreqDist) sent_tokenize)LazyConcatenation cut_string tokenwrapConcordanceLine)leftqueryrightoffset left_print right_printlinec@sTeZdZdZeddZddddfddZd d Zd d ZdddZ dddZ dS) ContextIndexa A bidirectional index between words and their 'contexts' in a text. The context of a word is usually defined to be the words that occur in a fixed window around the word; but other definitions may also be used by providing a custom context function. cCsH|dkr ||dnd}|t|dkr||dnd}||fS)z;One left token and one right token, normalized to lowercaser*START**END*)lowerlen)tokensirrr"8/var/www/auris/lib/python3.10/site-packages/nltk/text.py_default_context/s$zContextIndex._default_contextNcC|SNr"xr"r"r#6zContextIndex.csv|__|r |_nj_rfddDtfddtD_tfddtD_dS)Ncsg|]}|r|qSr"r").0t)filterr"r# >z)ContextIndex.__init__..c3s*|]\}}||fVqdSr&)_key _context_funcr+r!wselfr r"r# ? z(ContextIndex.__init__..c3s*|]\}}||fVqdSr&)r1r0r2r4r"r#r6Br7)r0_tokensr1r$CFD enumerate_word_to_contexts_context_to_words)r5r Z context_funcr-keyr")r-r5r r#__init__6s zContextIndex.__init__cC|jS)zw :rtype: list(str) :return: The document that this context index was created from. r8r5r"r"r#r FzContextIndex.tokenscCsF||}t|j|}i}|jD] \}}t|t|||<q|S)z Return a dictionary mapping from words to 'similarity scores,' indicating how often these two words occur in the same context. )r0setr;itemsr )r5wordZ word_contextsscoresr3Z w_contextsr"r"r#word_similarity_dictNs z!ContextIndex.word_similarity_dictcCsvtt}|j||D]"}|j|D]}||kr-|||j|||j||7<qq t||jddd|S)NT)r=reverse)rintr;r0r<sortedget)r5rEnrFcr3r"r"r# similar_words]szContextIndex.similar_wordsFcsfddDfddDfddttD}ttj|r2|r2tdds7tStfddD}|S) a Find contexts where the specified words can all appear; and return a frequency distribution mapping each context to the number of times that context was used. :param words: The words used to seed the similarity search :type words: str :param fail_on_unknown: If true, then raise a value error if any of the given words do not occur at all in the index. cg|]}|qSr")r0r+r3rAr"r#r.rz0ContextIndex.common_contexts..csg|] }tj|qSr")rCr;rQrAr"r#r.sscsg|] }|s|qSr"r")r+r!)contextswordsr"r#r.tsz%The following word(s) were not found: c3s,|]}j|D] }|vr |Vq qdSr&)r;r+r3rN)commonr5r"r#r6|s  z/ContextIndex.common_contexts..)rangerrrC intersection ValueErrorjoinr )r5rTZfail_on_unknownemptyfdr")rWrSr5rTr#common_contextsgs  zContextIndex.common_contextsrH)F) __name__ __module__ __qualname____doc__ staticmethodr$r>r rGrOr^r"r"r"r#r's   rc@sLeZdZdZddfddZddZdd Zd d Zdd dZdddZ dS)ConcordanceIndexzs An index that can be used to look up the offset locations at which a given word occurs in a document. cCr%r&r"r'r"r"r#r)r*zConcordanceIndex.cCsL||_ ||_ tt|_ t|D]\}}||}|j||qdS)a Construct a new concordance index. :param tokens: The document (list of tokens) that this concordance index was created from. This list can be used to access the context of a given word occurrence. :param key: A function that maps each token to a normalized version that will be used as a key in the index. E.g., if you use ``key=lambda s:s.lower()``, then the index will be case-insensitive. N)r8r0rlist_offsetsr:append)r5r r=indexrEr"r"r#r>s   zConcordanceIndex.__init__cCr?)z{ :rtype: list(str) :return: The document that this concordance index was created from. r@rAr"r"r#r rBzConcordanceIndex.tokenscCs||}|j|S)z :rtype: list(int) :return: A list of the offset positions at which the given word occurs. If a key function was specified for the index, then given word's key will be looked up. )r0rgr5rEr"r"r#offsetss  zConcordanceIndex.offsetscCsdt|jt|jfS)Nz+)rr8rgrAr"r"r#__repr__szConcordanceIndex.__repr__Pc sRt|tr|}n|g}d|}tdd|D}||dd}|d}g}||d} t|ddD]\}fd d ||D} t| | } q6| r| D]Td|jt |} |jt d|} |jt ||} t d| |  |}t d| |}d|| |g}t | | | |||}||qR|S) z Find all concordance lines given the query word. Provided with a list of words, these will be found as a phrase. rUcss|] }t|sdVqdS)rN) unicodedata combining)r+charr"r"r#r6sz4ConcordanceIndex.find_concordance..rrNcsh|]}|dqS)rr")r+rr!r"r# r/z4ConcordanceIndex.find_concordance..) isinstancerfr[sumrkr:rKrYr8rmaxrrjustrrh)r5rEwidthphraseZ phrase_strZ phrase_lenZ half_widthcontextconcordance_listrkZ word_offsetsZ query_wordZ left_contextZ right_contextrrZ line_printconcordance_liner"rsr#find_concordancesB   z!ConcordanceIndex.find_concordancecCsn|j||d}|stddSt|t|}td|dt|dt|d|D] \}}t|jq+dS)a Print concordance lines given the query word. :param word: The target word or phrase (a list of strings) :type word: str or list :param lines: The number of lines to display (default=25) :type lines: int :param width: The width of each line, in characters (default=80) :type width: int :param save: The option to save the concordance. :type save: bool )ryz no matchesz Displaying z of z matches:N)r~printminrr:r)r5rErylinesr|r!r}r"r"r#print_concordances   z"ConcordanceIndex.print_concordanceN)rm)rmr) r`rarbrcr>r rkrlr~rr"r"r"r#res 0rec@s eZdZdZddZddZdS) TokenSearchera A class that makes it easier to use regular expressions to search over tokenized strings. The tokenized string is converted to a string where tokens are marked with angle brackets -- e.g., ``''``. The regular expression passed to the ``findall()`` method is modified to treat angle brackets as non-capturing parentheses, in addition to matching the token boundaries; and to have ``'.'`` not match the angle brackets. cCsddd|D|_dS)Ncss|] }d|dVqdS)<>Nr"rQr"r"r#r6 sz)TokenSearcher.__init__..)r[_rawr4r"r"r#r> szTokenSearcher.__init__cCs~tdd|}tdd|}tdd|}tdd|}t||j}|D]}|ds5|dr5td q%d d |D}|S) a Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> from nltk.text import TokenSearcher >>> from nltk.book import text1, text5, text9 >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str z\srrz(?:<(?:rz)>)z (?]z$Bad regexp for TokenSearcher.findallcSsg|] }|dddqS)rz><splitr+hr"r"r#r.5sz)TokenSearcher.findall..)resubfindallr startswithendswithrZ)r5regexphitsrr"r"r#rszTokenSearcher.findallN)r`rarbrcr>rr"r"r"r#rs rc@seZdZdZdZd6ddZddZdd Zd7d d Zd7ddZ d8ddZ d8ddZ ddZ ddZ ddZd9ddZd9ddZd d!Zd:d#d$Zd;d'd(Zd)d*Zd+d,Zd-d.Zed/Zd0d1Zd2d3Zd4d5ZdS)<Texta A wrapper around a sequence of simple (string) tokens, which is intended to support initial exploration of texts (via the interactive console). Its methods perform a variety of analyses on the text's contexts (e.g., counting, concordancing, collocation discovery), and display the results. If you wish to write a program which makes use of these analyses, then you should bypass the ``Text`` class, and use the appropriate analysis function or class directly instead. A ``Text`` is typically initialized from a given document or corpus. E.g.: >>> import nltk.corpus >>> from nltk.text import Text >>> moby = Text(nltk.corpus.gutenberg.words('melville-moby_dick.txt')) TNcCs|jrt|}||_|r||_dSd|ddvr3|ddd}ddd|d|D|_dSddd|dd Dd |_dS) zv Create a Text object. :param tokens: The source text. :type tokens: sequence of str ]NrHrUcs|]}t|VqdSr&strr+tokr"r"r#r6bz Text.__init__..rcsrr&rrr"r"r#r6drz...) _COPY_TOKENSrfr namerir[)r5r rendr"r"r#r>Ss "&z Text.__init__cCs |j|Sr&)r )r5r!r"r"r# __getitem__j zText.__getitem__cCs t|jSr&)rr rAr"r"r#__len__mrz Text.__len__OrcCs.d|jvrt|jddd|_|j|||S)a Prints a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` _concordance_indexcS|Sr&rsr"r"r#r)z"Text.concordance..r=)__dict__rer rrr5rEryrr"r"r# concordancets  zText.concordancecCs4d|jvrt|jddd|_|j||d|S)a Generate a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` rcSrr&rrr"r"r#r)rz'Text.concordance_list..rN)rrer rr~rr"r"r#r|s  zText.concordance_listrHrqcsd|jvr|j|kr|j|ksB||_||_ddlm}|dt|j|}| d| fddt }t | |j||_|jS)a Return collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocation_list()[:2] [('United', 'States'), ('fellow', 'citizens')] :param num: The maximum number of collocations to return. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int :rtype: list(tuple(str, str)) _collocationsr) stopwordsenglishrqcst|dkp |vS)N)rr)r3Z ignored_wordsr"r#r)r/z'Text.collocation_list..)rZ_numZ _window_size nltk.corpusrrTrZ from_wordsr Zapply_freq_filterZapply_word_filterr rfZnbestZlikelihood_ratior)r5num window_sizerfinderZbigram_measuresr"rr#collocation_lists       zText.collocation_listcCs*dd|||D}tt|dddS)a Print collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocations() # doctest: +NORMALIZE_WHITESPACE United States; fellow citizens; years ago; four years; Federal Government; General Government; American people; Vice President; God bless; Chief Justice; one another; fellow Americans; Old World; Almighty God; Fellow citizens; Chief Magistrate; every citizen; Indian tribes; public debt; foreign nations :param num: The maximum number of collocations to print. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int cSsg|] \}}|d|qSrUr"r+Zw1Zw2r"r"r#r.sz%Text.collocations..; ) separatorN)rrr)r5rrZcollocation_stringsr"r"r# collocationss zText.collocationscC |j|S)zJ Count the number of times this word appears in the text. )r countrjr"r"r#r z Text.countcCr)zQ Find the index of the first occurrence of the word in the text. )r rirjr"r"r#rirz Text.indexcCstr&)NotImplementedError)r5methodr"r"r# readabilityszText.readabilitycsd|jvrt|jddddd|_|jjvrGttfddD}dd | |D}t t |d St d d S) a~ Distributional similarity: find other words which appear in the same contexts as the specified word; list most similar words first. :param word: The word used to seed the similarity search :type word: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.similar_words() _word_context_indexcSrr&)isalphar'r"r"r#r)rzText.similar..cSrr&rrr"r"r#r)r)r-r=c3s2|]}|D] }|vr|ks|VqqdSr&r"rVrSZwcirEr"r#r6szText.similar..cSg|]\}}|qSr"r"r+r3_r"r"r#r.z Text.similar..z No matchesN) rrr rrr;Z conditionsrCr most_commonrr)r5rErr]rTr"rr#similars    z Text.similarc Csd|jvrt|jddd|_z(|j|d}|s tdWd Sdd||D}ttd d |DWd StyN}z t|WYd }~d Sd }~ww) aY Find contexts where the specified words appear; list most frequent common contexts first. :param words: The words used to seed the similarity search :type words: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.common_contexts() rcSrr&rrr"r"r#r)rz&Text.common_contexts..rTzNo common contexts were foundcSrr"r"rr"r"r#r. rz(Text.common_contexts..css |] \}}|d|VqdS)rNr"rr"r"r#r6!sz'Text.common_contexts..N) rrr rr^rrrrZ)r5rTrr]Zranked_contextser"r"r#r^ s  zText.common_contextscCsddlm}|||dS)z Produce a plot showing the distribution of the words through the text. Requires pylab to be installed. :param words: The words to be plotted :type words: list(str) :seealso: nltk.draw.dispersion_plot() r)dispersion_plotN)Z nltk.drawr)r5rTrr"r"r#r&s zText.dispersion_plotrcCs(t||\}}t|d}||||S)N)order)r rfit)r5Ztokenized_sentsrMZ train_dataZ padded_sentsmodelr"r"r#_train_default_ngram_lm3s  zText._train_default_ngram_lmd*c Csddtd|jD|_t|ds#tdtjd|j|jdd|_ g}|d ks-Jd t ||kr]t |j j |||d D]\}}|d krGq>|d krMn| |q>|d7}t ||ks3|rfd|dnd}|t|d|}t||S)a Print random text, generated using a trigram language model. See also `help(nltk.lm)`. :param length: The length of text to generate (default=100) :type length: int :param text_seed: Generation can be conditioned on preceding context. :type text_seed: list(str) :param random_seed: A random seed or an instance of `random.Random`. If provided, makes the random sampling part of generation reproducible. (default=42) :type random_seed: int cSsg|]}|dqSrr)r+sentr"r"r#r.Is z!Text.generate..rU_trigram_modelzBuilding ngram index...)filer)rMrz!The `length` must be more than 0.) text_seed random_seedzzrrN)rr[r Z_tokenized_sentshasattrrsysstderrrrrr:generaterhr) r5lengthrrZgenerated_tokensidxtokenprefixZ output_strr"r"r#r9s6      z Text.generatecGs|j|S)zc See documentation for FreqDist.plot() :seealso: nltk.prob.FreqDist.plot() )vocabplot)r5argsr"r"r#rgsz Text.plotcCsd|jvr t||_|jS)z. :seealso: nltk.prob.FreqDist _vocab)rr rrAr"r"r#rns  z Text.vocabcCs@d|jvr t||_|j|}dd|D}tt|ddS)a Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> from nltk.book import text1, text5, text9 >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str _token_searchercSsg|]}d|qSr)r[rr"r"r#r.rRz Text.findall..rN)rrrrrr)r5rrr"r"r#rws   z Text.findallz \w+|[\.\!\?]cCs|d}|dkr |j||s |d8}|dkr |j||r|dkr(||nd}|d}|t|krN|j||sN|d7}|t|krN|j||r<|t|krX||nd}||fS)z One left & one right token, both case-normalized. Skip over non-sentence-final punctuation. Used by the ``ContextIndex`` that is created for ``similar()`` and ``common_contexts()``. rrrr) _CONTEXT_REmatchr)r5r r!jrrr"r"r#_contextsz Text._contextcC d|jSNz rrAr"r"r#__str__rz Text.__str__cCrrrrAr"r"r#rlrz Text.__repr__r&)rr)rHrqr_)r)rNr)r`rarbrcrr>rrrr|rrrrirrr^rrrrrrrcompilerrrrlr"r"r"r#r9s2     #  " . " rc@s0eZdZdZddZddZddZdd Zd S) TextCollectiona;A collection of texts, which can be loaded with list of texts, or with a corpus consisting of one or more texts, and which supports counting, concordancing, collocation discovery, etc. Initialize a TextCollection as follows: >>> import nltk.corpus >>> from nltk.text import TextCollection >>> from nltk.book import text1, text2, text3 >>> gutenberg = TextCollection(nltk.corpus.gutenberg) >>> mytexts = TextCollection([text1, text2, text3]) Iterating over a TextCollection produces all the tokens of all the texts in order. cs@tdrfddD|_t|ti|_dS)NrTcrPr")rT)r+fsourcer"r#r.rRz+TextCollection.__init__..)rZfileids_textsrr>r _idf_cache)r5rr"rr#r>s  zTextCollection.__init__cCs||t|S)z"The frequency of the term in text.)rrr5termtextr"r"r#tfszTextCollection.tfcsj|j}|dur3tfdd|jD}t|jdkr!td|r,tt|j|nd}||j<|S)zThe number of texts in the corpus divided by the number of texts that the term appears in. If a term does not appear in the corpus, 0.0 is returned.Ncsg|]}|vrdqS)Tr")r+rrr"r#r.r/z&TextCollection.idf..rz+IDF undefined for empty document collectiong)rrLrrrZr)r5ridfmatchesr"rr#rs  zTextCollection.idfcCs|||||Sr&)rrrr"r"r#tf_idfszTextCollection.tf_idfN)r`rarbrcr>rrrr"r"r"r#rs  rcCsddlm}t|jdd}t|ttd|dttd|dttd|ttd|gd ttd | d ttd td |dtd|ddtd| ddS)Nr)brownnews) categoriesz Concordance:zDistributionally similar words:z Collocations:zDispersion plot:)rreportZsaidZ announcedzVocabulary plot:2z Indexing:ztext[3]:rz text[3:5]:ztext.vocab()['news']:) rrrrTrrrrrrr)rrr"r"r#demos.    r__main__)rrerrr)(rcrrrn collectionsrrr functoolsrmathrZnltk.collocationsrZnltk.lmrZnltk.lm.preprocessingr Z nltk.metricsr r Znltk.probabilityr r9r Z nltk.tokenizerZ nltk.utilrrrrrrerrrrr`__all__r"r"r"r#s>        [8.