o rZŽhÙã@s0dZddlmZddlmZGdd„deƒZdS)a( Multi-Word Expression Tokenizer A ``MWETokenizer`` takes a string which has already been divided into tokens and retokenizes it, merging multi-word expressions into single tokens, using a lexicon of MWEs: >>> from nltk.tokenize import MWETokenizer >>> tokenizer = MWETokenizer([('a', 'little'), ('a', 'little', 'bit'), ('a', 'lot')]) >>> tokenizer.add_mwe(('in', 'spite', 'of')) >>> tokenizer.tokenize('Testing testing testing one two three'.split()) ['Testing', 'testing', 'testing', 'one', 'two', 'three'] >>> tokenizer.tokenize('This is a test in spite'.split()) ['This', 'is', 'a', 'test', 'in', 'spite'] >>> tokenizer.tokenize('In a little or a little bit or a lot in spite of'.split()) ['In', 'a_little', 'or', 'a_little_bit', 'or', 'a_lot', 'in_spite_of'] é)Ú TokenizerI)ÚTriec@s*eZdZdZd dd„Zdd„Zdd „ZdS) Ú MWETokenizerzhA tokenizer that processes tokenized text and merges multi-word expressions into single tokens. NÚ_cCs|sg}t|ƒ|_||_dS)a¥Initialize the multi-word tokenizer with a list of expressions and a separator :type mwes: list(list(str)) :param mwes: A sequence of multi-word expressions to be merged, where each MWE is a sequence of strings. :type separator: str :param separator: String that should be inserted between words in a multi-word expression token. (Default is '_') N)rÚ_mwesÚ _separator)ÚselfZmwesÚ separator©r ú@/var/www/auris/lib/python3.10/site-packages/nltk/tokenize/mwe.pyÚ__init__(s   zMWETokenizer.__init__cCs|j |¡dS)a–Add a multi-word expression to the lexicon (stored as a word trie) We use ``util.Trie`` to represent the trie. Its form is a dict of dicts. The key True marks the end of a valid MWE. :param mwe: The multi-word expression we're adding into the word trie :type mwe: tuple(str) or list(str) :Example: >>> tokenizer = MWETokenizer() >>> tokenizer.add_mwe(('a', 'b')) >>> tokenizer.add_mwe(('a', 'b', 'c')) >>> tokenizer.add_mwe(('a', 'x')) >>> expected = {'a': {'x': {True: None}, 'b': {True: None, 'c': {True: None}}}} >>> tokenizer._mwes == expected True N)rÚinsert)rZmwer r r Úadd_mwe9szMWETokenizer.add_mwecCsöd}t|ƒ}g}||kry|||jvrj|}|j}d}||kr?|||vr?|||}|d}tj|vr5|}||kr?|||vs$|dkrE|}tj|vsN|dkr^| |j |||…¡¡|}n| ||¡|d7}n | ||¡|d7}||ks |S)a¥ :param text: A list containing tokenized text :type text: list(str) :return: A list of the tokenized text with multi-words merged together :rtype: list(str) :Example: >>> tokenizer = MWETokenizer([('hors', "d'oeuvre")], separator='+') >>> tokenizer.tokenize("An hors d'oeuvre tonight, sir?".split()) ['An', "hors+d'oeuvre", 'tonight,', 'sir?'] réÿÿÿÿé)ÚlenrrZLEAFÚappendrÚjoin)rÚtextÚiÚnÚresultÚjZtrieZ last_matchr r r ÚtokenizeOs2  ü çzMWETokenizer.tokenize)Nr)Ú__name__Ú __module__Ú __qualname__Ú__doc__r rrr r r r r#s   rN)rZnltk.tokenize.apirZ nltk.utilrrr r r r Ús