o vZh*@sDddlZGdddZGdddZGdddZGdd d ZdS) Nc@*eZdZdZd ddZddZddZd S) RegexBuilderaBuilds regex using arguments passed into a pattern template. Builds a regex object for which the pattern is made from an argument passed into a template. If more than one argument is passed (iterable), each pattern is joined by "|" (regex alternation 'or') to create a single pattern. Args: pattern_args (iterable): String element(s) to be each passed to ``pattern_func`` to create a regex pattern. Each element is ``re.escape``'d before being passed. pattern_func (callable): A 'template' function that should take a string and return a string. It should take an element of ``pattern_args`` and return a valid regex pattern group string. flags: ``re`` flag(s) to compile with the regex. Example: To create a simple regex that matches on the characters "a", "b", or "c", followed by a period:: >>> rb = RegexBuilder('abc', lambda x: "{}\.".format(x)) Looking at ``rb.regex`` we get the following compiled regex:: >>> print(rb.regex) 'a\.|b\.|c\.' The above is fairly simple, but this class can help in writing more complex repetitive regex, making them more readable and easier to create by using existing data structures. Example: To match the character following the words "lorem", "ipsum", "meili" or "koda":: >>> words = ['lorem', 'ipsum', 'meili', 'koda'] >>> rb = RegexBuilder(words, lambda x: "(?<={}).".format(x)) Looking at ``rb.regex`` we get the following compiled regex:: >>> print(rb.regex) '(?<=lorem).|(?<=ipsum).|(?<=meili).|(?<=koda).' rcCs ||_||_||_||_dSN) pattern_args pattern_funcflags_compileregex)selfrrrr B/var/www/auris/lib/python3.10/site-packages/gtts/tokenizer/core.py__init__3szRegexBuilder.__init__cCsFg}|jD]}t|}||}||qd|}t||jS)N|)rreescaperappendjoincompiler)r altsargaltpatternr r r r;s     zRegexBuilder._compilecCs t|jSr)strr r r r r __repr__E zRegexBuilder.__repr__Nr)__name__ __module__ __qualname____doc__r rrr r r r rs  - rc@r) PreProcessorRegexaRegex-based substitution text pre-processor. Runs a series of regex substitutions (``re.sub``) from each ``regex`` of a :class:`gtts.tokenizer.core.RegexBuilder` with an extra ``repl`` replacement parameter. Args: search_args (iterable): String element(s) to be each passed to ``search_func`` to create a regex pattern. Each element is ``re.escape``'d before being passed. search_func (callable): A 'template' function that should take a string and return a string. It should take an element of ``search_args`` and return a valid regex search pattern string. repl (string): The common replacement passed to the ``sub`` method for each ``regex``. Can be a raw string (the case of a regex backreference, for example) flags: ``re`` flag(s) to compile with each `regex`. Example: Add "!" after the words "lorem" or "ipsum", while ignoring case:: >>> import re >>> words = ['lorem', 'ipsum'] >>> pp = PreProcessorRegex(words, ... lambda x: "({})".format(x), r'\\1!', ... re.IGNORECASE) In this case, the regex is a group and the replacement uses its backreference ``\\1`` (as a raw string). Looking at ``pp`` we get the following list of search/replacement pairs:: >>> print(pp) (re.compile('(lorem)', re.IGNORECASE), repl='\1!'), (re.compile('(ipsum)', re.IGNORECASE), repl='\1!') It can then be run on any string of text:: >>> pp.run("LOREM ipSuM") "LOREM! ipSuM!" See :mod:`gtts.tokenizer.pre_processors` for more examples. rcCs6||_g|_|D]}t|g||}|j|jqdSr)replregexesrrr )r Z search_args search_funcr"rrrbr r r r vs zPreProcessorRegex.__init__cCs|jD] }||j|}q|S)zRun each regex substitution on ``text``. Args: text (string): the input text. Returns: string: text after all substitutions have been sequentially applied. )r#subr")r textr r r r runs zPreProcessorRegex.runcCs.g}|jD] }|d||jqd|S)Nz({}, repl='{}'), )r#rformatr"r)r Z subs_strsrr r r rs  zPreProcessorRegex.__repr__Nrrrrr r r(rr r r r r!Is  , r!c@r) PreProcessorSuba@Simple substitution text preprocessor. Performs string-for-string substitution from list a find/replace pairs. It abstracts :class:`gtts.tokenizer.core.PreProcessorRegex` with a default simple substitution regex. Args: sub_pairs (list): A list of tuples of the style ``(, )`` ignore_case (bool): Ignore case during search. Defaults to ``True``. Example: Replace all occurrences of "Mac" to "PC" and "Firefox" to "Chrome":: >>> sub_pairs = [('Mac', 'PC'), ('Firefox', 'Chrome')] >>> pp = PreProcessorSub(sub_pairs) Looking at the ``pp``, we get the following list of search (regex)/replacement pairs:: >>> print(pp) (re.compile('Mac', re.IGNORECASE), repl='PC'), (re.compile('Firefox', re.IGNORECASE), repl='Chrome') It can then be run on any string of text:: >>> pp.run("I use firefox on my mac") "I use Chrome on my PC" See :mod:`gtts.tokenizer.pre_processors` for more examples. Tc CsNdd}|r tjnd}g|_|D]}|\}}t|g|||}|j|qdS)NcSs d|S)Nz{})r*)xr r r r$rz-PreProcessorSub.__init__..search_funcr)rIpre_processorsr!r) r Z sub_pairsZ ignore_caser$rZsub_pairrr"ppr r r r szPreProcessorSub.__init__cCs|jD]}||}q|S)zRun each substitution on ``text``. Args: text (string): the input text. Returns: string: text after all substitutions have been sequentially applied. )r0r()r r'r1r r r r(s zPreProcessorSub.runcCsddd|jDS)Nr)cSsg|]}t|qSr )r).0r1r r r sz,PreProcessorSub.__repr__..)rr0rr r r rszPreProcessorSub.__repr__N)Tr,r r r r r-s  ! r-c@s6eZdZdZejfddZddZddZdd Z d S) Tokenizera An extensible but simple generic rule-based tokenizer. A generic and simple string tokenizer that takes a list of functions (called `tokenizer cases`) returning ``regex`` objects and joins them by "|" (regex alternation 'or') to create a single regex to use with the standard ``regex.split()`` function. ``regex_funcs`` is a list of any function that can return a ``regex`` (from ``re.compile()``) object, such as a :class:`gtts.tokenizer.core.RegexBuilder` instance (and its ``regex`` attribute). See the :mod:`gtts.tokenizer.tokenizer_cases` module for examples. Args: regex_funcs (list): List of compiled ``regex`` objects. Each function's pattern will be joined into a single pattern and compiled. flags: ``re`` flag(s) to compile with the final regex. Defaults to ``re.IGNORECASE`` Note: When the ``regex`` objects obtained from ``regex_funcs`` are joined, their individual ``re`` flags are ignored in favour of ``flags``. Raises: TypeError: When an element of ``regex_funcs`` is not a function, or a function that does not return a compiled ``regex`` object. Warning: Joined ``regex`` patterns can easily interfere with one another in unexpected ways. It is recommended that each tokenizer case operate on distinct or non-overlapping characters/sets of characters (For example, a tokenizer case for the period (".") should also handle not matching/cutting on decimals, instead of making that a separate tokenizer case). Example: A tokenizer with a two simple case (*Note: these are bad cases to tokenize on, this is simply a usage example*):: >>> import re, RegexBuilder >>> >>> def case1(): ... return re.compile("\,") >>> >>> def case2(): ... return RegexBuilder('abc', lambda x: "{}\.".format(x)).regex >>> >>> t = Tokenizer([case1, case2]) Looking at ``case1().pattern``, we get:: >>> print(case1().pattern) '\\,' Looking at ``case2().pattern``, we get:: >>> print(case2().pattern) 'a\\.|b\\.|c\\.' Finally, looking at ``t``, we get them combined:: >>> print(t) 're.compile('\\,|a\\.|b\\.|c\\.', re.IGNORECASE) from: [, ]' It can then be run on any string of text:: >>> t.run("Hello, my name is Linda a. Call me Lin, b. I'm your friend") ['Hello', ' my name is Linda ', ' Call me Lin', ' ', " I'm your friend"] c CsJ||_||_z||_WdSttfy$}ztdt|d}~ww)Nz`Tokenizer() expects a list of functions returning regular expression objects (i.e. re.compile). ) regex_funcsr_combine_regex total_regex TypeErrorAttributeErrorr)r r5rer r r r "szTokenizer.__init__cCs>g}|jD]}||qddd|D}t||jS)Nrcss|]}|jVqdSr)r)r2rr r r 4sz+Tokenizer._combine_regex..)r5rrrrr)r rfuncrr r r r6/s  zTokenizer._combine_regexcCs |j|S)zTokenize `text`. Args: text (string): the input text to tokenize. Returns: list: A list of strings (token) split according to the tokenizer cases. )r7split)r r'r r r r(7s z Tokenizer.runcCst|jdt|jS)Nz from: )rr7r5rr r r rCszTokenizer.__repr__N) rrrr r IGNORECASEr r6r(rr r r r r4s J  r4)rrr!r-r4r r r r s DLB