o rZh@sddZddlmZmZddlmZmZmZddlm Z ddl m Z GdddeZ Gdd d e Z d S) z Tokenizer Interface )ABCabstractmethod)IteratorListTuple) overridden)string_span_tokenizec@seZdZdZededeefddZdedee e e ffddZ deedeeefd d Z deedeee e e ffd d Z d S) TokenizerIz A processing interface for tokenizing a string. Subclasses must define ``tokenize()`` or ``tokenize_sents()`` (or both). sreturncCst|jr ||gdSdS)zL Return a tokenized copy of *s*. :rtype: List[str] rN)rtokenize_sentsselfr r@/var/www/auris/lib/python3.10/site-packages/nltk/tokenize/api.pytokenizes zTokenizerI.tokenizecCst)z Identify the tokens using integer offsets ``(start_i, end_i)``, where ``s[start_i:end_i]`` is the corresponding token. :rtype: Iterator[Tuple[int, int]] NotImplementedErrorr rrr span_tokenize$szTokenizerI.span_tokenizestringscsfdd|DS)z Apply ``self.tokenize()`` to each element of ``strings``. I.e.: return [self.tokenize(s) for s in strings] :rtype: List[List[str]] csg|]}|qSr)r).0r rrr 5sz-TokenizerI.tokenize_sents..r)rrrrrr -szTokenizerI.tokenize_sentsccs |D] }t||VqdS)z Apply ``self.span_tokenize()`` to each element of ``strings``. I.e.: return [self.span_tokenize(s) for s in strings] :yield: List[Tuple[int, int]] N)listr)rrr rrrspan_tokenize_sents7s zTokenizerI.span_tokenize_sentsN)__name__ __module__ __qualname____doc__rstrrrrrintrr rrrrrr s   r c@s0eZdZdZeeddZddZddZdS) StringTokenizerzxA tokenizer that divides a string into substrings by splitting on the specified string (defined in subclasses). cCstNrrrrr_stringJszStringTokenizer._stringcCs ||jSr")splitr#r rrrrOs zStringTokenizer.tokenizeccst||jEdHdSr")rr#r rrrrRszStringTokenizer.span_tokenizeN) rrrrpropertyrr#rrrrrrr!Es  r!N)rabcrrtypingrrrZnltk.internalsrZnltk.tokenize.utilrr r!rrrrs  1