o ZhZ @s6dZddlmZddlmZmZmZGdddZdS)z Tokenization utils for RoFormer.)List)NormalizedStringPreTokenizedString normalizersc@s>eZdZd ddZdededeefddZd efd d Z dS) JiebaPreTokenizerreturnNcCsH||_tjddddd|_zddl}Wn tytdw||_dS)NFT)Z clean_textZhandle_chinese_charsZ strip_accentsZ lowercaserzkYou need to install rjieba to use RoFormerTokenizer. See https://pypi.org/project/rjieba/ for installation.)vocabrZBertNormalizerrjieba ImportErrorjieba)selfrr r ^/var/www/auris/lib/python3.10/site-packages/transformers/models/roformer/tokenization_utils.py__init__s   zJiebaPreTokenizer.__init__inormalized_stringcCsg}|jjt|ddD]4\}}}||jvr ||||q |j|}|D]}|r?|t|}|||||}q*q |S)NF)Zhmm) r tokenizestrrappendrZ normalize_strsplitlen)r rrZsplitstokenstartendZ token_listr r r jieba_split(s  zJiebaPreTokenizer.jieba_splitpretokcCs||jdS)N)rr)r rr r r pre_tokenizeCszJiebaPreTokenizer.pre_tokenize)rN) __name__ __module__ __qualname__rintrrrrrr r r rrs rN)__doc__typingrZ tokenizersrrrrr r r rs