fTh$SrSSKrSSKrSSKrSSKJrJrJrJrJ r J r SSK J r SSK JrJrJr \"5(aSSKr\"5(aSSKr\R("\5rSS0rS r"S S \ 5rS /rg) zTokenization class for VITS.N)AnyDictListOptionalTupleUnion)PreTrainedTokenizer)is_phonemizer_availableis_uroman_availablelogging vocab_filez vocab.jsonc\[R"S5nURU5nUSLnU$)Nz [^\x00-\x7F])recompilesearch) input_stringnon_roman_patternmatch has_non_romans b/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/vits/tokenization_vits.pyhas_non_roman_charactersr%s3 ?3  $ $\ 2E%M c ^\rSrSrSr\rSS/rSSU4Sjjjr\ S5r S r S r S r SS \S \S\\S\\\\\444SjjrS \S\\4SjrS\\S\4SjrSrSrSS\S\\S\\\S44SjjrSrU=r$) VitsTokenizer/a| Construct a VITS tokenizer. Also supports MMS-TTS. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. language (`str`, *optional*): Language identifier. add_blank (`bool`, *optional*, defaults to `True`): Whether to insert token id 0 in between the other tokens. normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the input text by removing all casing and punctuation. phonemize (`bool`, *optional*, defaults to `True`): Whether to convert the input text into phonemes. is_uroman (`bool`, *optional*, defaults to `False`): Whether the `uroman` Romanizer needs to be applied to the input text prior to tokenizing. input_idsattention_maskNreturnc d>[USS9n [R"U 5UlSSS5 URR 5V V s0sHupX_M sn n UlX@lXPlX`lXpl Xl [T U]0"SUUUUUUUS.U D6 g!,(df  Ny=fs sn n f)Nutf-8encoding) pad_token unk_tokenlanguage add_blank normalize phonemize is_uroman) openjsonloadencoderitemsdecoderr&r'r(r)r*super__init__)selfrr$r%r&r'r(r)r*kwargs vocab_handlekv __class__s rr3VitsTokenizer.__init__Hs*w /<99\2DL0*.););)=>)=)=>  """"    0 /?sB B, B)c,[UR5$N)lenr/)r4s r vocab_sizeVitsTokenizer.vocab_sizejs4<<  rc[UR5Vs0sHoRU5U_M nnURUR5 U$s snfr<)ranger>convert_ids_to_tokensupdateadded_tokens_encoder)r4ivocabs r get_vocabVitsTokenizer.get_vocabnsL;@;QR;Qa++A.1;QR T../ SsAc[URR55[URR55-nSnSnU[ U5:agSnUH-nXU[ U5-U:XdMX6- nU[ U5- nSn O U(dX1UR 5- nUS- nU[ U5:aMgU$)zfLowercase the input string, respecting any special token ids that may be part or entirely upper-cased.rFT)listr/keysrDr=lower)r4rall_vocabulary filtered_textrE found_matchwords rnormalize_textVitsTokenizer.normalize_textssdll//12T$:S:S:X:X:Z5[[ #l##K&AD M2d:!)MTNA"&K 'a!6!6!88 Q#l##rcJURS:XaURSS5nU$)z4Special treatment of characters in certain languagesronuțuţ)r&replace)r4texts r_preprocess_charVitsTokenizer._preprocess_chars# ==E !<<d+D rrXis_split_into_wordsr(c ^UbUO TRnU(aTRU5nTRU5n[U5(a\TR(aK[ 5(d[ RS5 O&[R"5nURU5nTR(aN[5(d [S5e[R"USSSSSS9n[R "SSU5nXT4$U(a6S R#[%['U4S jU555R)5nXT4$) am Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. normalize (`bool`, *optional*, defaults to `None`): Whether or not to apply punctuation and casing normalization to the text inputs. Typically, VITS is trained on lower-cased and un-punctuated text. Hence, normalization is used to ensure that the input text consists only of lower-case characters. kwargs (`Dict[str, Any]`, *optional*): Keyword arguments to use for the tokenization. Returns: `Tuple[str, Dict[str, Any]]`: The prepared text and the unused kwargs. aCText to the tokenizer contains non-Roman characters. To apply the `uroman` pre-processing step automatically, ensure the `uroman` Romanizer is installed with: `pip install uroman` Note `uroman` requires python version >= 3.10Otherwise, apply the Romanizer manually as per the instructions: https://github.com/isi-nlp/uromanzEPlease install the `phonemizer` Python package to use this tokenizer.zen-usespeakT)r&backendstrippreserve_punctuation with_stressz\s+ rJc">UTR;$r<)r/)charr4s r8VitsTokenizer.prepare_for_tokenization..sTT\\=Qr)r(rSrYrr*r loggerwarningurUromanromanize_stringr)r ImportError phonemizerrsubjoinrLfilterr_)r4rXr[r(r5rPuromans` rprepare_for_tokenization&VitsTokenizer.prepare_for_tokenizations4"+!6IDNN &&t,D--d3 #M 2 2t~~&((y & 6 6} E >>*,,!"ijj&00  %) MFF63 >M $$ GGD0QS`)a$bciikM$$rc[U5nUR(a-URS5/[U5S-S--nX#SSS2'UnU$)z]Tokenize a string by inserting the `` token at the boundary between adjacent characters.rrKN)rLr'_convert_id_to_tokenr=)r4rXtokens intersperseds r _tokenizeVitsTokenizer._tokenizesOd >> 55a89S[1_q=PQL!'A !F rrwctUR(a[U5S:aUSSS2nSRU5$)NrKrurJ)r'r=ro)r4rws rconvert_tokens_to_string&VitsTokenizer.convert_tokens_to_strings0 >>c&kAoADqD\Fwwvrc~URRXRRUR55$)z0Converts a token (str) in an id using the vocab.)r/getr%)r4tokens r_convert_token_to_id"VitsTokenizer._convert_token_to_ids*||||'7'7'GHHrc8URRU5$)z=Converts an index (integer) in a token (str) using the vocab.)r1r)r4indexs rrv"VitsTokenizer._convert_id_to_tokens||&&rsave_directoryfilename_prefixc [RRU5(d[R SUS35 g[RR X(aUS-OS[ S-5n[USSS9nUR[R"URS S S S 9S -5 SSS5 U4$!,(df  U4$=f)NzVocabulary path (z) should be a directory-rJrwr!r"ruTF)indent sort_keys ensure_ascii ) ospathisdirrgerrorroVOCAB_FILES_NAMESr,writer-dumpsr/)r4rrrfs rsave_vocabularyVitsTokenizer.save_vocabularysww}}^,, LL,^,<> C)r'r1r/r*r&r(r))zzNTTTF)rN)FNr<)__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesr3propertyr>rGrSrYstrboolrrrrrrrryr|rrvrr__static_attributes__ __classcell__)r9s@rrr/s**$&67       D!! *Y]?%?%.2?%GOPT~?% sDcN" #?%B c d3i tCyS I' c HSM ]bchilcmoscs]t  rr)rr-rrtypingrrrrrrtokenization_utilsr utilsr r r rmrqri get_loggerrrgrrr__all__r+rrrsv# ::5JJ   H %!<0D'DN  r