o Zh$@sdZddlZddlZddlZddlmZmZmZmZm Z m Z ddl m Z ddl mZmZmZer5ddlZerNTFreturnc st|dd} t| |_Wdn1swYdd|jD|_||_||_||_||_ ||_ t j d|||||||d| dS)Nutf-8encodingcSsi|]\}}||qSrr).0kvrrr Wsz*VitsTokenizer.__init__..) pad_token unk_tokenlanguage add_blank normalize phonemize is_uromanr) openjsonloadencoderitemsdecoderr#r$r%r&r'super__init__) selfr r!r"r#r$r%r&r'kwargsZ vocab_handle __class__rrr/Hs(  zVitsTokenizer.__init__cCs t|jSN)lenr+r0rrr vocab_sizejs zVitsTokenizer.vocab_sizecs(fddtjD}|j|S)Ncsi|]}||qSr)Zconvert_ids_to_tokens)rir6rrr osz+VitsTokenizer.get_vocab..)ranger7updateadded_tokens_encoder)r0Zvocabrr6r get_vocabns zVitsTokenizer.get_vocabcCst|jt|j}d}d}|t|krMd}|D]}|||t||kr8||7}|t|7}d}nq|sG|||7}|d7}|t|ks|S)zfLowercase the input string, respecting any special token ids that may be part or entirely upper-cased.rFT)listr+keysr;r5lower)r0rZall_vocabulary filtered_textr8 found_matchwordrrrnormalize_textss"    zVitsTokenizer.normalize_textcCs|jdkr |dd}|S)z4Special treatment of characters in certain languagesZronuțuţ)r#replace)r0textrrr_preprocess_chars  zVitsTokenizer._preprocess_charrGis_split_into_wordsr%c s|dur|nj}|r|}|}t|r.jr.ts%tdn t }| |}j rNt s8t dtj |dddddd}tdd |}||fS|r`d ttfd d |}||fS) a Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. normalize (`bool`, *optional*, defaults to `None`): Whether or not to apply punctuation and casing normalization to the text inputs. Typically, VITS is trained on lower-cased and un-punctuated text. Hence, normalization is used to ensure that the input text consists only of lower-case characters. kwargs (`Dict[str, Any]`, *optional*): Keyword arguments to use for the tokenization. Returns: `Tuple[str, Dict[str, Any]]`: The prepared text and the unused kwargs. NaCText to the tokenizer contains non-Roman characters. To apply the `uroman` pre-processing step automatically, ensure the `uroman` Romanizer is installed with: `pip install uroman` Note `uroman` requires python version >= 3.10Otherwise, apply the Romanizer manually as per the instructions: https://github.com/isi-nlp/uromanzEPlease install the `phonemizer` Python package to use this tokenizer.zen-usZespeakT)r#backendstripZpreserve_punctuationZ with_stressz\s+ r=cs |jvSr4)r+)charr6rrs z8VitsTokenizer.prepare_for_tokenization..)r%rErHrr'r loggerwarningurZUromanZromanize_stringr&r ImportError phonemizerrsubjoinr?filterrK)r0rGrIr%r1rBuromanrr6rprepare_for_tokenizations6    z&VitsTokenizer.prepare_for_tokenizationcCs@t|}|jr|dgt|dd}||ddd<|}|S)z]Tokenize a string by inserting the `` token at the boundary between adjacent characters.rr>N)r?r$_convert_id_to_tokenr5)r0rGtokensZ interspersedrrr _tokenizes zVitsTokenizer._tokenizer[cCs*|jrt|dkr|ddd}d|S)Nr>rYr=)r$r5rU)r0r[rrrconvert_tokens_to_strings z&VitsTokenizer.convert_tokens_to_stringcCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)r+getr")r0tokenrrr_convert_token_to_idsz"VitsTokenizer._convert_token_to_idcCs |j|S)z=Converts an index (integer) in a token (str) using the vocab.)r-r^)r0indexrrrrZs z"VitsTokenizer._convert_id_to_tokensave_directoryfilename_prefixc Cstj|std|ddStj||r|dndtd}t|ddd}|t j |j d d d d d Wd|fS1sEwY|fS)NzVocabulary path (z) should be a directory-r=r wrrrYTF)indent sort_keys ensure_ascii ) ospathisdirrOerrorrUVOCAB_FILES_NAMESr(writer)dumpsr+)r0rbrcr frrrsave_vocabularys   zVitsTokenizer.save_vocabulary)rrNTTTF)rN)FNr4)__name__ __module__ __qualname____doc__rnZvocab_files_namesZmodel_input_namesr/propertyr7r<rErHstrboolrrrrrXrr\r]r`rZrrr __classcell__rrr2rr/sD "  A 0r)rvr)rjrtypingrrrrrrZtokenization_utilsr utilsr r r rSrWrQZ get_loggerrsrOrnrr__all__rrrrs"     H