o ZhD#@sdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl m Z ddlmZddlmZd d lmZeeZd d iZed dGddde ZdgZdS)z Tokenization class for SpeechT5.N)copyfile)AnyDictListOptionalTuple)PreTrainedTokenizer)logging)requires)EnglishNumberNormalizer vocab_filezspm_char.model) sentencepiece)backendsc s,eZdZdZeZddgZ      d/d eee e fd d ffd d Z d0ddZ e ddZe ddZejddZddZddZddZde d ee fddZddZd d!Zd"d#Zd1d eefd$d%Z d2d&eed'eeed(ed eeffd)d* Zd1d+e d,ee d ee fd-d.ZZS)3SpeechT5Tokenizera Construct a SpeechT5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. bos_token (`str`, *optional*, defaults to `""`): The begin of sequence token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. normalize (`bool`, *optional*, defaults to `False`): Whether to convert numeric quantities in the text to their spelt-out english counterparts. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Attributes: sp_model (`SentencePieceProcessor`): The *SentencePiece* processor that is used for every conversion (string, tokens and IDs). Z input_idsZattention_maskFNsp_model_kwargsreturnc sj|durin||_||_||_d|_tjdi|j|_|j|tj d||||||jd|dS)N) bos_token eos_token unk_token pad_token normalizer) rrr _normalizerspmSentencePieceProcessorsp_modelLoadsuper__init__) selfrrrrrrrkwargs __class__ra/var/www/auris/lib/python3.10/site-packages/transformers/models/speecht5/tokenization_speecht5.pyr$Qs    zSpeechT5Tokenizer.__init__cKs0|d|j}|r d|}|r||}||fS)Nr )popr normalizer)r%textZis_split_into_wordsr&rrrr)prepare_for_tokenizationns  z*SpeechT5Tokenizer.prepare_for_tokenizationcCs |jSN)r!Zget_piece_sizer%rrr) vocab_sizev zSpeechT5Tokenizer.vocab_sizecCs|jdur t|_|jSr/)rr r0rrr)r,zs zSpeechT5Tokenizer.normalizercCs ||_dSr/)r)r%valuerrr)r,r2cs(fddtjD}|j|S)Ncsi|]}||qSr)Zconvert_ids_to_tokens).0ir0rr) sz/SpeechT5Tokenizer.get_vocab..)ranger1updateZadded_tokens_encoder)r%Zvocabrr0r) get_vocabs zSpeechT5Tokenizer.get_vocabcCs|j}d|d<|S)Nr!)__dict__copy)r%staterrr) __getstate__s zSpeechT5Tokenizer.__getstate__cCs<||_t|ds i|_tjdi|j|_|j|jdS)Nrr)r:hasattrrrr r!r"r)r%drrr) __setstate__s  zSpeechT5Tokenizer.__setstate__r-cCs|jj|tdS)zPTake as input a string and return a list of strings (tokens) for words/sub-words)Zout_type)r!encodestr)r%r-rrr) _tokenizeszSpeechT5Tokenizer._tokenizecCs |j|S)z0Converts a token (str) in an id using the vocab.)r!Z piece_to_id)r%tokenrrr)_convert_token_to_ids z&SpeechT5Tokenizer._convert_token_to_idcCs|j|}|S)z=Converts an index (integer) in a token (str) using the vocab.)r!Z IdToPiece)r%indexrDrrr)_convert_id_to_tokens z&SpeechT5Tokenizer._convert_id_to_tokencCspg}d}d}|D]#}||jvr$|s|d7}||j||7}d}g}q||d}q||j|7}|S)z:Converts a sequence of tokens (string) in a single string.Fr*T)Zall_special_tokensr!decodeappendstrip)r%tokensZcurrent_sub_tokensZ out_stringZprev_is_specialrDrrr)convert_tokens_to_strings  z*SpeechT5Tokenizer.convert_tokens_to_stringcCs$|dur ||jgS|||jgS)z=Build model inputs from a sequence by appending eos_token_id.N)Z eos_token_id)r% token_ids_0 token_ids_1rrr) build_inputs_with_special_tokenss z2SpeechT5Tokenizer.build_inputs_with_special_tokensrNrOalready_has_special_tokenscsV|r tj||ddSdg}|durdgt||Sdgt|dgt||S)NT)rNrOrQr r)r#get_special_tokens_masklen)r%rNrOrQZ suffix_onesr'rr)rRs z)SpeechT5Tokenizer.get_special_tokens_masksave_directoryfilename_prefixcCstj|std|ddStj||r|dndtd}tj|jtj|kr?tj |jr?t |j||fStj |jsgt |d}|j }||Wd|fS1sbwY|fS)NzVocabulary path (z) should be a directory-rHrwb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathrisfileropenr!Zserialized_model_protowrite)r%rTrUZout_vocab_filefiZcontent_spiece_modelrrr)save_vocabularys" (     z!SpeechT5Tokenizer.save_vocabulary)rrrrFN)Fr/)NF) __name__ __module__ __qualname____doc__r^Zvocab_files_namesZmodel_input_namesrrrBrr$r.propertyr1r,setterr9r=r@rrCrErGrMintrPboolrRrrd __classcell__rrr'r)r"sR*       ( r)rhrXshutilrtypingrrrrrrrZtokenization_utilsr utilsr Zutils.import_utilsr Znumber_normalizerr Z get_loggerrer[r^r__all__rrrr)s       =