fThD# S r SSKrSSKJr SSKJrJrJrJrJ r SSK
r SSK J
r
SSKJr SSKJr S S
KJr \R( " \5 rS S 0r\" S
S9 " S S\
5 5 rS/rg)z Tokenization class for SpeechT5. N)copyfile)AnyDictListOptionalTuple )PreTrainedTokenizer)logging)requires )EnglishNumberNormalizer
vocab_filezspm_char.model)
sentencepiece)backendsc
j ^ \ rS rSrSr\rSS/r S S\\ \
\ 4 SS4U 4S jjjr S S jr
\S
5 r\S 5 r\R" S 5 rS
rS rS rS\
S\\
4S jrS rS rS rS S\\ 4S jjr S!S\\ S\\\ S\ S\\ 4U 4S jjjr S S\
S\\
S\ \
4S jjr S r U =r!$ )"SpeechT5Tokenizer" a
Construct a SpeechT5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
bos_token (`str`, *optional*, defaults to `""`):
The begin of sequence token.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
normalize (`bool`, *optional*, defaults to `False`):
Whether to convert numeric quantities in the text to their spelt-out english counterparts.
sp_model_kwargs (`dict`, *optional*):
Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
to set:
- `enable_sampling`: Enable subword regularization.
- `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
- `nbest_size = {0,1}`: No sampling is performed.
- `nbest_size > 1`: samples from the nbest_size results.
- `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
- `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
Attributes:
sp_model (`SentencePieceProcessor`):
The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
input_idsattention_maskNsp_model_kwargsreturnc
> Uc 0 OUU l Xl X`l S U l [ R
" S0 U R D6U l U R R U5 [ T U ]$ " SUUUUUU R S.UD6 g )N) bos_token eos_token unk_token pad_token normalizer )
r r r _normalizerspmSentencePieceProcessorsp_modelLoadsuper__init__)
selfr r r r r r r kwargs __class__s
j/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/speecht5/tokenization_speecht5.pyr& SpeechT5Tokenizer.__init__Q s &5% U R R U[ S9$ )zPTake as input a string and return a list of strings (tokens) for words/sub-words)out_type)r# encodestr)r' r1 s r* _tokenizeSpeechT5Tokenizer._tokenize s }}##D3#77r, c 8 U R R U5 $ )z0Converts a token (str) in an id using the vocab.)r# piece_to_id)r' tokens r* _convert_token_to_id&SpeechT5Tokenizer._convert_token_to_id s }}((//r, c < U R R U5 nU$ )z=Converts an index (integer) in a token (str) using the vocab.)r# IdToPiece)r' indexr[ s r* _convert_id_to_token&SpeechT5Tokenizer._convert_id_to_token s
''. r, c " / nSnSnU HW nXPR ; a2 U( d US-
nX0R R U5 U- -
nSn/ nMD UR U5 SnMY X0R R U5 -
nUR 5 $ )z:Converts a sequence of tokens (string) in a single string. Fr. T)all_special_tokensr# decodeappendstrip)r' tokenscurrent_sub_tokens
out_stringprev_is_specialr[ s r* convert_tokens_to_string*SpeechT5Tokenizer.convert_tokens_to_string s
E///& #%Jmm 2 23E F NN
"&%'""))%0"' mm**+=>>
!!r, c J Uc XR /- $ X- U R /- $ )z=Build model inputs from a sequence by appending eos_token_id.) eos_token_id)r' token_ids_0 token_ids_1s r* build_inputs_with_special_tokens2SpeechT5Tokenizer.build_inputs_with_special_tokens s1 "3"3!44 4(D,=,=+>>>r, rq rr already_has_special_tokensc > U( a [ TU ] XSS9$ S/nUc S/[ U5 - U- $ S/[ U5 - S/[ U5 - - U- $ )NT)rq rr ru r
r )r% get_special_tokens_masklen)r' rq rr ru suffix_onesr) s r* rw )SpeechT5Tokenizer.get_special_tokens_mask ss &72 ']a 3
c C#k**k9 9c+&&A3[1A+AB[PPr, save_directoryfilename_prefixc [ R R U5 ( d [ R SU S35 g [ R R X( a US- OS[ S - 5 n[ R R U R 5 [ R R U5 :w aG [ R R U R 5 ( a [ U R U5 U4$ [ R R U R 5 ( dC [ US5 nU R R 5 nUR U5 S S S 5 U4$ U4$ ! , ( d f U4$ = f)NzVocabulary path (z) should be a directory-rd r wb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr isfiler openr# serialized_model_protowrite)r' r{ r| out_vocab_fileficontent_spiece_models r* save_vocabulary!SpeechT5Tokenizer.save_vocabulary s, ww}}^,, LL,^,<zzzFN)Fr6 )NF)"__name__
__module__ __qualname____firstlineno____doc__r vocab_files_namesmodel_input_namesr r rV r r& r3 propertyr9 r0 setterrF rL rQ r rW r\ ra rm intrs boolrw r r __static_attributes__
__classcell__)r) s @r* r r " s] (T *$&67
48
"$sCx.1
: . .
! !
,8c 8d3i 80 "&?QUVYQZ ? sx Q 9 Q3;DI3F Qko Q
c Q Q!c !HSM !]bcf]g ! !r, r )r r shutilr typingr r r r r r r! tokenization_utilsr
utilsr utils.import_utilsr number_normalizerr
get_loggerr r r r __all__r r, r* r ss ' 3 3 5 * 6
H %!#34
%&y!+ y! 'y!x
r,