fTh7SSKrSSKJr SSKJrJrJrJrJrJ r SSK r SSK J r Jr SSKJr SSKJr \(aSSKJr \R*"\5rS S 0rS r\"S S 9"SS\55rS/rg)N)copyfile) TYPE_CHECKINGAnyDictListOptionalTuple) AddedTokenPreTrainedTokenizer)logging)requires) TextInput vocab_fileztokenizer.modelu▁) sentencepiece)backendsc ^\rSrSrSr\rSS/rS S\\ \ \ 44U4Sjjjr Sr Sr\S 5rS rS S S \\ 4U4SjjrSrSrSrSrS!S\\ S \\ 4SjjrS!SjrS"S\\S\\\S\S \\4U4SjjjrS!S\\S\\\S \\4SjjrS#S\\S\S\S \ 4SjjrSr U=r!$)$GemmaTokenizer+aa Construct a Gemma tokenizer. Based on byte-level Byte-Pair-Encoding. The default padding token is unset as there is no padding token in the original model. Args: vocab_file (`str`): Path to the vocabulary file. unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. sp_model_kwargs (`Dict[str, Any]`, `Optional`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. add_bos_token (`bool`, *optional*, defaults to `True`): Whether or not to add an `bos_token` at the start of sequences. add_eos_token (`bool`, *optional*, defaults to `False`): Whether or not to add an `eos_token` at the end of sequences. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like extra spaces. use_default_system_prompt (`bool`, *optional*, defaults to `False`): Whether or not the default system prompt for Gemma should be used. spaces_between_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to add spaces between special tokens. input_idsattention_masksp_model_kwargsc  >Uc0OUUl[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUnXlXplXlXl[R"S0URD6Ul URRU5 [T U]4"SUUUUUUUU U U S. U D6 g)NFT) normalizedspecial) bos_token eos_token unk_token pad_token add_bos_token add_eos_tokenrclean_up_tokenization_spacesuse_default_system_promptspaces_between_special_tokens)r isinstancestrr rr r!r#spmSentencePieceProcessorsp_modelLoadsuper__init__)selfrrrrrrr r!r"r#r$kwargs __class__s d/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/gemma/tokenization_gemma.pyr-GemmaTokenizer.__init__^s&5%[TU]"U40UD6$)zE Args: text: TextInput Simply calls PreTrainedTokenizer's method )r,tokenize)r.rNr/r0s r1rQGemmaTokenizer.tokenizes w///r3c >URRU[S9$)zf Args: text: TextInput Returns a tokenized string. The Gemma tokenizer never adds a prefix space. )out_type)r*encoder')r.rNr/s r1 _tokenizeGemmaTokenizer._tokenizes }}##D3#77r3c8URRU5$)z0Converts a token (str) in an id using the vocab.)r* piece_to_id)r.tokens r1_convert_token_to_id#GemmaTokenizer._convert_token_to_ids}}((//r3c<URRU5nU$)z=Converts an index (integer) in a token (str) using the vocab.)r* IdToPiece)r.indexrZs r1_convert_id_to_token#GemmaTokenizer._convert_id_to_tokens ''. r3c/nSnUHGnX@R;a$X0RRU5U-- n/nM6URU5 MI X0RRU5- nU$)z:Converts a sequence of tokens (string) in a single string.)_added_tokens_encoderr*decodeappend)r.tokenscurrent_sub_tokens out_stringrZs r1convert_tokens_to_string'GemmaTokenizer.convert_tokens_to_stringst E222mm223EFNN %'""))%0  mm**+=>> r3filename_prefixc[RRU5(d[R SUS35 g[RR X(aUS-OS[ S-5n[RRUR5[RRU5:waG[RRUR5(a[URU5 U4$[RRUR5(dC[US5nURR5nURU5 SSS5 U4$U4$!,(df  U4$=f)z Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. zVocabulary path (z) should be a directoryN-rcrwb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathrisfileropenr*r8write)r.save_directoryrlout_vocab_fileficontent_spiece_models r1save_vocabularyGemmaTokenizer.save_vocabularys.ww}}^,, LL,^,<U(a[TU]XSS9$UR(aS/O/nUR(aS/O/nUcUS/[ U5--U-$US/[ U5--U-U-S/[ U5--U-$)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrr)r,get_special_tokens_maskr r!len)r.rrrrrr0s r1r&GemmaTokenizer.get_special_tokens_masks$ &72']a3 #00sb "00sb  A3[)9#9:\I I sS%% '  sS%%  '    r3cUR(a UR/O/nUR(a UR/O/nS/[ X1-U-5-nUbUS/[ X2-U-5-- nU$)aM Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). rr)r rr!rrrs r1$create_token_type_ids_from_sequences3GemmaTokenizer.create_token_type_ids_from_sequencessv./3.@.@))*b .2.@.@))*b s<5 DEE  " qcC :\ IJJ JF r3 token_idsskip_special_tokensr$c (/n/nUHnU(aXpR;aMXpR;a]U(a*URURR U55 URURUR 5 /nMURU5 M U(a*URURR U55 U(aSR U5nOSR U5nUR[S5$)N rc) all_special_ids_added_tokens_decoderrfr*recontentrureplaceSPIECE_UNDERLINE)r.rrr$r/ sub_textscurrent_sub_textidss r1_decodeGemmaTokenizer._decode1s C"s.B.B'B000#$$T]]%9%9:J%KL  !;!;C!@!H!HI#%  '',    T]]112BC D (+I *I  !1377r3)r r!r*rr#r) zzzzNTFFFFr)NF)FF)"__name__ __module__ __qualname____firstlineno____doc__rvvocab_files_namesmodel_input_namesrrr'rr-r:r@propertyrDrLrrQrVr[r`rjr rrintboolrrr__static_attributes__ __classcell__)r0s@r1rr+s,\*$&67 48%*"'&+( "$sCx.1( ( T C .. 0[0tCy080 !x}!X]^aXb!6 sx# 9# 3;DI3F# ko# c# # LJN93;DI3F cH%*.3 898"8(, 8 88r3r)rpshutilrtypingrrrrrr rr(tokenization_utilsr r utilsr utils.import_utilsrtokenization_utils_baser get_loggerrrsrvrr__all__r%r3r1rsy, BBA*4   H %!#45 %&`8(`8'`8F  r3