fTh>SrSSKrSSKrSSKrSSKrSSKJr SSKJrJ r J r J r J r J r SSKrSSKJr SSKJr SSKJr \(aSS KJr SS KJrJr SS KJr \R8"\5rS S 0rSr \"SS9"SS\55r!S/r"g)z$Tokenization class for SigLIP model.N)copyfile) TYPE_CHECKINGAnyDictListOptionalTuple)import_protobuf)PreTrainedTokenizer) AddedToken) TextInput)loggingrequires_backends)requires vocab_filez spiece.modelu▁) sentencepiece)backendsc ^\rSrSrSr\rSS/rS%S\\ \ \ 4SS4U4Sjjjr S r \S 5rS rS&S \\S \\\S\S\\4U4SjjjrS\\S\\4SjrS'S \\S \\\S\\4SjjrS'S \\S \\\S\\4SjjrSrSrS\ S\ 4SjrSS.SjrS(SSS\\ 4U4Sjjjr\S5rSrSrSr S r!S'S!\ S"\\ S\"\ 4S#jjr#S$r$U=r%$))SiglipTokenizer-ad Construct a Siglip tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. additional_special_tokens (`List[str]`, *optional*): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. model_max_length (`int`, *optional*, defaults to 64): The maximum length (in number of tokens) for model inputs. do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. input_idsattention_maskNsp_model_kwargsreturnc >[US5 [U[5(a [USSSSS9OUn[U[5(a [USSSSS9OUn[U[5(a [USSSSS9OUnUc0OUUlXlXlUR5UlXl[T U]("SUUUUURUUS.U D6 g)NprotobufTF)rstriplstrip normalizedspecial) eos_token unk_token pad_tokenadditional_special_tokensrmodel_max_length do_lower_case) r isinstancestrr rr'rget_spm_processorsp_modelsuper__init__) selfrr"r#r$r%rr&r'kwargs __class__s f/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/siglip/tokenization_siglip.pyr.SiglipTokenizer.__init__Zs $ +)S)) yduVZ [ )S)) yduVZ [ )S)) yduVZ [  &5%U(a[TU]XSS9$UcS/[U5-S/-$S/[U5-S/-S/[U5--S/-$)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rWrXrYr)r-get_special_tokens_masklen)r/rWrXrYr1s r2r\'SiglipTokenizer.get_special_tokens_masksw$ &72']a3   C#k**qc1 1c+&&1#-!s;7G1GHA3NNr4 token_idsc[U5S:a9USUR:Xa&[R"SURS35 U$XR/-$)z.Do not add eos again if user already added it.rzThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)r] eos_token_idwarningswarnr")r/r_s r2_add_eos_if_not_present'SiglipTokenizer._add_eos_if_not_presents[ y>A )B-43D3D"D MM,T^^,<=++   1 122 2r4crUR/nUc[X-5S/-$[X-U-U-5S/-$)aw Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. r)rbr])r/rWrXeoss r2$create_token_type_ids_from_sequences4SiglipTokenizer.create_token_type_ids_from_sequencessL   !  {()QC/ /;${2S89QC??r4cXURU5nUcU$URU5nX-$)a" Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )re)r/rWrXs r2 build_inputs_with_special_tokens0SiglipTokenizer.build_inputs_with_special_tokenss9&22;?   66{CK, ,r4cDURR5nSUS'U$)Nr,)__dict__copy)r/states r2 __getstate__SiglipTokenizer.__getstate__s# ""$ j r4cXl[US5(d0Ul[R"S0URD6UlUR R UR5 g)Nrr()rohasattrrr7r8r,Loadr)r/ds r2 __setstate__SiglipTokenizer.__setstate__sP t.//#%D 22JT5I5IJ  4??+r4textcjUR[RSS[R55$)N) translater* maketransstring punctuation)r/rzs r2remove_punctuation"SiglipTokenizer.remove_punctuations$~~cmmBF4F4FGHHr4keep_punctuation_exact_stringc^U(a+URU4SjURU555nOTRU5n[R"SSU5nUR 5nU$)a]Returns canonicalized `text` (puncuation removed). Args: text (`str`): String to be canonicalized. keep_punctuation_exact_string (`str`, *optional*): If provided, then this exact string is kept. For example providing '{}' will keep any occurrences of '{}' (but will still remove '{' and '}' that appear separately). c3F># UHnTRU5v M g7frI)r).0partr/s r2 4SiglipTokenizer.canonicalize_text..s#6:c$''--:cs!z\s+ )joinsplitrresubstrip)r/rzrs` r2canonicalize_text!SiglipTokenizer.canonicalize_textsc )0556:>**Eb:c6D**40Dvvfc4(zz| r4rc >[TU]"[UR[S5-40UD6n[ U5S:a%US[:XaUSUR ;aUSSnU$)z( Converts a string to a list of tokens. rr[rN)r-tokenizeSPIECE_UNDERLINEreplacer]all_special_tokens)r/rzadd_special_tokensr0tokensr1s r2rSiglipTokenizer.tokenize(se!"2T\\BRTW5X"Xc\bc v;?vay,<<dNeNeAeABZF r4cp[URR[UR555$rI)r]r,encoder*r#rKs r2unk_token_length SiglipTokenizer.unk_token_length2s'4==''DNN(;<==r4c URUSS9nURRU[S9nURRURU-[S9n[ U5UR :aX0R S$U$)u Returns a tokenized string. We de-activated the `add_dummy_prefix` option, thus the sentencepiece internals will always strip any SPIECE_UNDERLINE. For example: `self.sp_model.encode(f"{SPIECE_UNDERLINE}Hey", out_type = str)` will give `['H', 'e', 'y']` instead of `['▁He', 'y']`. Thus we always encode `f"{unk_token}text"` and strip the `unk_token`. Here is an example with `unk_token = ""` and `unk_token_length = 4`. `self.tokenizer.sp_model.encode(" Hey", out_type = str)[4:]`. Nr)out_type)rr,rr*r#r]r)r/rzr0rs r2 _tokenizeSiglipTokenizer._tokenize7s%%d$%O%%dS%9%%dnnt&;c%J25f+AVAV2Vv++-.b\bbr4c8URRU5$)z0Converts a token (str) in an id using the vocab.)r, piece_to_id)r/tokens r2_convert_token_to_id$SiglipTokenizer._convert_token_to_idLs}}((//r4c<URRU5nU$)z=Converts an index (integer) in a token (str) using the vocab.)r, IdToPiece)r/indexrs r2_convert_id_to_token$SiglipTokenizer._convert_id_to_tokenQs ''. r4c"/nSnSnUHWnXPR;a2U(dUS- nX0RRU5U-- nSn/nMDURU5 SnMY X0RRU5- nUR 5$)z:Converts a sequence of tokens (string) in a single string.r|FrT)rr,decodeappendr)r/rcurrent_sub_tokens out_stringprev_is_specialrs r2convert_tokens_to_string(SiglipTokenizer.convert_tokens_to_stringVs E///&#%Jmm223EFNN "&%'""))%0"' mm**+=>> !!r4save_directoryfilename_prefixc[RRU5(d[R SUS35 g[RR X(aUS-OS[ S-5n[RRUR5[RRU5:waG[RRUR5(a[URU5 U4$[RRUR5(dC[US5nURR5nURU5 SSS5 U4$U4$!,(df  U4$=f)NzVocabulary path (z) should be a directory-r|rwb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathrisfilerr9r,serialized_model_protowrite)r/rrout_vocab_fileficontent_spiece_models r2save_vocabularySiglipTokenizer.save_vocabularyjs,ww}}^,, LL,^,<zrNN@T)NFrI)F)&__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesrrr*rr.r+propertyrLrUrintboolr\rerirlrrrxrrrrrrrrr r__static_attributes__ __classcell__)r1s@r2rr-s&P*$&67 "&48/ "$sCx.1/  / / b ..sxO9O3;DI3FOkoO cOO: 3c 3tCy 3JN@9@3;DI3F@ c@0JN-9-3;DI3F- c-6 ,IsIsIHL*[QUVYQZ>>c*0  "(!c!HSM!]bcf]g!!r4r)#rrrrrcshutilrtypingrrrrrr rr7convert_slow_tokenizerr tokenization_utilsr tokenization_utils_baser rutilsrrutils.import_utilsr get_loggerrrrrr__all__r(r4r2rs+ BB5514/*   H %!>2 %&K!)K!'K!\  r4