o Zh>@sdZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZddlmZddlmZerFdd lmZdd lmZmZdd lmZeeZd d iZdZ eddGdddeZ!dgZ"dS)z$Tokenization class for SigLIP model.N)copyfile) TYPE_CHECKINGAnyDictListOptionalTuple)import_protobuf)PreTrainedTokenizer) AddedToken) TextInput)loggingrequires_backends)requires vocab_filez spiece.modelu▁) sentencepiece)backendsc seZdZdZeZddgZ       d;d eee e fd dffd d Z d dZ e ddZddZ dd$d*d ee ffd+d, Ze d-d.Zd/d0Zd1d2Zd3d4Zd5d6Z d=d7e d8ee d e!e fd9d:Z"Z#S)?SiglipTokenizera Construct a Siglip tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. additional_special_tokens (`List[str]`, *optional*): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. model_max_length (`int`, *optional*, defaults to 64): The maximum length (in number of tokens) for model inputs. do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. Z input_idsZattention_maskN@Tsp_model_kwargsreturnc st|dt|trt|dddddn|}t|tr#t|dddddn|}t|tr3t|dddddn|}|dur;in||_||_||_||_||_t j d|||||j||d| dS)NprotobufTF)rstriplstrip normalizedZspecial) eos_token unk_token pad_tokenadditional_special_tokensrmodel_max_length do_lower_case) r isinstancestrr rr#rget_spm_processorsp_modelsuper__init__) selfrrrr r!rr"r#kwargs __class__r$]/var/www/auris/lib/python3.10/site-packages/transformers/models/siglip/tokenization_siglip.pyr*Zs:   zSiglipTokenizer.__init__cCstjdi|j}t|jd,}|}t}|j|}| }d|_ |j || }||Wd|S1s>wY|S)NrbFr$)spmSentencePieceProcessorropenrreadr Z ModelProtoZ FromStringZNormalizerSpecZadd_dummy_prefixnormalizer_specZ MergeFromZSerializeToStringZLoadFromSerializedProto)r+Z tokenizerfr(Z model_pb2modelr5r$r$r/r's      z!SiglipTokenizer.get_spm_processorcCs |jSN)r(Zget_piece_sizer+r$r$r/ vocab_sizes zSiglipTokenizer.vocab_sizecs(fddtjD}|j|S)Ncsi|]}||qSr$)Zconvert_ids_to_tokens).0ir9r$r/ sz-SiglipTokenizer.get_vocab..)ranger:updateZadded_tokens_encoder)r+Zvocabr$r9r/ get_vocabs zSiglipTokenizer.get_vocabF token_ids_0 token_ids_1already_has_special_tokenscsZ|r tj||ddS|durdgt|dgSdgt|dgdgt|dgS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rArBrCNr)r)get_special_tokens_masklen)r+rArBrCr-r$r/rEs(z'SiglipTokenizer.get_special_tokens_mask token_idscCs>t|dkr|d|jkrtd|jd|S||jgS)z.Do not add eos again if user already added it.rzThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)rF eos_token_idwarningswarnr)r+rGr$r$r/_add_eos_if_not_presents   z'SiglipTokenizer._add_eos_if_not_presentcCs<|jg}|durt||dgSt||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)rIrF)r+rArBZeosr$r$r/$create_token_type_ids_from_sequencessz4SiglipTokenizer.create_token_type_ids_from_sequencescCs(||}|dur |S||}||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)rL)r+rArBr$r$r/ build_inputs_with_special_tokenss  z0SiglipTokenizer.build_inputs_with_special_tokenscCs|j}d|d<|S)Nr()__dict__copy)r+stater$r$r/ __getstate__s zSiglipTokenizer.__getstate__cCs<||_t|ds i|_tjdi|j|_|j|jdS)Nrr$)rOhasattrrr1r2r(Loadr)r+dr$r$r/ __setstate__s  zSiglipTokenizer.__setstate__textcCs|tddtjS)N) translater& maketransstring punctuation)r+rWr$r$r/remove_punctuationsz"SiglipTokenizer.remove_punctuationkeep_punctuation_exact_stringcsH|r|fdd||D}n|}tdd|}|}|S)aReturns canonicalized `text` (puncuation removed). Args: text (`str`): String to be canonicalized. keep_punctuation_exact_string (`str`, *optional*): If provided, then this exact string is kept. For example providing '{}' will keep any occurrences of '{}' (but will still remove '{' and '}' that appear separately). c3s|]}|VqdSr8)r])r;partr9r$r/ s  z4SiglipTokenizer.canonicalize_text..z\s+ )joinsplitr]resubstrip)r+rWr_r$r9r/canonicalize_texts  z!SiglipTokenizer.canonicalize_textr c sVtjt|tdfi|}t|dkr)|dtkr)|d|jvr)|dd}|S)z8 Converts a string to a list of tokens. rbrDrN)r)tokenizeSPIECE_UNDERLINEreplacerFall_special_tokens)r+rWZadd_special_tokensr,tokensr-r$r/ri(s & zSiglipTokenizer.tokenizecCst|jt|jSr8)rFr(encoder&rr9r$r$r/unk_token_length2sz SiglipTokenizer.unk_token_lengthcKsT|j|dd}|jj|td}|jj|j|td}t||jkr(||jdS|S)u* Returns a tokenized string. We de-activated the `add_dummy_prefix` option, thus the sentencepiece internals will always strip any SPIECE_UNDERLINE. For example: `self.sp_model.encode(f"{SPIECE_UNDERLINE}Hey", out_type = str)` will give `['H', 'e', 'y']` instead of `['▁He', 'y']`. Thus we always encode `f"{unk_token}text"` and strip the `unk_token`. Here is an example with `unk_token = ""` and `unk_token_length = 4`. `self.tokenizer.sp_model.encode(" Hey", out_type = str)[4:]`. Nr^)Zout_type)rhr(rnr&rrFro)r+rWr,rmr$r$r/ _tokenize7s  zSiglipTokenizer._tokenizecCs |j|S)z0Converts a token (str) in an id using the vocab.)r(Z piece_to_id)r+tokenr$r$r/_convert_token_to_idLs z$SiglipTokenizer._convert_token_to_idcCs|j|}|S)z=Converts an index (integer) in a token (str) using the vocab.)r(Z IdToPiece)r+indexrqr$r$r/_convert_id_to_tokenQs z$SiglipTokenizer._convert_id_to_tokencCspg}d}d}|D]#}||jvr$|s|d7}||j||7}d}g}q||d}q||j|7}|S)z:Converts a sequence of tokens (string) in a single string.rXFrbT)rlr(decodeappendrg)r+rmZcurrent_sub_tokensZ out_stringZprev_is_specialrqr$r$r/convert_tokens_to_stringVs  z(SiglipTokenizer.convert_tokens_to_stringsave_directoryfilename_prefixcCstj|std|ddStj||r|dndtd}tj|jtj|kr?tj |jr?t |j||fStj |jsgt |d}|j }||Wd|fS1sbwY|fS)NzVocabulary path (z) should be a directory-rXrwb)ospathisdirloggererrorrcVOCAB_FILES_NAMESabspathrisfilerr3r(Zserialized_model_protowrite)r+rxryZout_vocab_filefiZcontent_spiece_modelr$r$r/save_vocabularyjs" (     zSiglipTokenizer.save_vocabulary)rrrNNrT)NFr8)F)$__name__ __module__ __qualname____doc__rZvocab_files_namesZmodel_input_namesrrr&rr*r'propertyr:r@rintboolrErLrMrNrRrVr]rhrirorprrrtrwrr __classcell__r$r$r-r/r-sv( 1         (r)#rr|rer[rJshutilrtypingrrrrrrrr1Zconvert_slow_tokenizerr Ztokenization_utilsr Ztokenization_utils_baser r utilsrrZutils.import_utilsrZ get_loggerrrrrjr__all__r$r$r$r/s.         P