o Zh~P@sdZddlZddlZddlZddlmZddlmZmZm Z ddl Z ddl m Z mZmZmZmZddlmZeeZdd d Zed d Zd dZddZddZGdddZGdddeZdgZdS)zTokenization classes for CLIP.N) lru_cache)ListOptionalTuple) AddedTokenPreTrainedTokenizer _is_control_is_punctuation_is_whitespace)loggingz vocab.jsonz merges.txt) vocab_file merges_filecCstttdtddtttdtddtttdtdd}|dd}d }td D]}||vrI|||d ||d7}q3d d |D}tt||S) a8 Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. !~¡¬®ÿNrcSsg|]}t|qS)chr).0nrrY/var/www/auris/lib/python3.10/site-packages/transformers/models/clip/tokenization_clip.py :sz$bytes_to_unicode..)listrangeordappenddictzip)bscsrbrrrbytes_to_unicode%sL    r&cCs6t}|d}|ddD] }|||f|}q |S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrN)setadd)wordpairsZ prev_charcharrrr get_pairs>s r,cCstdd|}|}|S)Nz\s+ )resubstrip)textrrrwhitespace_cleanLsr2cCs|}|sgS|}|S)z@Runs basic whitespace cleaning and splitting on a piece of text.)r0split)r1tokensrrrwhitespace_tokenizeSs r5c@sXeZdZdZ     dddZdddZdd Zdd d Zd d ZddZ ddZ dS)BasicTokenizera Constructs a BasicTokenizer that will run basic tokenization (punctuation splitting, lower casing, etc.). Args: do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). do_split_on_punc (`bool`, *optional*, defaults to `True`): In some instances we want to skip the basic punctuation splitting so that later tokenization can capture the full context of the words, such as contractions. TNcCs2|durg}||_t||_||_||_||_dSN) do_lower_caser' never_splittokenize_chinese_chars strip_accentsdo_split_on_punc)selfr8r9r:r;r<rrr__init__ts  zBasicTokenizer.__init__cCs|r |jt|n|j}||}|jr||}td|}t|}g}|D])}||vrH|j r@| }|j dur?| |}n|j rH| |}| |||q(td|}|S)aj Basic Tokenization of a piece of text. For sub-word tokenization, see WordPieceTokenizer. Args: never_split (`List[str]`, *optional*) Kept for backward compatibility purposes. Now implemented directly at the base class level (see [`PreTrainedTokenizer.tokenize`]) List of token not to split. NFCFr-)r9unionr' _clean_textr:_tokenize_chinese_chars unicodedata normalizer5r8lowerr;_run_strip_accentsextend_run_split_on_puncjoin)r=r1r9Zunicode_normalized_textZ orig_tokensZ split_tokenstokenZ output_tokensrrrtokenizes&      zBasicTokenizer.tokenizecCsBtd|}g}|D]}t|}|dkrq ||q d|S)z$Strips accents from a piece of text.ZNFDZMn)rCrDcategoryr rI)r=r1outputr+catrrrrFs    z!BasicTokenizer._run_strip_accentscCs|jr |dur||vr|gSt|}d}d}g}|t|krI||}t|r/||gd}n|r6|gd}|d||d7}|t|ksdd|DS) z&Splits punctuation on a piece of text.NrTFrcSsg|]}d|qS)rL)rI)rxrrrrz5BasicTokenizer._run_split_on_punc..)r<rlenr r )r=r1r9charsiZstart_new_wordrNr+rrrrHs$     z!BasicTokenizer._run_split_on_punccCsTg}|D] }t|}||r|d|||dq||qd|S)z)Adds whitespace around any CJK character.r-rL)r_is_chinese_charr rIr=r1rNr+cprrrrBs      z&BasicTokenizer._tokenize_chinese_charscCs|dkr|dks@|dkr|dks@|dkr|dks@|dkr |dks@|d kr(|d ks@|d kr0|d ks@|d kr8|dks@|dkrB|dkrBdSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr)r=rXrrrrVs zBasicTokenizer._is_chinese_charcCsXg}|D]"}t|}|dks|dkst|rqt|r!|dq||qd|S)zBPerforms invalid character removal and whitespace cleanup on text.rir-rL)rr r r rIrWrrrrAs   zBasicTokenizer._clean_text)TNTNTr7) __name__ __module__ __qualname____doc__r>rKrFrHrBrVrArrrrr6]s &  r6c seZdZdZeZddgZ     d'fdd Zed d Z d d Z d(de e de e e de e fddZ d)de e de e e dede e ffdd Z d(de e de e e de e fddZddZddZddZdd Zd!d"Zd(d#ed$e edeefd%d&ZZS)* CLIPTokenizera Construct a CLIP tokenizer. Based on byte-level Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. See [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `"<|startoftext|>"`): The beginning of sequence token. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding, for example when batching sequences of different lengths. Z input_idsZattention_maskreplace <|endoftext|><|startoftext|>c  st|tr t|dddn|}t|trt|dddn|}t|tr(t|dddn|}z ddl} | j|_WntyLtdtddd|_ d|_Ynwt |dd} t | |_ Wdn1sdwYdd |j D|_||_t|_d d |jD|_t |dd} | d d d } Wdn1swYdd| D} tt| tt| |_ddd|_tdtj|_ t!j"d|||||d|dS)NF)lstriprstriprzKftfy or spacy is not installed using custom BasicTokenizer instead of ftfy.)r;r<utf-8encodingcSi|]\}}||qSrrrkvrrr 4rRz*CLIPTokenizer.__init__..cSrfrrrgrrrrj7rR ricSsg|]}t|qSr)tupler3)rmergerrrr:sz*CLIPTokenizer.__init__..r`r_)r`r_z[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)errors unk_token bos_token eos_token pad_tokenr)# isinstancestrrftfyfix_text ImportErrorloggerinfor6nlpopenjsonloadencoderitemsdecoderrnr& byte_encoder byte_decoderreadr0r3r!r"rrS bpe_rankscacher.compile IGNORECASEpatsuperr>) r=r rrnrorprqrrkwargsruZ vocab_handleZ merges_handleZ bpe_merges __class__rrr>sJ       zCLIPTokenizer.__init__cCs t|jSr7)rSr~r=rrr vocab_sizeLs zCLIPTokenizer.vocab_sizecCst|jfi|jSr7)r!r~Zadded_tokens_encoderrrrr get_vocabPszCLIPTokenizer.get_vocabN token_ids_0 token_ids_1returncCs<|jg}|jg}|dur|||S||||||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CLIP sequence has the following format: - single sequence: `<|startoftext|> X <|endoftext|>` Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N) bos_token_id eos_token_idr=rrrprqrrr build_inputs_with_special_tokensSs  z.CLIPTokenizer.build_inputs_with_special_tokensFalready_has_special_tokenscsl|r tj||ddS|durdgdgt|dgSdgdgt|dgdgdgt|dgS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNrr)rget_special_tokens_maskrS)r=rrrrrrrns4z%CLIPTokenizer.get_special_tokens_maskcCsP|jg}|jg}|durt|||dgSt||||||dgS)a Create a mask from the two sequences passed. CLIP does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)rrrSrrrr$create_token_type_ids_from_sequencess "z2CLIPTokenizer.create_token_type_ids_from_sequencesc sr|jvr j|St|dd|ddf}t|}|s#|dS t|fddd}|jvr4ny|\}}g}d}|t|krz|||} Wnty\|||dYn?w|||| | }|||kr|t|dkr||d|kr| |||d 7}n | |||d7}|t|ksBt|}|}t|dkrnt|}q$d |}|j|<|S) NrPTcsj|tdS)Ninf)rgetfloat)pairrrrsz#CLIPTokenizer.bpe..keyrrr-) rrlr,minrrSindex ValueErrorrGr rI) r=rJr)r*ZbigramfirstsecondZnew_wordrUjrrrbpesJ     ,     zCLIPTokenizer.bpecsg}jdurdj|}n t|}tj|D]!}dfdd| dD}| dd | dDq!|S)zTokenize a string.Nr-rLc3s|]}j|VqdSr7)r)rr%rrr s  z*CLIPTokenizer._tokenize..rccss|]}|VqdSr7r)rZ bpe_tokenrrrrs) rvrIrzrKr2rEr.findallrencoderGrr3)r=r1 bpe_tokensrJrrr _tokenizes "zCLIPTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)r~rro)r=rJrrr_convert_token_to_idsz"CLIPTokenizer._convert_token_to_idcCs |j|S)z=Converts an index (integer) in a token (str) using the vocab.)rr)r=rrrr_convert_id_to_tokens z"CLIPTokenizer._convert_id_to_tokencs@d|}tfdd|D}|jdjddd}|S)z:Converts a sequence of tokens (string) in a single string.rLcsg|]}j|qSr)r)rcrrrrrRz:CLIPTokenizer.convert_tokens_to_string..rc)rnrr-)rI bytearraydecodernr^r0)r=r4r1Z byte_arrayrrrconvert_tokens_to_strings z&CLIPTokenizer.convert_tokens_to_stringsave_directoryfilename_prefixc CsRtj|std|dStj||r|dndtd}tj||r+|dndtd}t|ddd}| t j |j d d d d d Wdn1sSwYd}t|ddd<}| dt |jdddD] \}} || krtd|| }| d|d |d7}qrWd||fS1swY||fS)Nz*Vocabulary path ({}) should be a directory-rLr rwrcrdrTF)indent sort_keys ensure_asciirkrz#version: 0.2 cSs|dS)Nrr)kvrrrrsz/CLIPTokenizer.save_vocabulary..rzqSaving vocabulary to {}: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!r-r)ospathisdirrxerrorformatrIVOCAB_FILES_NAMESr{writer|dumpsr~sortedrrwarning) r=rrr Z merge_filefrwriterrZ token_indexrrrsave_vocabularys:       zCLIPTokenizer.save_vocabulary)r^r_r`r_r_r7)NF)rYrZr[r\rZvocab_files_namesZmodel_input_namesr>propertyrrrintrrboolrrrrrrrrtrr __classcell__rrrrr]sZ1      *(r])r\r|rrC functoolsrtypingrrrregexr.Ztokenization_utilsrrr r r utilsr Z get_loggerrYrxrr&r,r2r5r6r]__all__rrrrs0     "