fTh~PSrSSKrSSKrSSKrSSKJr SSKJrJrJ r SSK r SSK J r JrJrJrJr SSKJr \R("\5rSS S .r\"5S 5rS rS rSr"SS5r"SS\5rS/rg)zTokenization classes for CLIP.N) lru_cache)ListOptionalTuple) AddedTokenPreTrainedTokenizer _is_control_is_punctuation_is_whitespace)loggingz vocab.jsonz merges.txt) vocab_file merges_filec [[[S5[S5S-55[[[S5[S5S-55-[[[S5[S5S-55-nUSSnS n[S 5H4nX0;dM URU5 URS U-5 US- nM6 UVs/sHn[ U5PM nn[ [ X55$s snf) a Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. !~¡¬®ÿNr)listrangeordappendchrdictzip)bscsnbs b/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/clip/tokenization_clip.pybytes_to_unicoder%%s U3s8SX\ *+d5TCIPQM3R.SSVZ[`adeiajloptluxyly[zV{{ AB A 4[ ; IIaL IIdQh  FA   "Q#a&"B  B  sC:cd[5nUSnUSSHnURX#45 UnM U$)zy Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrN)setadd)wordpairs prev_charchars r$ get_pairsr->s? EEQIQR 9#$  LcV[R"SSU5nUR5nU$)Nz\s+ )resubstrip)texts r$whitespace_cleanr5Ls$ 66&#t $D ::SWScSc %  & &//5D"-"7"7t"D)*AB   E'%%!KKME))6 $ 7 7 >'' 33E:E    7 7 K L!,CHH\,BC r.c[R"SU5n/nUH2n[R"U5nUS:XaM!URU5 M4 SR U5$)z$Strips accents from a piece of text.NFDMn)rLrMcategoryrrR)rDr4outputr,cats r$rO!BasicTokenizer._run_strip_accentssY$$UD1D&&t,Cd{ MM$   wwvr.cUR(aUbX;aU/$[U5nSnSn/nU[U5:amX4n[U5(aUR U/5 SnO.U(aUR /5 SnUSR U5 US- nU[U5:aMmUVs/sHnSR U5PM sn$s snf)z&Splits punctuation on a piece of text.rTFrr])rCrlenr rrR) rDr4r@charsistart_new_wordr_r,xs r$rQ!BasicTokenizer._run_split_on_puncs$$)@TEX6MT  #e*n8Dt$$ tf%!%!MM"%!&r !!$' FA#e*n%++Fq F+++s.C c/nUHjn[U5nURU5(a5URS5 URU5 URS5 MYURU5 Ml SRU5$)z)Adds whitespace around any CJK character.r0r])r_is_chinese_charrrRrDr4r_r,cps r$rK&BasicTokenizer._tokenize_chinese_charsskDTB$$R(( c" d# c" d#wwvr.cUS:aUS::dTUS:aUS::dHUS:aUS::d<US:aUS::d0US :aUS ::d$US :aUS ::dUS :aUS::d US:aUS::agg)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTF)rDrms r$rkBasicTokenizer._is_chinese_charsr6\bFlf vg "-g "-g "-g "-f vg "-r.c/nUH`n[U5nUS:XdUS:Xd[U5(aM,[U5(aURS5 MOURU5 Mb SR U5$)zBPerforms invalid character removal and whitespace cleanup on text.rir0r])rr r rrRrls r$rJBasicTokenizer._clean_textsgDTBQw",+d*;*;d## c" d#wwvr.)r?rCr@rBrA)TNTNTr>) __name__ __module__ __qualname____firstlineno____doc__rErXrOrQrKrkrJ__static_attributes__rpr.r$r;r;]s<0# 1 $L ,, 0 r.r;c F^\rSrSrSr\rSS/rSU4Sjjr\ S5r Sr SS\ \ S \\ \ S \ \ 4S jjrSS\ \ S \\ \ S \S \ \ 4U4S jjjrSS\ \ S \\ \ S \ \ 4SjjrSrSrSrSrSrSS\S\\S \\4SjjrSrU=r$) CLIPTokenizera Construct a CLIP tokenizer. Based on byte-level Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. See [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `"<|startoftext|>"`): The beginning of sequence token. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc >[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUnSSKn U RUl[USS9n [R"U 5Ul SSS5 URR5V V s0sHupX_M sn n UlX0l[#5UlUR$R5V V s0sHupX_M sn n Ul[USS9n U R)5R+5R-S5S S nSSS5 WVs/sHn[/UR-55PM nn[1[3U[5[7U5555UlS S S .Ul[<R>"S[<R@5Ul![DTU]"SUUUUUS.UD6 g![ a/ [ RS5 [SSS9Ul SUlGNf=f!,(df  GN=fs sn n fs sn n f!,(df  GN =fs snf)NF)lstriprstriprzKftfy or spacy is not installed using custom BasicTokenizer instead of ftfy.)rBrCutf-8encoding ri<|startoftext|> <|endoftext|>)rrz[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)errors unk_token bos_token eos_token pad_tokenrp)$ isinstancestrrftfyfix_text ImportErrorloggerinfor;nlpopenjsonloadencoderitemsdecoderrr% byte_encoder byte_decoderreadr3r7tuplerrrrd bpe_rankscacher1compile IGNORECASEpatsuperrE)rDrrrrrrrkwargsr vocab_handlekv merges_handle bpe_mergesmerge __class__s r$rECLIPTokenizer.__init__sJTT]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir  !  MMDM *w /<99\2DL0)-););)=>)=)=>  ,..2.?.?.E.E.GH.GdaQT.GH + 0M&++-335;;DA!FYZJ18BC ueEKKM* Cc*eC O.DEF):_] :: n MM       - ! KKe f%EERDH DM ! 0 />I 0 0Cs<)H I II 31I&0#I8 5II I& I5c,[UR5$r>)rdrrDs r$ vocab_sizeCLIPTokenizer.vocab_sizeLs4<<  r.cB[UR40URD6$r>)rradded_tokens_encoderrs r$ get_vocabCLIPTokenizer.get_vocabPsDLL>D$=$=>>r. token_ids_0 token_ids_1returncjUR/nUR/nUcX1-U-$X1-U-U-U-U-$)a| Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CLIP sequence has the following format: - single sequence: `<|startoftext|> X <|endoftext|>` Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) bos_token_id eos_token_idrDrrrrs r$ build_inputs_with_special_tokens.CLIPTokenizer.build_inputs_with_special_tokensSsQ(&&' &&'  *Y6 6&2Y>LyXXr.already_has_special_tokensc>U(a[TU]XSS9$UcS/S/[U5--S/-$S/S/[U5--S/-S/-S/[U5--S/-$)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrrr)rget_special_tokens_maskrd)rDrrrrs r$r%CLIPTokenizer.get_special_tokens_maskns& &72']a3   31#K 001QC7 7sqcC ,,-3qc9aS3{CS=STXYWZZZr.cUR/nUR/nUc[X1-U-5S/-$[X1-U-U-U-U-5S/-$)aG Create a mask from the two sequences passed. CLIP does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. r)rrrdrs r$$create_token_type_ids_from_sequences2CLIPTokenizer.create_token_type_ids_from_sequencessl &&' &&'  y.:;qcA A9*Y6B[PS\\]ab`cccr.c ^UTR;aTRU$[USS5USS-4-n[U5nU(dUS-$[UU4SjS9nUTR;aOUupV/nSnU[ U5:aUR XX5n URX(U 5 U nX(U:Xa6U[ U5S- :a$X(S-U:XaURXV-5 US- nOURX(5 US- nU[ U5:aM[U5nUn[ U5S:XaO [U5nMSRU5nUTRU'U$![a URX(S5 Mlf=f) NrccN>TRRU[S55$)Ninf)rgetfloat)pairrDs r$#CLIPTokenizer.bpe..s1C1CD%PU,1Wr.keyrrr0) rrr-minrrdindexrP ValueErrorrrR) rDrVr)r*bigramfirstsecondnew_wordrfjs ` r$bpeCLIPTokenizer.bpes DJJ ::e$ $U3BZ E"I$6#88$6> !$WXFT^^+"MEHAc$i- 5,A OOD1I.A7e#CIM(9dq5kV>SOOEN3FAOODG,FAc$i- XHD4yA~!$9:xx~  5 +"OODH-sE!!FFc^/nTRc+SRTRRU55nO([ TRU55R 5n[ R"TRU5HcnSRU4SjURS555nURSTRU5RS555 Me U$)zTokenize a string.r0r]c3B># UHnTRUv M g7fr>)r).0r#rDs r$ *CLIPTokenizer._tokenize..s .C!!!$.Csrc3$# UHov M g7fr>rp)r bpe_tokens r$rrsT9SIi9Ss) rrRrrXr5rNr1findallrencoderPrr7)rDr4 bpe_tokensrVs` r$ _tokenizeCLIPTokenizer._tokenizes == 88DHH--d34D#DMM$$78>>@DZZ$/EGG.3ll7.CE   T%9N9Ns9ST T 0 r.c~URRXRRUR55$)z0Converts a token (str) in an id using the vocab.)rrr)rDrVs r$_convert_token_to_id"CLIPTokenizer._convert_token_to_ids*||||'7'7'GHHr.c8URRU5$)z=Converts an index (integer) in a token (str) using the vocab.)rr)rDrs r$_convert_id_to_token"CLIPTokenizer._convert_id_to_tokens||&&r.cSRU5n[UVs/sHo0RUPM sn5nURSURS9R SS5R 5nU$s snf)z:Converts a sequence of tokens (string) in a single string.r]r)rrr0)rR bytearrayrdecoderreplacer3)rDr8r4c byte_arrays r$convert_tokens_to_string&CLIPTokenizer.convert_tokens_to_stringskwwvdCd 1 1! 4dCD    =EEfcRXXZ  DsA4save_directoryfilename_prefixc [RRU5(d%[R SR U55 g[RR X(aUS-OS[S-5n[RR X(aUS-OS[S-5n[USSS9nUR[R"URS S S S 9S -5 SSS5 Sn[USSS9nURS5 [URR5SS9HXupXi:wa&[R!SR U55 U nURSR U5S -5 US- nMZ SSS5 X44$!,(df  N=f!,(df  X44$=f)Nz*Vocabulary path ({}) should be a directory-r]rrwrrrTF)indent sort_keys ensure_asciirrz#version: 0.2 c US$)Nrrp)kvs r$r/CLIPTokenizer.save_vocabulary..s Y[\]Y^r.rzqSaving vocabulary to {}: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!r0r)ospathisdirrerrorformatrRVOCAB_FILES_NAMESrwriterdumpsrsortedrrwarning) rDrrr merge_filefrwriterr token_indexs r$save_vocabularyCLIPTokenizer.save_vocabularysww}}^,, LLELL^\ ] WW\\ o_s22QbcoQpp WW\\ o_s22QbcpQqq *cG 4 GGDJJt||ATYZ]aa b5 *cG 4 LL* ++1$..2F2F2HN^+_' 'NNMMSVT^M_(E SXXj1D89 ,`5%%!5 45 4%%s?4F$BF5$ F25 G) rrrrrrrrrr)rrrrrr>)NF)rtrurvrwrxrvocab_files_namesmodel_input_namesrEpropertyrrrintrrboolrrrrrrrrrrry __classcell__)rs@r$r{r{sK2*$&67 !#!!/ b!!?JNY9Y3;DI3FY cY8sx[9[3;DI3F[ko[ c[[:JNd9d3;DI3Fd cd.(T I'&c&HSM&]bcf]g&&r.r{)rxrrrL functoolsrtypingrrrregexr1tokenization_utilsrr r r r utilsr get_loggerrtrrr%r-r5r9r;r{__all__rpr.r$rs% ((oo   H %   0 ^^BF&'F&R  r.