o Zh@s^dZddlZddlZddlZddlmZddlmZddlm Z m Z m Z ddl m ZddlmZddlmZddlmZdd lmZmZmZmZd d lmZd d lmZd d lm Z d dl!m"Z"d dl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-d dl.m/Z/m0Z0m1Z1e12e3Z4dZ5dZ6dZ7dZ8dZ9e$d7Z$eeeedZ:e5e8dZ;e0e$Gddde)Zd?Z+d@e,dAe-dBedCedDeedEeef dFdGZ.de,j/e-j0ddHddddddddddddfdIe#e!e1e!e2e!e3e!e4fd=e d@e,dAe-dBeedCedJe dDeedEeedKeed ee d!ee d"e d#e d$e d%e d&e dLe de5f&dMdNZ6dde,j/e-j0ddHddddddddddddfdS)fPreTrainedTokenizerFastaQ Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). Nslow_tokenizer_classcs|dd}|dd}|dd}|dd}|dd}|di}|dd|_|r:|dur:|jdur:td |durDt|} nw|durP|sPt|} nk|rWt |} nd|durt |d } | d d } | d } | d} t | | \} }| | t |dkr| |n3|jdur|dur|j|i|}t |} n|s|d d|_|dg|_t |dd} d}ntd| |_|dur| |jd|_|jj}|dur|jjd+i||d|d|d|d|d|d|d|dn|j|jj}|dur<|jjd+i||d|d|d|d|d|d|d|d|d|dtjd+i||j|j_d d!|jDfd"d#t|d$d%d&Dt |j!"d'd#Dfd(d#|j#D7t dkrg}|j$}D].}t%|t&r|j'pt(||vnt(||v}t%|t(rt&||d)}n||_'|)|q|r|*|z0t+,|j-j./}|d|j|jkrt0t1|d*}|j|d<|d+i||j-_.WdSWdSt2yYdSw),Ntokenizer_objectZ__slow_tokenizer gguf_filer$ from_slowFadded_tokens_decoderadd_prefix_spacezCannot instantiate this tokenizer from a slow version. If it's based on sentencepiece, make sure you have sentencepiece installed.r%configZ model_type tokenizertokenizer_configradditional_special_tokensT)Z from_tiktokena9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one. max_lengthtruncation_side directionstridetruncation_strategystrategy pad_tokenpad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofcSsh|]}tt|qShashrepr.0tokenr=r=S/var/www/auris/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py z3PreTrainedTokenizerFast.__init__..cs$g|]\}}tt|vr|qSr=r>)rBindexrC)added_tokens_decoder_hashr=rD s z4PreTrainedTokenizerFast.__init__..cS|dSNrr=)xr=r=rDz2PreTrainedTokenizerFast.__init__..keycSsg|]}t|qSr=)strrAr=r=rDrIscs g|] }|vr|vr|qSr=r=rA)encoder tokens_to_addr=rDrIs)specialtyper=)3popgetr,r' ValueErrorcopydeepcopy TokenizerFast from_filerrrupdatelenr%r0 _tokenizer init_kwargs_decode_use_source_tokenizer truncationenable_truncation setdefault no_truncationpaddingenable_paddingsuper__init__split_special_tokensencode_special_tokensr+sorteditemslistadded_tokens_encoderkeysZall_special_tokens_extendedZall_special_tokens isinstancerrTrQappend add_tokensjsonloadsbackend_tokenizer pre_tokenizer __getstate__getattrpre_tokenizers_fast Exception)selfargskwargsr(Zslow_tokenizerr)Zfast_tokenizer_filer*r+Zfast_tokenizerZ gguf_param architectureZtokenizer_dictr/Zadditional_kwargs _truncation_paddingtokensspecial_tokensrCZ is_specialZ pre_tok_stateZ pre_tok_class __class__)rHrRrSrDribs                         z PreTrainedTokenizerFast.__init__returncCdS)NTr=r|r=r=rDis_fastszPreTrainedTokenizerFast.is_fastcCr)z `bool`: Whether or not the slow tokenizer can be saved. Usually for sentencepiece based slow tokenizer, this can only be `True` if the original `"sentencepiece.model"` was not deleted. Tr=rr=r=rDcan_save_slow_tokenizersz/PreTrainedTokenizerFast.can_save_slow_tokenizercC|jjddS)zP `int`: Size of the base vocabulary (without the added tokens). FZwith_added_tokensr_Zget_vocab_sizerr=r=rD vocab_sizesz"PreTrainedTokenizerFast.vocab_sizecCr)NTr)r_ get_vocabrr=r=rDrsz!PreTrainedTokenizerFast.get_vocabcCs|SN)rrr=r=rDvocabszPreTrainedTokenizerFast.vocabcC ddt|jdddDS)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. cSi|]\}}|j|qSr=contentrBvkr=r=rD rFz@PreTrainedTokenizerFast.added_tokens_encoder..cSrJrKr=itemr=r=rDrMrNz>PreTrainedTokenizerFast.added_tokens_encoder..rOrlr+rmrr=r=rDros z,PreTrainedTokenizerFast.added_tokens_encodercCs |jS)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `Dict[str, int]`: The added tokens. )r_Zget_added_tokens_decoderrr=r=rDr+s z,PreTrainedTokenizerFast.added_tokens_decodercCr)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. cSrr=rrr=r=rDrrFz;PreTrainedTokenizerFast.get_added_vocab..cSrJrKr=rr=r=rDrMrNz9PreTrainedTokenizerFast.get_added_vocab..rOrrr=r=rDget_added_vocab s z'PreTrainedTokenizerFast.get_added_vocabcCr)zD Size of the full vocabulary with the added tokens. Trrrr=r=rD__len__szPreTrainedTokenizerFast.__len__cCs|jS)zc `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )r_rr=r=rDrvsz)PreTrainedTokenizerFast.backend_tokenizercCs|jjS)zU `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )r_decoderrr=r=rDr szPreTrainedTokenizerFast.decoderFTencodingreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosec Cs|dur d|jv}|durd|jv}|r |jdur |g|j} n|g} tt} | D]>} | d| j|r=| d| j|rG| d| j|rQ| d| j|r[| d| j |rg| dt | jq)| | fS)a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) NZtoken_type_idsattention_mask input_idsspecial_tokens_maskZoffset_mappingr;) Zmodel_input_namesZ overflowingrrnrridsZtype_idsrroffsetsr^) r|rrrrrrrr encodingsZ encoding_dicter=r=rD_convert_encoding's,  z)PreTrainedTokenizerFast._convert_encodingrcs&t|tr |Sfdd|DS)aX Converts a token string (or a sequence of tokens) in a single integer id (or a Iterable of ids), using the vocabulary. Args: tokens (`str` or `Iterable[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. cg|]}|qSr=)#_convert_token_to_id_with_added_vocrArr=rDrIdzAPreTrainedTokenizerFast.convert_tokens_to_ids..)rqrQrr|rr=rrDconvert_tokens_to_idsVs z-PreTrainedTokenizerFast.convert_tokens_to_idsrCcCs|j|}|dur |jS|Sr)r_ token_to_idZ unk_token_id)r|rCrGr=r=rDrfs z;PreTrainedTokenizerFast._convert_token_to_id_with_added_vocrGcCs|jt|Sr)r_ id_to_tokenint)r|rGr=r=rD_convert_id_to_tokenlsz,PreTrainedTokenizerFast._convert_id_to_token new_tokenscCs|r|j|S|j|Sr)r_add_special_tokensrs)r|rrr=r=rD _add_tokensos  z#PreTrainedTokenizerFast._add_tokenspaircCs |j|S)aG Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )r_num_special_tokens_to_add)r|rr=r=rDrus z1PreTrainedTokenizerFast.num_special_tokens_to_addrskip_special_tokenscCs`t|tr |j|Sg}|rt|jnt}|D]}t|}||vr$q||j|q|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `List[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `List[str]`: The decoded token(s). )rqrr_rsetZall_special_idsrr)r|rrrZ ids_to_skiprGr=r=rDconvert_ids_to_tokenss  z-PreTrainedTokenizerFast.convert_ids_to_tokenstextrcKs|jd|||d|S)N)r text_pairrr=)Z encode_plusr)r|rrrr~r=r=rDtokenizesz PreTrainedTokenizerFast.tokenizepadding_strategyr5r1r4r<r:c s|jj|jj}|tjkrdur|jn&|||j|jd}dur'd} n fdd|D} | |kr=|jjdi||t j krO|durM|j dSdS|t j krV|nd} | |dur_|n|j |j|j|j|d}||krz|jjdi|dSdS)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)r1r4r6r3csi|] }||dqSrrW)rBrrr=rDrszFPreTrainedTokenizerFast.set_truncation_and_padding..)r;r3Zpad_idr7r9r<r=)r_rbrfrDO_NOT_TRUNCATErevaluer2rcr DO_NOT_PADZ no_paddingZ MAX_LENGTHr:Z pad_token_idr7r8rg) r|rr5r1r4r<r:rtargetcurrentr;r=rrDset_truncation_and_paddings>!    z2PreTrainedTokenizerFast.set_truncation_and_paddingrbatch_text_or_text_pairsis_split_into_wordsreturn_tensorsrjcs&t|ttfstdt|dj|||||| djj|kr&|j_jj|||d}fdd|D}i}|dd D]fdd|D}||<qId d|D}r~g}t |D]\}\}}||gt |d 7}qh||d <|d D] } ||qt ||| d S) Nz:batch_text_or_text_pairs has to be a list or a tuple (got ))rr5r1r4r<r:)rZis_pretokenizedc s&g|]}j|dqS))rrrrrrrr)r)rBr)rrrrrrr|rr=rDrI)s z>PreTrainedTokenizerFast._batch_encode_plus..rcs"g|] \}}|D]}|q qSr=r=)rBr_rrOr=rDrI?s"cSsg|] \}}|D]}|qqSr=r=)rBrrrr=r=rDrIAsroverflow_to_sample_mapping)Z tensor_type)rqtuplern TypeErrorrUrr_rkZ encode_batchrp enumerater^&_eventual_warn_about_too_long_sequencer)r|rrrr5r1r4rr<r:rrrrrrrrrjrZtokens_and_encodingsZsanitized_tokensstackZsanitized_encodingsritoksrrr=) rPrrrrrrr|rrD_batch_encode_plussF     z*PreTrainedTokenizerFast._batch_encode_plusrcKs|r||fgn|g}|j|fid|d|d|d|d|d|d| d| d | d | d | d |d |d|d|d|d||}| durY|sYtdd|D|j}||d|||S)Nrrrr5r1r4r<r:rrrrrrrrrjcSs8i|]\}}|t|dkrt|dtr|dn|qS)r)r^rqrn)rBrPrr=r=rDrs&z8PreTrainedTokenizerFast._encode_plus..r)rrrmrr)r|rrrrr5r1r4rr<r:rrrrrrrrrjr~Z batched_inputZbatched_outputr=r=rD _encode_plusOs`      z$PreTrainedTokenizerFast._encode_pluscCs$|jjdur |jj|Sd|S)N )rvrdecodejoinrr=r=rDconvert_tokens_to_strings  z0PreTrainedTokenizerFast.convert_tokens_to_string token_idsclean_up_tokenization_spacescKsV|dd|_t|tr|g}|jj||d}|dur|n|j}|r)||}|S|S)NZuse_source_tokenizerF)r)rVrarqrr_rrZclean_up_tokenization)r|rrrr~rZ clean_textr=r=rD_decodes  zPreTrainedTokenizerFast._decodesave_directory file_names legacy_formatfilename_prefixc s6t|}jdur|durtd|dus|duo jduo j}|dup(|du}|r}tj||r5|dndt}fddj D}|rot |d d d } t j |d ddd d} | | Wdn1sjwYj||d} || |f}|rtj||r|dndt} j| || f}|S)z Save a tokenizer using the slow-tokenizer/legacy format: vocabulary + added tokens as well as in a unique JSON file containing {config + vocab + added-tokens}. NTzYour tokenizer does not have a legacy version defined and therefore cannot register this version. You might consider leaving the legacy_format at `None` or setting it to `False`.F-cs i|] \}}|jkr||qSr=)r)rBtokrGrr=rDrs z.wzutf-8)r)indent sort_keys ensure_ascii )r)rQr'rXrospathrADDED_TOKENS_FILErormopenrtdumpswriteZsave_vocabularyTOKENIZER_FILErvsave) r|rrrrZ save_slowZ save_fastZadded_tokens_fileZ added_vocabfZout_strZ vocab_filesr$r=rrD_save_pretraineds<    z(PreTrainedTokenizerFast._save_pretrainedc s,t|j}|d}|d} d} |dddkr)i|dd<g|dd<nW|ddd kre|dd durd|dd } |dd| d } durU| vrU| } d |dd <| d gg|dd<n|ddd vrti|dd<n td|ddddurd|dvr|ddvr|dd|dd<tt|g} |D]5} | dd}| dd}|ddd kr|sqdur| dvrՈ| d| d<| t d+i| q|dur| ||dddkrd|vr|dddur|dd|d<|dddkr'd|vr'|dddur'|dd|d<|ddd kr9| dur9| |d<|ddurn|dddksg|dddkrnd|dvrnt dd|ddDrnt j|d<t|dd}|d+|| d|}j|||d| dur!t}d| vr| dD]D}| d|d }durfd!d"|D}|| d|d <|D]}|}|durtd#qfd$d"|D| d|d%<qd&D]0}|| vr| |\}}dur|vr|}|}|dur td#||g| |<q| |d<tt||j}tj}|d'|D]A}t||durrt||}durO|vrO|}|j|d}t|t rnt ||j|j|j|jd(d)||<q2|||<q2|j }|dur| |t!|d kr||d'<|j"d+d*i|S),uf Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `List[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`Dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. added_tokenspost_processorNmodelrUr rZmergesr!unk_idrg)r"r#z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece. unk_tokenrTidrZcontinuing_subword_prefixZend_of_word_suffixrw ByteLevelSequenceZ pretokenizerscss|] }|ddkVqdS)rUrNr=)rBZ pretokenizerr=r=rD Ds  zBPreTrainedTokenizerFast.train_new_from_iterator..Zinitial_alphabet)rr)r;trainerrrcsg|]}||qSr=rrA)special_tokens_mapr=rDrIVrFzCPreTrainedTokenizerFast.train_new_from_iterator..zQAttempted to set a token in the post processor that does not exist in the mappingcrr=)rrA)r.r=rDrI_rr)clssepr0T) single_wordlstriprstrip normalizedrTr(r=)#rtrur_Zto_strrVrXr[Zfrom_strrrrrextendanyrzralphabetMODEL_TO_TRAINER_MAPPINGZtrain_from_iteratorrr`rYrZSPECIAL_TOKENS_ATTRIBUTESremoveryZ_special_tokens_maprWrqrrr r r0r^r)r|Z text_iteratorrr;Znew_special_tokensrr~Ztokenizer_jsonrrrrrZ added_tokenrTrZ trainer_classrZtrained_tokenizer_jsonrPrrCZtoken_idZ special_tokenZspecial_tokens_listZspecial_token_fullr0r=)rr.rDtrain_new_from_iterators"                  "             z/PreTrainedTokenizerFast.train_new_from_iterator)NNFFFFT)F)NF)FN)NN)NNN)?__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESZvocab_files_namesr'r__annotations__ripropertyboolrrrrdictrQrrrorr+rrr[rv DecoderFastr EncodingFastrrrrnrrrrrrrrrrrrrrrrrrrrrrrrrPathLikerr __classcell__r=r=rrDr&Qs   |     */  $ P      ^       =    5r&)=rrYrtr collectionsrcollections.abcrtypingrrrZtokenizers.pre_tokenizersZpre_tokenizersrzZ tokenizersrrrr[Ztokenizers.decodersr rZtokenizers.trainersr r r r rZintegrations.ggmlrZmodeling_gguf_pytorch_utilsrZtokenization_utilsrZtokenization_utils_baserrrrrrrrrrutilsrrrZ get_loggerrloggerrZSPECIAL_TOKENS_MAP_FILEZTOKENIZER_CONFIG_FILEZTIKTOKEN_VOCAB_FILErrrr&r=r=r=rDsB           0