eThSrSSKrSSKrSSKrSSKJr SSKJr SSKJ r J r J r SSK J r SSKJr SSKJr SSKJr SS KJrJrJrJr S S KJr S S KJr S S KJ r S SK!J"r" S SK#J$r$J%r%J&r&J'r'J(r(J)r)J*r*J+r+J,r,J-r- S SK.J/r/J0r0J1r1 \1Rd"\35r4Sr5Sr6Sr7Sr8Sr9\$S- r$\\\\S.r:\5\8S.r;\0"\$5"SS\)55r\\ S\4S?jjr:SKS@\$\\;Rx4SA\ \SB\\ SC\\S\ \4 SDjjr=SLSEjr>SFr?U=r@$)MPreTrainedTokenizerFastQa5 Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). Nslow_tokenizer_classc  >URSS5nURSS5nURSS5nURSS5nURSS5nURS05nURSS5UlU(aUcURc [ S 5eUb[ R "U5n GOUbU(d[R"U5n OU(a [U5n OUbk[URS 55n U S S n U S n U Sn [X5upURU 5 [U5S:aURU5 O}URb#USLaUR"U0UD6n[U5n OMU(d;URS S5UlURS/5Ul[USS9n SnO [ S5eXlUbURUR"5 SUlUR R&nUbqUR R("S$0UD6 UR+SUS5 UR+SUS5 UR+SUS5 UR+SUS5 OUR R-5 UR R.nUbUR R0"S$0UD6 UR+SUS5 UR+SUS5 UR+SUS5 UR+SUS5 UR+SUS5 [2TU]h"S$0UD6 UR6UR lUR:Vs1sHn[=[?U55iM nn[AURC5S S!9VVs/sH"unn[=[?U55U;dM UPM$ nnn[EURFRI55UVs/sHn[KU5PM sn-nUURLVs/sHnUU;dM UU;dMUPM sn- n[U5S:a/nURNnUHn[QU[R5(a!URT=(d [KU5U;O [KU5U;n[QU[J5(a [SUUS"9nOUUl*URWU5 M U(aURYU5 [ZR\"UR^R`Rc55nURSUR5UR:waF[e[fURS#55nURUS'U"S$0UD6UR^l0ggs snfs snnfs snfs snf![ha gf=f)%Ntokenizer_object__slow_tokenizer gguf_filer% from_slowFadded_tokens_decoderadd_prefix_spacezCannot instantiate this tokenizer from a slow version. If it's based on sentencepiece, make sure you have sentencepiece installed.r&config model_type tokenizertokenizer_configradditional_special_tokensT) from_tiktokena9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one. max_lengthtruncation_side directionstridetruncation_strategystrategy pad_tokenpad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofc US$Nr)xs \/var/www/auris/envauris/lib/python3.13/site-packages/transformers/tokenization_utils_fast.py2PreTrainedTokenizerFast.__init__..s STUVSWkey)specialtyperF)5popgetr1r* ValueErrorcopydeepcopy TokenizerFast from_filerrrupdatelenr&r6 _tokenizer init_kwargs_decode_use_source_tokenizer truncationenable_truncation setdefault no_truncationpaddingenable_paddingsuper__init__split_special_tokensencode_special_tokensr0hashreprsorteditemslistadded_tokens_encoderkeysstrall_special_tokens_extendedall_special_tokens isinstancerrNappend add_tokensjsonloadsbackend_tokenizer pre_tokenizer __getstate__getattrpre_tokenizers_fast Exception)selfargskwargsr,slow_tokenizerr.fast_tokenizer_filer/r0fast_tokenizer gguf_param architecturetokenizer_dictr5additional_kwargs _truncation_paddingtokenadded_tokens_decoder_hashindex tokens_to_addencodertokensspecial_tokens is_special pre_tok_state pre_tok_class __class__s rHrc PreTrainedTokenizerFast.__init__bs!::&8$?$6=JJ{D1 $jj)94@JJ{E2 %zz*@"E & +=u E /D4M4M4U0   '!]]+;   3[5L M OO ) ) +??**   OO * * 6X 6   k8K+@ A   18M3J K   nh{.C D   lHX,> ?   2H=Q4R S "6"040I0I-DHD]D]$^D]5T$u+%6D]!$^!'';'A'A'C X X uDK (AA  X  t005578Ta;bTa5CJTa;bb#?? ?e5PWCWE\aiv\vE?   }  !F!44N&"%44]]Bc%jN&BU~5 eS))&ujAE$.EM e$''  JJt'='='K'K'X'X'Z[M  !3T5J5JKtOdOdd '(;]=N=Nv=V W 484I4I 017D7U}7U&&4e?%_ W WWB'W W%$W%returncg)NTrFr{s rHis_fastPreTrainedTokenizerFast.is_fastsrKcg)z `bool`: Whether or not the slow tokenizer can be saved. Usually for sentencepiece based slow tokenizer, this can only be `True` if the original `"sentencepiece.model"` was not deleted. TrFrs rHcan_save_slow_tokenizer/PreTrainedTokenizerFast.can_save_slow_tokenizers rKc4URRSS9$)z@ `int`: Size of the base vocabulary (without the added tokens). Fwith_added_tokensrYget_vocab_sizers rH vocab_size"PreTrainedTokenizerFast.vocab_sizes ---FFrKc4URRSS9$)NTr)rY get_vocabrs rHr!PreTrainedTokenizerFast.get_vocabs((4(@@rKc"UR5$N)rrs rHvocabPreTrainedTokenizerFast.vocabs~~rKc[URR5SS9VVs0sHupURU_M snn$s snnf)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. c US$rErFitems rHrI>PreTrainedTokenizerFast.added_tokens_encoder.. dhijdkrKrLrhr0ricontentr{vks rHrk,PreTrainedTokenizerFast.added_tokens_encoders? *00I0I0O0O0QWk)lm)l 1 )lmmmAc6URR5$)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `Dict[str, int]`: The added tokens. )rYget_added_tokens_decoderrs rHr0,PreTrainedTokenizerFast.added_tokens_decoders7799rKc[URR5SS9VVs0sHupURU_M snn$s snnf)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. c US$rErFrs rHrI9PreTrainedTokenizerFast.get_added_vocab..rrKrLrrs rHget_added_vocab'PreTrainedTokenizerFast.get_added_vocab s?*00I0I0O0O0QWk)lm)l 1 )lmmmrc4URRSS9$)z4 Size of the full vocabulary with the added tokens. Trrrs rH__len__PreTrainedTokenizerFast.__len__s---EErKcUR$)zS `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rYrs rHru)PreTrainedTokenizerFast.backend_tokenizers rKc.URR$)zE `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rYdecoderrs rHrPreTrainedTokenizerFast.decoder s &&&rKFTencodingreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosec UcSUR;nUcSUR;nU(aURbU/UR-n OU/n [[5n U Hn U SR U R 5 U(aU SR U R 5 U(aU SR U R5 U(aU SR U R5 U(aU SR U R5 U(dMU SR [U R 55 M X4$)ar Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) token_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingrB) model_input_names overflowingrrjrqidstype_idsrroffsetsrX) r{rrrrrrrr encodings encoding_dictes rH_convert_encoding)PreTrainedTokenizerFast._convert_encoding's( ! ($48N8N$N ! ($48N8N$N ! $)=)=)I! X%9%99I! I#D) A + & - -aee 4$./66qzzB$./66q7G7GH)34;;A This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rYnum_special_tokens_to_add)r{rs rHr1PreTrainedTokenizerFast.num_special_tokens_to_addus&88>>rKrskip_special_tokensc@[U[5(aURRU5$/nU(a[ UR 5O [ 5nUH?n[U5nXT;aMUR URRU55 MA U$)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `List[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `List[str]`: The decoded token(s). )rprrYrsetall_special_idsrq)r{rrr ids_to_skiprs rHconvert_ids_to_tokens-PreTrainedTokenizerFast.convert_ids_to_tokenss c3  ??..s3 33Fc$../CE EJE# MM$//55e< =   rKtextrc HUR"SXUS.UD6R5$)N)r text_pairrrF) encode_plusr)r{rrrr}s rHtokenize PreTrainedTokenizerFast.tokenizes(kTN`kdjkrrttrKpadding_strategyr<r8r;rCrAcURRnURRnU[R:XaUbURR 5 OdUUUR URS.n UcSn O"U V s0sHoURU S5_M n n X:waURR"S0U D6 U[R:XaUbURR5 ggU[R:XaUOSn U UbUO URURUR UR"US.n X:waURR$"S0U D6 ggs sn f)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)r8r;r=r:)rBr:pad_idr>r@rCrF)rYr\r`rDO_NOT_TRUNCATEr_valuer9rQr]r DO_NOT_PAD no_padding MAX_LENGTHrA pad_token_idr>r?ra) r{rr<r8r;rCrArrtargetcurrentrrBs rHset_truncation_and_padding2PreTrainedTokenizerFast.set_truncation_and_paddingsRBoo00 ??** "4"D"D D&--/) /55!11 F"@FG1kooa66G 11;F; 99 9#**,$$47Q7Q#QZW[F -9-E\4K\K\++!^^#55&8 F!..88"%HsE'rbatch_text_or_text_pairsis_split_into_wordsreturn_tensorsrdc2[U[[45(d[S[ U5S35eUR UUUUUU S9 UR RU:waUUR lUR RUUUS9nUVs/sHnURUU U U UUUUS9PM nn0nUSSR5H.nUVVVs/sHunnUUHnUPM M nnnnUUU'M0 UVVVs/sHunnUHnUPM M nnnnU (a4/n[U5HununnUU/[US5-- nM UUS'USHn URU UU5 M [UUU S 9$s snfs snnnfs snnnf) Nz:batch_text_or_text_pairs has to be a list or a tuple (got ))rr<r8r;rCrA)ris_pretokenized)rrrrrrrrrroverflow_to_sample_mapping) tensor_type)rptuplerj TypeErrorrOr rYre encode_batchrrl enumeraterX&_eventual_warn_about_too_long_sequencer)!r{rrrr<r8r;rrCrArrrrrrrrrdrrtokens_and_encodingssanitized_tokensrMr_rstacksanitized_encodingsritoksrs! rH_batch_encode_plus*PreTrainedTokenizerFast._batch_encode_pluss.2UDMBBLTRjMkLllmn  ''- 3!1% (  ?? 0 04H H4HDOO 1OO00 $1/1 .& &  " "!&;&;*C+E'=+ # &  ('*1-224C&:N&:74DIqQIQ&:EN$) S !51ES0DWQdqdq0DS %)+ & )*> ? 9D!*qcC[8I4J.JJ*!@=W 9 :)+6I  7 7 :w W7-/BP^__I ,OSs FF Frc U(aX4/OU/nUR"U40SU_SU_SU_SU_SU_SU_SU _SU _S U _S U _S U _S U_S U_SU_SU_SU_SU_UD6nU cnU(dg[UR5VVs0sH5unnU[U5S:a[ US[ 5(aUSOU_M7 snnUR 5nURUSUU5 U$s snnf)Nrrrr<r8r;rCrArrrrrrrrrdrr)r"rrirXrprjrr)r{rrrrr<r8r;rrCrArrrrrrrrrdr} batched_inputbatched_outputrMrs rH _encode_plus$PreTrainedTokenizerFast._encode_plusOsk.09$*+tf 00  3  2 .  !4  "    2 & * #8 #8 '@ (B $: (! "# $"6' 0  !*C*'5&:&:&<&< Uc%j1nE!Hd9S9S%(Y^^&<(( N 33N;4OQ[]des7[SUSS55(a![RR5US'[ USSnU"S'X-S.UD6nU R#XUS9 U Gb@[R"U R55nSU ;aU SHnU SUSnUb"UVs/sHnUR%UU5PM nnUU SUS'UH"nU R'U5nUbM[ S 5e UVs/sHnU R'U5PM snU SUS!'M S"HGnUU ;dM U UunnUb UU;aUUnU R'U5nUc [ S 5eUU/U U'MI U US'[ R"[R"U55n UR(R+5n[,R.R+5nUR1S#5 UHn[3UU5cM[3UU5nUb UU;aUUnUR4R%US5n[7U[5(a;[UUR8UR:UR<UR>S$S%9UU'MUUU'M UR@nUbURU5 [CU5S :aUUS#'URD"S'S&U 0UD6$s snfs snf)(u Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `List[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`Dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. added_tokenspost_processorNmodelrOr!rmergesr"unk_idrg)r#r$z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece. unk_tokenrNidrcontinuing_subword_prefixend_of_word_suffixrv ByteLevelSequence pretokenizersc32# UH nUSS:Hv M g7f)rOraNrF).0 pretokenizers rH BPreTrainedTokenizerFast.train_new_from_iterator..Ds!(X !(K7(Xsinitial_alphabet)rr)rBtrainerrrzQAttempted to set a token in the post processor that does not exist in the mappingr)clssepr6T) single_wordlstriprstrip normalizedrNr,rF)#rsrtrYto_strrPrRrUfrom_strrHrqrextendanyryraalphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorrQrrZrSrSPECIAL_TOKENS_ATTRIBUTESremoverx_special_tokens_maprprmrnrorpr6rXr)r{ text_iteratorrrBnew_special_tokensspecial_tokens_mapr}tokenizer_jsonrXrYr]r\r4r added_tokenrNr trainer_classrjtrained_tokenizer_jsonrMrrtoken_id special_tokenspecial_tokens_listspecial_token_fullr6s rHtrain_new_from_iterator/PreTrainedTokenizerFast.train_new_from_iteratorsdDDOO$:$:$<=%)).9 '++,<= ' "6 *e 3/1N7 #G ,02N7 #H - G $V , 9g&x0<'0:*73G4D3Ew'0 G $V ,0J J/1N7 #G ,Mn]dNeflNmMno>>   *~g66w' 48JJ3EU\F]^iFj3kN7 #K 0!**4::n+EF 'K!ooi6Gd+Ag&v.);G!-+i2HL^2^); 1)*:;C+,<=cB8LF)5TZ![TZ5"4"8"8"FTZ![FLN#34S9(C!'#,#8#8#?#+", s#"(ouCuntejIDYDYZ_D`ntCuN#34S9%@<"0  N2-m?(EtU#/ 'e 4 %1mGY6Y$6}$EM%)%=%=%A%A%%N"0*==$.%$6$B$B188188#5#@#@ $ %F5M%2F5M%)(%)$B$B!  ) % , ,-? @ ( )A -2KF. /~~CyCFCCq"\Dvs 4T$ T))r[rYr1r6r&)NNFFFFT)F)NF)FN)NN)NNN)A__name__ __module__ __qualname____firstlineno____doc__VOCAB_FILES_NAMESvocab_files_namesr*r__annotations__rcpropertyboolrrrrdictrmrrrkrr0rrrUru DecoderFastr EncodingFastrrrrjrrrrrrrrrrrrr rrrrrrrr"r'r-r5rDPathLikerUr__static_attributes__ __classcell__)rs@rHr(r(Qs *04-4zxGCGG A4S>A tCH~  nd38nnn:d3 ?&;::nc3hnFF = '''1504*/+0',#-(-( (~-( (~ -( $( -( %) -(!%-(-(-( tCH~tL11 2-(^UE#x}2D,EU%PSUYZ]U^P^J_U  7#7(3-76d5j+A&B6]`6 ?d?s?,GLd3i(?C sDI~ 8uSu uRVumqrumvuI9)I90I9 I9  I9 %SM I9smI9`$(,;,F,F2D2T2T$($),0&*(,0404*/+0',#%*+Y`"' OT-0$7H2I4PeKf f# Y` ! Y` * Y`0Y`SMY`Y`"Y`%SMY`smY`! Y` (~Y` (~Y` $(!Y`"%)#Y`$!%%Y`&'Y`()Y`*#+Y`, -Y`|DH#',;,F,F2D2T2T$($),0&*)-0404*/+0',#%*);I001;E)->">?@;! ; * ; 0 ;SM;;";%SM;sm;!; (~; (~;$(; %)!;"!%#;$%;&';(#);, -;z tCy S %*7; d3i("'/tn  8)-)- /c2;;.//#J/ ~ / "# / s /j rDrDrKr()=rrSrsrD collectionsrcollections.abcrtypingrrrtokenizers.pre_tokenizerspre_tokenizersry tokenizersrrr rUtokenizers.decodersr rtokenizers.trainersr r r rrintegrations.ggmlrmodeling_gguf_pytorch_utilsrtokenization_utilsrtokenization_utils_baserrrrrrrrrrutilsrrr get_loggerrloggerrKSPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILETIKTOKEN_VOCAB_FILErFrvrr(rFrKrHrs  #$''7/16^^:5=3   @?   H %"3/'( !! (6EXY,-| D5| D.| DrK