fThBPSSKrSSKrSSKrSSKrSSKJr SSKJr SSKJ r J r J r J r J r Jr SSKrSSKJr SSKJr SSKJr \R,"\5rS S S S S S.rSr\"SS9"SS\55rS\S\ \\ 4S\R:4SjrS\SS4SjrS\S\\ \ 44Sjr S/r!g)N)Path)copyfile)AnyDictListOptionalTupleUnion)PreTrainedTokenizer)logging)requiresz source.spmz target.spmz vocab.jsonztarget_vocab.jsonztokenizer_config.json) source_spm target_spmvocabtarget_vocab_filetokenizer_config_fileu▁) sentencepiece)backendsc ^\rSrSrSr\rSS/r\R"S5r S+S\ \ \ \4SS4U4S jjjrS rS \ S\ 4S jrS rS\ 4SjrS\ S\\ 4SjrS\S\ 4SjrU4SjrU4SjrS\\ S\ 4SjrS,S\\4SjjrSrSr\S\4Sj5rS,S\ S\ \ S\ \ 4Sjjr!S\ 4Sjr"Sr#S r$S\ 4S!jr%S"\ SS4S#jr&S$r'S%r(S-S&\S'\ \S(\)S\\4S)jjr*S*r+U=r,$).MarianTokenizer-a Construct a Marian tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: source_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the source language. target_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the target language. source_lang (`str`, *optional*): A string representing the source language. target_lang (`str`, *optional*): A string representing the target language. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. model_max_length (`int`, *optional*, defaults to 512): The maximum sentence length the model accepts. additional_special_tokens (`List[str]`, *optional*, defaults to `["", ""]`): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Examples: ```python >>> from transformers import MarianForCausalLM, MarianTokenizer >>> model = MarianForCausalLM.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> src_texts = ["I am a small frog.", "Tom asked his teacher for advice."] >>> tgt_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."] # optional >>> inputs = tokenizer(src_texts, text_target=tgt_texts, return_tensors="pt", padding=True) >>> outputs = model(**inputs) # should work ``` input_idsattention_maskz>>.+<U c0OU Ul[U5R5(d SU35eXl[ U5Ul[ U5UR ;a [S5e[ U 5UR ;deU (aL[ U5UlURR5VVs0sHupX_M snnUl /Ul OUR R5VVs0sHupX_M snnUl UR Vs/sH4oRS5(dMURS5(dM2UPM6 snUl XPlX`lX/Ul[#XR5Ul[#X R5UlUR$UlUR UlUR-5 [.TU]`"SUUUUU U URUU S. U D6 gs snnfs snnfs snf)Nzcannot find spm source z token must be in the vocabz>>z<<) source_lang target_lang unk_token eos_token pad_tokenmodel_max_lengthrrseparate_vocabs)rrexistsr$ load_jsonencoderstrKeyErrortarget_encoderitemsdecodersupported_language_codes startswithendswithrr spm_filesload_spm spm_source spm_target current_spmcurrent_encoder_setup_normalizersuper__init__)selfrrrrrrr r!r"r#rr$kwargskv __class__s f/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/marian/tokenization_marian.pyr9MarianTokenizer.__init__ms &5% -=> >9~--- "+,=">D -1-@-@-F-F-HI-HTQAD-HIDL,.D )-1\\-?-?-AB-ATQAD-ABDL>Bll2vlll[_N`1efeoeopteu1l2vD )&&$1#:/C/CD":/C/CD??#||    ##- 00/+  )JC2vs?H;H H=HHcSSKJn U"UR5RUlg![ [ 4a! [R"S5 SUlgf=f)Nr)MosesPunctNormalizerz$Recommended: pip install sacremoses.cU$Nr%)xs r?3MarianTokenizer._setup_normalizer..sQ) sacremosesrBr normalizepunc_normalizer ImportErrorFileNotFoundErrorwarningswarn)r:rBs r?r7!MarianTokenizer._setup_normalizersM / 7#78H8H#I#S#SD ./ / MM@ A#.D  /s'*.AArEc6U(aURU5$S$)zHCover moses empty string edge case. They return empty list for '' input!)rK)r:rEs r?rJMarianTokenizer.normalizes*+t##A&33rHcfURRXRUR5$rD)r6getr )r:tokens r?_convert_token_to_id$MarianTokenizer._convert_token_to_ids(##''/C/CDNN/STTrHtextcURRU5nU(aURS5/O/nX0RRSU54$)z6Remove language codes like >>fr<< before sentencepiecerrR)language_code_rematchgroupsub)r:rYr\codes r?remove_language_code$MarianTokenizer.remove_language_codesH%%++D1).ekk!n%B**..r4888rHclURU5up!URRU[S9nX#-$)N)out_type)r`r5encoder))r:rYr_piecess r? _tokenizeMarianTokenizer._tokenizes7..t4 !!(((<}rHindexcLURRXR5$)z?Converts an index (integer) in a token (str) using the decoder.)r-rUr )r:rhs r?_convert_id_to_token$MarianTokenizer._convert_id_to_tokens||~~66rHc &>[TU]"U40UD6$)a Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `List[str]`: The list of decoded sentences. )r8 batch_decode)r: sequencesr;r>s r?rmMarianTokenizer.batch_decodes*w#I888rHc &>[TU]"U40UD6$)aj Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. )r8decode)r: token_idsr;r>s r?rqMarianTokenizer.decodes0w~i2622rHtokenscZUR(a URO URn/nSnUH@nXPR;aXBR U5U-S-- n/nM/UR U5 MB XBR U5- nUR [S5nUR5$)zQUses source spm if _decode_use_source_tokenizer is True, and target spm otherwiserR ) _decode_use_source_tokenizerr3r4all_special_tokens decode_piecesappendreplaceSPIECE_UNDERLINEstrip)r:rtsp_modelcurrent_sub_tokens out_stringrVs r?convert_tokens_to_string(MarianTokenizer.convert_tokens_to_strings&*&G&G4??T__ E///445GH5PSVVV %'""))%0  ,,-?@@ ''(8#> !!rHcJUcXR/-$X-UR/-$)z=Build model inputs from a sequence by appending eos_token_id.) eos_token_id)r: token_ids_0 token_ids_1s r? build_inputs_with_special_tokens0MarianTokenizer.build_inputs_with_special_tokens s1  "3"3!44 4(D,=,=+>>>rHcHURUlURUlgrD)r3r5r(r6r:s r?_switch_to_input_mode%MarianTokenizer._switch_to_input_modes??#||rHclURUlUR(aURUlggrD)r4r5r$r+r6rs r?_switch_to_target_mode&MarianTokenizer._switch_to_target_modes*??   #'#6#6D  rHc,[UR5$rD)lenr(rs r? vocab_sizeMarianTokenizer.vocab_sizes4<<  rHsave_directoryfilename_prefixc[RRU5(d[R SUS35 g/nUR (a[RR UU(aUS-OS[S-5n[RR UU(aUS-OS[S-5n[URU5 [URU5 URU5 URU5 O\[RR X(aUS-OS[S-5n[URU5 URU5 [[S[S/URURUR/5GH$upxn [RR X(aUS-OSU-5n [RR!U5[RR!U 5:waB[RR#U5(a[%X5 URU 5 M[RR#U5(aM['U S 5n U R)5n U R+U 5 SSS5 URU 5 GM' [-U5$!,(df  N/=f) NzVocabulary path (z) should be a directory-rRrrrrwb)ospathisdirloggererrorr$joinVOCAB_FILES_NAMES save_jsonr(r+rzzipr1r3r4abspathisfileropenserialized_model_protowritetuple) r:rr saved_filesout_src_vocab_fileout_tgt_vocab_fileout_vocab_filespm_save_filename spm_orig_path spm_model spm_save_pathficontent_spiece_models r?save_vocabularyMarianTokenizer.save_vocabularysEww}}^,, LL,^,<   1 2   1 2WW\\/3!6rUfgnUo oN dllN 3   ~ .;> | ,.? .M N NN __doo .< 7 i GGLL/3!6rUf fMww}-1OOTVT[T[TbTbcpTqTq6""=1WW^^M22-."+4+K+K+M(HH12/""=1< "[!! /.s "K K( c"UR5$rD) get_src_vocabrs r? get_vocabMarianTokenizer.get_vocabLs!!##rHcB[UR40URD6$rD)dictr(added_tokens_encoderrs r?rMarianTokenizer.get_src_vocabOsDLL>D$=$=>>rHcB[UR40URD6$rD)rr+added_tokens_decoderrs r? get_tgt_vocabMarianTokenizer.get_tgt_vocabRsD''E4+D+DEErHcURR5nUR[R /SQ55 U$)N)r3r4r5rKr)__dict__copyupdaterfromkeys)r:states r? __getstate__MarianTokenizer.__getstate__Us4 ""$ MMm n  rHdc^UTl[TS5(d0TlU4SjTR5uTlTlTRTlTR5 g)Nrc3P># UHn[UTR5v M g7frD)r2r).0fr:s r? /MarianTokenizer.__setstate__..cs$+fWeRSHQ8L8L,M,MWes#&)rhasattrrr1r3r4r5r7)r:rs` r? __setstate__MarianTokenizer.__setstate__\sT t.//#%D +fW[WeWe+f(??  rHcg)zJust EOSr%)r:argsr;s r?num_special_tokens_to_add)MarianTokenizer.num_special_tokens_to_addgsrHc[UR5nURUR5 UVs/sH o3U;aSOSPM sn$s snf)Nrr)setall_special_idsremove unk_token_id)r:seqrrEs r?_special_token_mask#MarianTokenizer._special_token_maskksHd223t001:=>#Q/)q0#>>>sA rralready_has_special_tokenscU(aURU5$UcURU5S/-$URX-5S/-$)zCGet list where entries are [1] if a token is [eos] or [pad] else 0.r)r)r:rrrs r?get_special_tokens_mask'MarianTokenizer.get_special_tokens_maskpsQ &++K8 8  ++K8A3> >++K,EF!L LrH)rr6r5r-r(rKr$rrr1r3r4r.r+r) NNNzzziNFrD)NF)-__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesrecompiler[rrr)rr9r7rJrWr`rrfintrjrmrqrrrrpropertyrr rrrrrrrrboolr__static_attributes__ __classcell__)r>s@r?rr-s8 t*$&67zz(+48< "$sCx.1<  < < |/43434U99 cd3i 7#7#79.34"tCy"S" ?QUVYQZ?,7 !C!!+"c+"HSM+"]bcf]g+"Z$4$?Fd !d !t !? in M M.6tn Mae M c M MrHrrrrcT[R"S0UD6nURU5 U$)Nr%)rSentencePieceProcessorLoad)rrspms r?r2r2|s%  . . A ACHHTN JrHcz[US5n[R"XSS9 SSS5 g!,(df  g=f)Nw)indent)rjsondump)datarrs r?rrs% dCA $!$ s, :c|[US5n[R"U5sSSS5 $!,(df  g=f)Nr)rrload)rrs r?r'r's" dCAyy| s- ;)"rrrrNpathlibrshutilrtypingrrrrr r rtokenization_utilsr utilsr utils.import_utilsr get_loggerrrrr|rr)rr2rr'__all__r%rHr?rs ::5*   H % ,4   %&KM)KM'KM\ 3c3hM<`<` %#%$% CE$*-  rH