fTh?KSrSSKrSSKrSSKrSSKrSSKJrJrJrJ r SSK J r SSK J r \ R"\5rSSS S .rS rS rS r"SS\ 5rS/rg)zTokenization classes for FSMT.N)DictListOptionalTuple)PreTrainedTokenizer)loggingzvocab-src.jsonzvocab-tgt.jsonz merges.txt)src_vocab_filetgt_vocab_file merges_filecd[5nUSnUSSHnURX#45 UnM U$)zw Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rN)setadd)wordpairs prev_charchars b/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/fsmt/tokenization_fsmt.py get_pairsr$s? EEQIQR 9#$  Lc.URSS5n[R"SSU5nURSS5nURSS5nURSS5nURS S 5nURS S 5nURS S 5nURSS5nURSS5nURSS5nURSS5nURSS5nURSS5nURSS5nURSS5nURSS5nURSS5nURSS5nURS S!5nURS"S#5nURS$S%5nURS&S'5nURS(S)5nURS*S+5nURS,S-5n[R"S.SU5nURS/S05nURS1S25nURS3S45nURS5S65nURS7S85nURS9S:5nURS;S<5nURS=S>5nURS?S@5nU$)Azr Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/replace-unicode-punctuation.perl u，,u。\s*z. u、u”"u“u∶:u：u？?u《u》u）)u！!u（(u；;u１1u」u「u０0u３3u２2u５5u６6u９9u７7u８8u４4u．\s*u～~u’'u…z...u━-u〈u【[u】]u％%)replaceresub)texts rreplace_unicode_punctr71sM <<s #D 66)T4 (D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D 66)T4 (D <<s #D <<s #D <<u %D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D Krc/nUHBn[R"U5nURS5(aM1URU5 MD SR U5$)zo Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/remove-non-printing-char.perl C) unicodedatacategory startswithappendjoin)r6outputrcats rremove_non_printing_charrB\sQF""4( >>#    d  776?rc ^\rSrSrSr\rSS/rS"U4SjjrS\ \ \ 44Sjr \ S\ 4Sj5rS rS rS rS r\ S 5r\ S5rSrSrSrS#SjrSrSrSrS$S\\ S\\\ S\\ 4SjjrS%S\\ S\\\ S\S\\ 4U4Sjjjr S$S\\ S\\\ S\\ 4Sjjr!S$S\ S\\ S\"\ 4Sjjr#Sr$S r%S!r&U=r'$)& FSMTTokenizerrau Construct an FAIRSEQ Transformer tokenizer. Based on Byte-Pair Encoding. The tokenization process is the following: - Moses preprocessing and tokenization. - Normalizing all inputs text. - The arguments `special_tokens` and the function `set_special_tokens`, can be used to add additional symbols (like "__classify__") to a vocabulary. - The argument `langs` defines a pair of languages. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: langs (`List[str]`, *optional*): A list of two languages to translate from and to, for instance `["en", "ru"]`. src_vocab_file (`str`, *optional*): File containing the vocabulary for the source language. tgt_vocab_file (`st`, *optional*): File containing the vocabulary for the target language. merges_file (`str`, *optional*): File containing the merges. do_lower_case (`bool`, *optional*, defaults to `False`): Whether or not to lowercase the input when tokenizing. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc >SSKn XlX lX0lX@lXPl0Ul0Ul0Ul U(a[U5S:XaUuUl Ul O[SUS35e[USS9n [R "U 5UlSSS5 [USS9n [R "U 5nUR%5VVs0sH unnUU_M snnUlSSS5 [USS9nUR)5R+S5SS nSSS5 WVs/sHn[-UR+5SS5PM! nn[/[1U[3[U5555Ul0UlU]t"S UUUUUUUUU S . U D6 g![a [S5ef=f!,(df  GN,=fs snnf!,(df  N=f!,(df  N=fs snf) NrnYou need to install sacremoses to use XLMTokenizer. See https://pypi.org/project/sacremoses/ for installation.zFarg `langs` needs to be a list of 2 langs, e.g. ['en', 'ru'], but got zw. Usually that means that tokenizer can't find a mapping for the given model path in and other maps of this tokenizer.utf-8encoding ) langsr r r do_lower_case unk_token bos_token sep_token pad_token) sacremoses ImportErrorsmr r r rQcache_moses_punct_normalizercache_moses_tokenizercache_moses_detokenizerlensrc_langtgt_lang ValueErroropenjsonloadencoderitemsdecoderreadsplittupledictziprange bpe_rankscachesuper__init__)selfrPr r r rQrRrSrTrUkwargsrWsrc_vocab_handletgt_vocab_handle tgt_vocabkv merges_handlemergesmerge __class__s rrpFSMTTokenizer.__init__s  ,,&*-/)%'"')$ SZ1_+0 (DM4=XY^X_`88  .7 37G99%56DL4 .7 37G "23I-6__->?->TQAqD->?DL4+ 0M"'')//5cr:F18>?u% bq)*?c&%F *<=>   ))#'  M M  84 3@4 31 0?sGF!9F:'*GG ! G<#G#+&G4!F7: G  G G # G1returnc"UR5$N) get_src_vocabrqs r get_vocabFSMTTokenizer.get_vocabs!!##rcUR$r)src_vocab_sizers r vocab_sizeFSMTTokenizer.vocab_sizes"""rcX R;a'URRUS9nX0RU'URURU5$Nlang)rZrYMosesPunctNormalizer normalize)rqr6rpunct_normalizers rmoses_punct_normFSMTTokenizer.moses_punct_normsP 88 8#ww;;;F 6F - -d 3006@@FFrcX R;a'URRUS9nX0RU'URURUSSSS9$)NrTF)aggressive_dash_splits return_strescape)r[rYMosesTokenizertokenize)rqr6rmoses_tokenizers rmoses_tokenizeFSMTTokenizer.moses_tokenizes_ 11 1"gg44$4?O/> & &t ,))$/88 %9  rcX R;a'URRUS9nX0RU'URURU5$r)r\rYMosesDetokenizer detokenize)rqtokensrmoses_detokenizers rmoses_detokenizeFSMTTokenizer.moses_detokenizesP 33 3 $ 8 8d 8 C 1B ( ( .++D1<D$=$=>>rcB[UR40URD6$r)rjrfadded_tokens_decoderrs r get_tgt_vocabFSMTTokenizer.get_tgt_vocabrrc^[USS5USS-4-nUTR;aTRU$[U5nU(dUS-$[UU4SjS9nUTR;aOUupV/nSnU[ U5:aUR XX5n URX(U 5 U nX(U:Xa6U[ U5S- :a$X(S-U:XaURXV-5 US- nOURX(5 US- nU[ U5:aM[U5nUn[ U5S:XaO [U5nMSRU5nUS :XaS nUTRU'U$![a URX(S5 Mtf=f) NrOcN>TRRU[S55$)Ninf)rmgetfloat)pairrqs r#FSMTTokenizer.bpe.. s1C1CD%PU,1WrkeyrrrJ z z ) rirnrminrmr]indexextendr`r>r?) rqtokenrrbigramfirstsecondnew_wordijs ` rbpeFSMTTokenizer.bpesU3BZ E"I$6#88 DJJ ::e$ $$6> !$WXFT^^+"MEHAc$i- 5,A OOD1I.A7e#CIM(9dq5kV>SOOEN3FAOODG,FAc$i- XHD4yA~!$9:xx~ : D  5 /"OODH-sE))F  F cdURnUR(aUR5nU(aUR5nOUR XS9nUR XS9n/nUHDnU(dM UR [URU5RS555 MF U$)a Tokenize a string given language code using Moses. Details of tokenization: - [sacremoses](https://github.com/alvations/sacremoses): port of Moses - Install with `pip install sacremoses` Args: - lang: ISO language code (default = 'en') (string). Languages should belong of the model supported languages. However, we don't enforce it. - bypass_tokenizer: Allow users to preprocess and tokenize the sentences externally (default = False) (bool). If True, we only apply BPE. Returns: List of tokens. rr) r^rQlowerrhrrrlistr)rqr6rbypass_tokenizer split_tokensrs r _tokenizeFSMTTokenizer._tokenizeBs*}}   ::)>s)C$DErc~URRXRRUR55$)z0Converts a token (str) in an id using the vocab.)rdrrR)rqrs r_convert_token_to_id"FSMTTokenizer._convert_token_to_idis*||||'7'7'GHHrcLURRXR5$)z=Converts an index (integer) in a token (str) using the vocab.)rfrrR)rqrs r_convert_id_to_token"FSMTTokenizer._convert_id_to_tokenms||~~66rcUVs/sH$o"RSS5RSS5PM& nnSRU5R5nURXR5nU$s snf)z:Converts a sequence of tokens (string) in a single string.rr:r)r3r?rhrr_)rqrtr6s rconvert_tokens_to_string&FSMTTokenizer.convert_tokens_to_stringqseDJJ6a))C$,,VS96J&&($$V]]; Ks+A. token_ids_0 token_ids_1c>UR/nUcX-$X-U-U-$)a> Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A FAIRSEQ Transformer sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) sep_token_idrqrrseps r build_inputs_with_special_tokens.FSMTTokenizer.build_inputs_with_special_tokens{s6&  !  $ $ ;.44ralready_has_special_tokensc>U(a[TU]XSS9$Ub'S/[U5-S/-S/[U5--S/-$S/[U5-S/-$)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrrr)roget_special_tokens_maskr])rqrrrr{s rr%FSMTTokenizer.get_special_tokens_masksx& &72']a3   "C#k**qc1aS3{;K5KLPQsR Rc+&&1#--rcUR/nUc[X-5S/-$[X-5S/-[X#-5S/--$)a  Create a mask from the two sequences passed to be used in a sequence-pair classification task. A FAIRSEQ Transformer sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An FAIRSEQ_TRANSFORMER sequence pair mask has the following format: rr)rr]rs r$create_token_type_ids_from_sequences2FSMTTokenizer.create_token_type_ids_from_sequencessX4  !  {()QC/ /;$%+c+2C.Ds.JJJrsave_directoryfilename_prefixc [RRU5(d[R SUS35 g[RR X(aUS-OS[ S-5n[RR X(aUS-OS[ S-5n[RR X(aUS-OS[ S-5n[USS S 9nUR[R"URS S S S9S-5 SSS5 [USS S 9nURR5VVs0sHupxX_M n nnUR[R"U S S S S9S-5 SSS5 Sn [USS S 9n [URR5SS9HMupX:wa[R!SUS35 U n U RSR U 5S-5 U S- n MO SSS5 X4U4$!,(df  GN=fs snnf!,(df  N=f!,(df  N;=f)NzVocabulary path (z) should be a directoryr-r:r r r wrKrLrJTF)indent sort_keys ensure_asciirNrc US$)NrrV)kvs rr/FSMTTokenizer.save_vocabulary..s Y[\]Y^rrzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rr)ospathisdirloggererrorr?VOCAB_FILES_NAMESrawriterbdumpsrdrfresortedrmwarning)rqrrr r r frvrwrurwriter bpe_tokens token_indexs rsave_vocabularyFSMTTokenizer.save_vocabularys ww}}^,, LL,^,<?*>$!*>I? GGDJJydQVWZ^^ _9 +sW 5+1$..2F2F2HN^+_' 'NN/ }=MM(E SXXj1D89 ,`6{::'9 8@9 8 6 5s7)4H0H6H0,H6A5I H-0H66 I IcDURR5nSUS'U$)NrY)__dict__copy)rqstates r __getstate__FSMTTokenizer.__getstate__s" ""$d  rcXXlSSKnX lg![a [S5ef=f)NrrI)rrWrXrY)rqdrWs r __setstate__FSMTTokenizer.__setstate__s:     M  s))rrmrnr\rZr[rfrQrdr rYr^r r_r ) NNNNFzzzz)enFr)NF)(__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesrprstrintrpropertyrrrrrrrrrrrrrrrrrboolrrrrrr __static_attributes__ __classcell__)r{s@rrDrDrs+Z*$&67@ F$4S>$#C##G  E  !!!!??*X%NI7JN5953;DI3F5 c56sx.9.3;DI3F.ko. c..:JNK9K3;DI3FK cKB";c";HSM";]bcf]g";H  rrD)rrbrr4r;typingrrrrtokenization_utilsrutilsr get_loggerrrrrr7rBrD__all__rVrrr!sq% ..5   H %'& (V ,T'Tn  r