fTh<SrSSKrSSKrSSKrSSKJr SSKJrJrJ r J r J r SSK J r Jr SSKJr \R""\5rSS 0r"S S 5r"S S \5rS /rg)z"Tokenization class for model MyT5.N) defaultdict)DictListOptionalTupleUnion) AddedTokenPreTrainedTokenizer)logging vocab_filezbyte_maps.jsonc \rSrSrSrSrS\\\\\444Sjr S\\\\ \ \44S\S\4S jr S\\\4S \\\\ \ \444S jr S \ \S \S \ \44SjrSS\ \S \ \4SjjrSrg ) ByteRewriter!aB Byte rewriter class for MyT5 tokenizer. This class is used to rewrite bytes using a hash tree. The hash tree is constructed from a set of rewriting rules. Args: rewriting_rules (`str` or `Dict[str, str]`): A path to a json file containing the rewriting rules or a dictionary containing the rewriting rules. z[LEAF]rewriting_rulesc[U[5(a,[US5n[R"U5nSSS5 O,[U[ 5(d[ S[U535eURU5Ul UR5VVs0sHup4XC_M nnnURU5Ul g!,(df  N_=fs snnf)NrzDrewriting_rules should be either a path to json file or a dict, got ) isinstancestropenjsonloaddict ValueErrortypeconstruct_hash_tree hash_treeitemsreverse_hash_tree)selfrfkvreverse_rewriting_ruless b/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/myt5/tokenization_myt5.py__init__ByteRewriter.__init__.s os + +os+q"&))A,,+OT22VW[\kWlVmn 11/B4C4I4I4K"L4KDA144K"L!%!9!9:Q!R,+#MsB?C? C rbyte_in_sequencebyte_out_sequencecURS5nURS5nUnUHnXv;a0Xg'XgnM XVUR'g)z< Add a leaf with the output byte sequence to the hash tree.  N)splitLEAF)r rr(r) byte_in_list byte_out_list tree_pointerbs r%add_leafByteRewriter.add_leaf;sU(--c2 )//4  A$"$ '?L #0TYYreturnc[[5nS[S55HnU/X#UR'M UR 5HupEUR X$U5 M U$)z5 Construct a hash tree for rewritten byte sequences. c3(# UHoSv M g7f)02xN).0xs r% 3ByteRewriter.construct_hash_tree..Os1jsG*js)rrranger-rr2)r rrr1 in_sequence out_sequences r%r ByteRewriter.construct_hash_treeJsb % 1eCj1A'(cIL #2*9)>)>)@ %K MM), ?*Ar4 byte_sequenceNc^URnUHnX2;aX#nM g X R$)zG Search the hash tree and return the rewritten byte sequence if found. N)rr-)r rCr0r1s r%search_hash_treeByteRewriter.search_hash_treeWs6~~ A +   II&&r4in_bytescp/nSnSnU[U5:aU(d URO URn[U[U55HAnXnX;aXhnOXt:XaU/n Un O( O&URU;dM1X`Rn UnMC UR W 5 US-nU[U5:aMU$)z Rewrite a sequence of bytes using the hash tree. Args: in_bytes (`List[str]`): A list of bytes to be rewritten. reverse (`bool`): If True, decoding is performed with the reverse hash tree. Returns: `List[str]`: The rewritten byte sequence. r)lenrrr?r-extend) r rGreverse out_bytesb_startb_endr0jr1cur_leafs r% rewrite_bytesByteRewriter.rewrite_bytesds H %184>>d>T>TL7CM2K$#/?L\ !sHE99 ,+II6HE3   X &aiG!H %$r4)rr)F)__name__ __module__ __qualname____firstlineno____doc__r-rrrr&rrr2rrErR__static_attributes__r9r4r%rr!s D Sc4S>.A(B S 0$sE$S /,B'B"C 0WZ 0or 0 4S> d3PUVZ\`ad\eVePfKfFg  'd3i 'E$S /"`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. extra_ids (`int`, *optional*, defaults to 125): Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are accessible as "" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are indexed from the end of the vocabulary up to beginning ("" is the last token in the vocabulary like in ByT5 preprocessing see [here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)). additional_special_tokens (`List[str]`, *optional*): Additional special tokens used by the tokenizer. input_idsattention_maskr5c >US:a#Uc [U5Vs/sH nSUS3PM nnONUS:aHUbE[U5S:a6[[[SU555n X:wa[ SUSUS35e[ U[ 5(a [USSS 9OUn[ U[ 5(a [USSS 9OUn[ U[ 5(a [USSS 9OUnXBUS .Ul[UR5Ul S Ul [R"[US 55Ul[URS 5Ul[URS5Ul[$T U]L"SUUUSUS.UD6 gs snf)Nrz c0[S[U5;5$)Nextra_id)boolr)r;s r%(MyT5Tokenizer.__init__..sDs1v9M4Nr4zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to MyT5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensT)lstriprstrip)rrIr>r decompose_map merge_map) eos_token unk_token pad_token extra_idsadditional_special_tokensr9)r?rJsetfilterrrrr _added_tokens_decoderoffset_utf_vocab_sizerrr byte_mapsrdecompose_rewritermerge_rewritersuperr&) r r rkrlrmrnrokwargsi extra_tokens __class__s r%r&MyT5Tokenizer.__init__s q=6>DI)DT(UDTq:aS):DT %(U % ]8DMfIgjkIks6*NPi#jklL( &yk1RSlRmn(( HRR[]`GaGaJydCgp GQR[]`GaGaJydCgp GQR[]`GaGaJydCgp )2Y%O"$445 #4 C#89".t~~o/N"O*4>>++FG  &?    3)VsFcUR$N)rt)r s r% vocab_sizeMyT5Tokenizer.vocab_sizes###r4c[URUR-5Vs0sHoRU5U_M nnUR UR 5 U$s snfr)r?rrsconvert_ids_to_tokensupdateadded_tokens_encoder)r rzvocabs r% get_vocabMyT5Tokenizer.get_vocabsX;@SWS^S^A^;_`;_a++A.1;_` T../ asA token_ids_0 token_ids_1already_has_special_tokensc>U(a[TU]XSS9$UcS/[U5-S/-$S/[U5-S/-S/[U5--S/-$)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrrrI)rxget_special_tokens_maskrJ)r rrrr|s r%r%MyT5Tokenizer.get_special_tokens_masksw$ &72']a3   C#k**qc1 1c+&&1#-!s;7G1GHA3NNr4 token_idsc[U5S:a9USUR:Xa&[R"SURS35 U$XR/-$)z.Do not add eos again if user already added it.rzThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)rJ eos_token_idwarningswarnrk)r rs r%_add_eos_if_not_present%MyT5Tokenizer._add_eos_if_not_presents[ y>A )B-43D3D"D MM,T^^,<=++   1 122 2r4crUR/nUc[X-5S/-$[X-U-U-5S/-$)ay Create a mask from the two sequences passed to be used in a sequence-pair classification task. MyT5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. r)rrJ)r rreoss r%$create_token_type_ids_from_sequences2MyT5Tokenizer.create_token_type_ids_from_sequencessL   !  {()QC/ /;${2S89QC??r4cXURU5nUcU$URU5nX-$)a" Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r)r rrs r% build_inputs_with_special_tokens.MyT5Tokenizer.build_inputs_with_special_tokenss9&22;?   66{CK, ,r4textc vURS5Vs/sHo3SPM nnURU5nU$s snf)zTake as input a string and return a list of strings (tokens) for words/sub-words. Represents tokens in two character hex formatutf-8r8)encodemorphological_encode)r rryrztokenss r% _tokenizeMyT5Tokenizer._tokenize6sA'+kk'&:;&:sG*&:;**62 s6 u:?H5"~ 3Hr4c&XR- SnU$)z=Converts an index (integer) in a token (str) using the vocab.r8)rs)r indexrs r%_convert_id_to_token"MyT5Tokenizer._convert_id_to_tokenHs;;&s+ r4indicescnURRUSS9nURRUSS9nU$)NFrL)rvrRrwr rs r%r"MyT5Tokenizer.morphological_encodeMs=))777O%%33GU3Kr4cnURRUSS9nURRUSS9nU$)NTr)rwrRrvrs r%morphological_decode"MyT5Tokenizer.morphological_decodeSs=%%33GT3J))777Nr4cSn/nUHenX@R;a URURU5 M2X@R;aURU5 MTURU5 Mg URU5n[ URR 55[ UR5-nUH1nXE;aU[ US5- nMU[ RU5- nM3 URSSS9nU$)z:Converts a sequence of tokens (string) in a single string.r4rignore)errors) added_tokens_decoderappendrrrpvaluesbytesfromhexdecode)r rbstring out_tokensr _added_tokensstrings r%convert_tokens_to_string&MyT5Tokenizer.convert_tokens_to_stringYs E111!!$";";E"BC333!!%(!!%( ..z: D55<<>?#dF_F_B`` E%5005==//  9 r4save_directoryfilename_prefixc [RRU5(a6[RRX(aUS-OS[S-5nOU(aUS-OSU-n[ USSS9nUR [R"URSSS 95 SSS5 U4$!,(df  U4$=f) N-r wr)encodingrhF)indent ensure_ascii) ospathisdirjoinVOCAB_FILES_NAMESrwriterdumpsru)r rrr writers r%save_vocabularyMyT5Tokenizer.save_vocabularyps 77== ( (/3!6rUfgsUt tJ4C/C/n\J *cG 4 LLDNN15Q R5}5 4}s 70B22 C)rrrtrurvrwrs)zzz}N)r5N)NFr)rTrUrVrWrXmodel_input_namesrvocab_files_namesr&propertyrrrrrrcrrrrrrrrrrrrrrY __classcell__)r|s@r%r[r[s4%&67) "&,  , , \$$sxO9O3;DI3FOkoO cOO8 3c 3tCy 3JN@9@3;DI3F@ c@0JN-9-3;DI3F- c-4cS  DI$s) DI$s) . c HSM ]bcf]g  r4r[)rXrrr collectionsrtypingrrrrrtokenization_utilsr r utilsr get_loggerrTloggerrrr[__all__r9r4r%rsi) #55A   H %"#34ccLr'rj  r4