o Zh<@sdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z ddl m Z mZddlmZeeZdd iZGd d d ZGd d d eZd gZdS)z"Tokenization class for model MyT5.N) defaultdict)DictListOptionalTupleUnion) AddedTokenPreTrainedTokenizer)logging vocab_filezbyte_maps.jsonc @seZdZdZdZdeeeeefffddZdeeee e effdedefd d Z deeefd eeee e efffd d Z de ed ede effddZ dde ed e efddZdS) ByteRewriteraZ Byte rewriter class for MyT5 tokenizer. This class is used to rewrite bytes using a hash tree. The hash tree is constructed from a set of rewriting rules. Args: rewriting_rules (`str` or `Dict[str, str]`): A path to a json file containing the rewriting rules or a dictionary containing the rewriting rules. z[LEAF]rewriting_rulescCst|tr t|d }t|}Wdn1swYnt|ts.tdt||||_ dd| D}|||_ dS)NrzDrewriting_rules should be either a path to json file or a dict, got cSsi|]\}}||qSr).0kvrrY/var/www/auris/lib/python3.10/site-packages/transformers/models/myt5/tokenization_myt5.py 8sz)ByteRewriter.__init__..) isinstancestropenjsonloaddict ValueErrortypeconstruct_hash_tree hash_treeitemsreverse_hash_tree)selfrfZreverse_rewriting_rulesrrr__init__.s      zByteRewriter.__init__rbyte_in_sequencebyte_out_sequencecCsH|d}|d}|}|D]}||vri||<||}q|||j<dS)zL Add a leaf with the output byte sequence to the hash tree.  N)splitLEAF)r"rr%r&Z byte_in_listZ byte_out_list tree_pointerbrrradd_leaf;s   zByteRewriter.add_leafreturncCsTtt}ddtdDD] }|g|||j<q |D] \}}||||q|S)zE Construct a hash tree for rewritten byte sequences. css|]}|dVqdS)02xNr)rxrrr Osz3ByteRewriter.construct_hash_tree..)rrranger)r r,)r"rrr+Z in_sequenceZ out_sequencerrrrJs z ByteRewriter.construct_hash_tree byte_sequenceNcCs0|j}|D] }||vr||}qdS||jS)zW Search the hash tree and return the rewritten byte sequence if found. N)rr))r"r3r*r+rrrsearch_hash_treeWs   zByteRewriter.search_hash_treeFin_bytesc Csg}d}d}|t|krS|s|jn|j}t|t|D](}||}||vr*||}n ||kr5|g} |}nn |j|vrC||j} |}q|| |d}|t|ks |S)a6 Rewrite a sequence of bytes using the hash tree. Args: in_bytes (`List[str]`): A list of bytes to be rewritten. reverse (`bool`): If True, decoding is performed with the reverse hash tree. Returns: `List[str]`: The rewritten byte sequence. r)lenrr!r2r)extend) r"r5reverseZ out_bytesZb_startZb_endr*jr+Zcur_leafrrr rewrite_bytesds,       zByteRewriter.rewrite_bytes)F)__name__ __module__ __qualname____doc__r)rrrr$rrr,rr4r;rrrrr !s * ." r c sreZdZdZddgZeZ     d/ d0fd d Zed d Z ddZ d1de e de e e ded e e ffdd Zde e d e e fddZ d2de e de e e d e e fddZ d2de e de e e d e e fddZded e efddZd d!Zd"d#Zd$e ed e efd%d&Zd$e ed e efd'd(Zd)d*Zd2d+ed,e ed eefd-d.ZZS)3 MyT5Tokenizera Construct a MyT5 tokenizer. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): The file containing the byte rewriting rules. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. extra_ids (`int`, *optional*, defaults to 125): Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are accessible as "" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are indexed from the end of the vocabulary up to beginning ("" is the last token in the vocabulary like in ByT5 preprocessing see [here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)). additional_special_tokens (`List[str]`, *optional*): Additional special tokens used by the tokenizer. Z input_idsZattention_mask}Nr-c  s<|dkr|durddt|D}n(|dkr:|dur:t|dkr:tttdd|}||kr:td|d|dt|trFt|d d d n|}t|trTt|d d d n|}t|trbt|d d d n|}|||d |_t|j|_ d |_ t t |d |_t|jd|_t|jd|_tjd|||d|d|dS)NrcSsg|]}d|dqS)z rrirrr z*MyT5Tokenizer.__init__..cSstdt|vS)NZextra_id)boolr)r/rrrsz(MyT5Tokenizer.__init__..zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to MyT5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensT)lstriprstrip)rr6r1rZ decompose_mapZ merge_map) eos_token unk_token pad_token extra_idsadditional_special_tokensr)r2r7setfilterrrrr Z_added_tokens_decoderoffset_utf_vocab_sizerrr byte_mapsr decompose_rewritermerge_rewritersuperr$) r"r rOrPrQrRrSkwargsZ extra_tokens __class__rrr$s4   zMyT5Tokenizer.__init__cCs|jSN)rWr"rrr vocab_sizeszMyT5Tokenizer.vocab_sizecs.fddtjjD}|j|S)Ncsi|]}||qSr)Zconvert_ids_to_tokensrFr`rrrrIz+MyT5Tokenizer.get_vocab..)r2rarVupdateadded_tokens_encoder)r"Zvocabrr`r get_vocabs zMyT5Tokenizer.get_vocabF token_ids_0 token_ids_1already_has_special_tokenscsZ|r tj||ddS|durdgt|dgSdgt|dgdgt|dgS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rerfrgNrr6)r[get_special_tokens_maskr7)r"rerfrgr]rrrhs(z%MyT5Tokenizer.get_special_tokens_mask token_idscCs>t|dkr|d|jkrtd|jd|S||jgS)z.Do not add eos again if user already added it.rzThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)r7 eos_token_idwarningswarnrO)r"rirrr_add_eos_if_not_presents   z%MyT5Tokenizer._add_eos_if_not_presentcCs<|jg}|durt||dgSt||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. MyT5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)rkr7)r"rerfZeosrrr$create_token_type_ids_from_sequencessz2MyT5Tokenizer.create_token_type_ids_from_sequencescCs(||}|dur |S||}||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)rn)r"rerfrrr build_inputs_with_special_tokenss  z.MyT5Tokenizer.build_inputs_with_special_tokenstextcKs"dd|dD}||}|S)zTake as input a string and return a list of strings (tokens) for words/sub-words. Represents tokens in two character hex formatcSsg|]}|dqS)r.rrFrrrrH:sz+MyT5Tokenizer._tokenize..utf-8)encodemorphological_encode)r"rqr\tokensrrr _tokenize6s zMyT5Tokenizer._tokenizecCs(t|dkr d}|St|d|j}|S)z0Converts a token (str) in an id using the vocab.rNN)r7intrV)r"tokenZtoken_idrrr_convert_token_to_id>s z"MyT5Tokenizer._convert_token_to_idcCs||jd}|S)z=Converts an index (integer) in a token (str) using the vocab.r.)rV)r"indexryrrr_convert_id_to_tokenHsz"MyT5Tokenizer._convert_id_to_tokenindicescC$|jj|dd}|jj|dd}|S)NFr9)rYr;rZr"r}rrrrtMz"MyT5Tokenizer.morphological_encodecCr~)NTr)rZr;rYrrrrmorphological_decodeSrz"MyT5Tokenizer.morphological_decodecCsd}g}|D] }||jvr||j|q||jvr!||q||q||}t|jt|jB}|D]}||vrH|t|d7}q:|t|7}q:|jddd}|S)z:Converts a sequence of tokens (string) in a single string.rrignore)errors) Zadded_tokens_decoderappendrcrrTvaluesbytesfromhexdecode)r"rubstringZ out_tokensryZ _added_tokensstringrrrconvert_tokens_to_stringYs      z&MyT5Tokenizer.convert_tokens_to_stringsave_directoryfilename_prefixcCstj|rtj||r|dndtd}n |r|dnd|}t|ddd}|tj|j ddd Wd|fS1sBwY|fS) N-r wrr)encodingrNF)indent ensure_ascii) ospathisdirjoinVOCAB_FILES_NAMESrwriterdumpsrX)r"rrr writerrrrsave_vocabularyps  zMyT5Tokenizer.save_vocabulary)rArBrCrDN)r-N)NFr_)r<r=r>r?Zmodel_input_namesrZvocab_files_namesr$propertyrardrrxrrJrhrnrorprrvrzr|rtrrrr __classcell__rrr]rr@sb .        (r@)r?rrrl collectionsrtypingrrrrrZtokenization_utilsr r utilsr Z get_loggerr<loggerrr r@__all__rrrrs   f v