o
Zh' @ s d Z ddlZddlZddlZddlmZ ddlmZmZm Z ddl
m Z ddl m
Z
mZ e
r6dd lmZ ndZeeZd
d d ZG d
d de ZdgZdS )z Tokenization class for model T5. N)copyfile)ListOptionalTuple )PreTrainedTokenizerFast)is_sentencepiece_availablelogging ) T5Tokenizerz spiece.modelztokenizer.json)
vocab_filetokenizer_filec s e Zd ZU dZeZddgZeZg Z e
e e d< d fd
d Z
ed efd
dZedd Zd dedee d ee fddZ d de
e dee
e d e
e fddZ d de
e dee
e d e
e fddZdd Zd d Z ZS )!T5TokenizerFasta`
Construct a "fast" T5 tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models).
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as
"" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by
calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method
additional_special_tokens (`List[str]`, *optional*):
Additional special tokens used by the tokenizer.
add_prefix_space (`bool`, *optional*):
Whether or not the tokenizer should automatically add a prefix space
from_slow (`book`, *optional*, defaults to `False`):
Whether or not the tokenizer should be converted from a slow one. If `add_prefix_space` is set, this will be set to `True`.
Z input_idsZattention_mask
prefix_tokensNd c s |d ur3dd |D }
t |
dk r |dd t|D 7 }n!|dkr2|t |
kr2td| d| dn d d t|D }
|
}|d urKtd
d | d < t jd||||||||d
| || _|| _d S )Nc S s g | ]
}d t |v r|qS )
d s z,T5TokenizerFast.__init__..r
c S g | ]}d | dqS r >r r ir r r r f r zBoth extra_ids (z!) and additional_special_tokens (zk) are provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensc S r r r r r r r r n r zXYou set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizersTZ from_slow)r r
eos_token unk_token pad_token extra_idsadditional_special_tokensadd_prefix_spacer ) lenrange
ValueErrorloggerZ warning_oncesuper__init__r Z
_extra_ids) selfr r
r! r" r# r$ r% r& kwargsZ extra_tokens __class__r r r, V s<
zT5TokenizerFast.__init__returnc C s | j r
tj| j S dS )NF)r ospathisfiler- r r r can_save_slow_tokenizer s z'T5TokenizerFast.can_save_slow_tokenizerc C sZ | t jv r+t j| }|d ur||kr|S |d u r+td| d| d| d| d t |S )NzGThis tokenizer was incorrectly instantiated with a model max length of z which will be corrected in Transformers v5.
For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on z( automatically truncating your input to zM when padding/encoding.
- If you want to encode/pad to sequences longer than z you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.)r Zmax_model_input_sizeswarningswarn
FutureWarning)Z pretrained_model_name_or_pathZmax_model_lengthZinit_max_model_lengthZdeprecated_max_model_lengthr r r !_eventually_correct_t5_max_length s$
z1T5TokenizerFast._eventually_correct_t5_max_lengthsave_directoryfilename_prefixc C s | j stdtj|std| d d S tj||r"|d ndtd }tj | j
tj |krDt | j
| t d| |fS )NzhYour fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.zVocabulary path (z) should be a directory- r zCopy vocab file to )
r6 r) r2 r3 isdirr* errorjoinVOCAB_FILES_NAMESabspathr r info)r- r; r< Zout_vocab_filer r r save_vocabulary s z T5TokenizerFast.save_vocabulary token_ids_0 token_ids_1c C s8 || j g }|du r| j| S || j g }| j| | S )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:
- single sequence: `X `
- pair of sequences: `A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
N) eos_token_idr )r- rF rG r r r build_inputs_with_special_tokens s
z0T5TokenizerFast.build_inputs_with_special_tokensc C s< | j g}|du rt|| dg S t|| | | dg S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
Nr )rH r' )r- rF rG Zeosr r r $create_token_type_ids_from_sequences s z4T5TokenizerFast.create_token_type_ids_from_sequencesc C s t ttdd | jS )Nc S s t td| d uS )Nz)boolresearch)r r r r s z5T5TokenizerFast.get_sentinel_tokens..)listsetfilterr% r5 r r r get_sentinel_tokens s z#T5TokenizerFast.get_sentinel_tokensc s fdd D S )Nc s g | ]} |qS r )Zconvert_tokens_to_ids)r tokenr5 r r r s z:T5TokenizerFast.get_sentinel_token_ids..)rR r5 r r5 r get_sentinel_token_ids s z&T5TokenizerFast.get_sentinel_token_ids)NNr r r r NN)N) __name__
__module__ __qualname____doc__rB Zvocab_files_namesZmodel_input_namesr Zslow_tokenizer_classr r int__annotations__r, propertyrK r6 staticmethodr: r r r rE rI rJ rR rT
__classcell__r r r/ r r ) sL
&0
r )rX r2 rL r7 shutilr typingr r r Ztokenization_utils_fastr utilsr r Ztokenization_t5r Z
get_loggerrU r* rB r __all__r r r r s
E