o
Zhu @ s\ d Z ddlmZmZmZmZ ddlmZmZ ddl m
Z
e
e Z
G dd deZdgZdS ) z!Tokenization class for Perceiver. )DictListOptionalTuple )
AddedTokenPreTrainedTokenizer)loggingc
s e Zd ZdZddgZ
d( d) fd
d
Zd eeef fddZ e
dd Z d*de e de
e e ded e e f fdd
Z d+de e de
e e d e e fddZded e e fd d Zd d Zd d! Zd"d# Zd+d$ed%e
e d ee fd&d'Z ZS ),PerceiverTokenizeraS
Construct a Perceiver tokenizer. The Perceiver simply uses raw bytes utf-8 encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
pad_token (`str`, *optional*, defaults to `"[PAD]"`):
The token used for padding, for example when batching sequences of different lengths.
bos_token (`str`, *optional*, defaults to `"[BOS]"`):
The BOS token (reserved in the vocab, but not actually used).
eos_token (`str`, *optional*, defaults to `"[EOS]"`):
The end of sequence token (reserved in the vocab, but not actually used).
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
mask_token (`str`, *optional*, defaults to `"[MASK]"`):
The MASK token, useful for masked language modeling.
cls_token (`str`, *optional*, defaults to `"[CLS]"`):
The CLS token (reserved in the vocab, but not actually used).
sep_token (`str`, *optional*, defaults to `"[SEP]"`):
The separator token, which is used when building a sequence from two sequences.
Z input_idsZattention_mask[PAD][BOS][EOS][MASK][CLS][SEP] returnNc
s t |tr t|dddn|}t |trt|dddn|}t |tr(t|dddn|}t |tr6t|dddn|}t |trDt|dddn|}t |trRt|dddn|}d| _||||||d| _t| j| _t jd|||||||d| d S )NF)lstriprstrip )r r ) pad_token bos_token eos_token
mask_token cls_token sep_tokenmodel_max_length )
isinstancestrr _utf_vocab_sizeZ_added_tokens_decoderlen_num_special_tokenssuper__init__) selfr r r r r r r kwargs __class__r! c/var/www/auris/lib/python3.10/site-packages/transformers/models/perceiver/tokenization_perceiver.pyr( ; s4
zPerceiverTokenizer.__init__c C s: i }t | jD ]
}t|}|| j ||< q|| j |S N)ranger$ chrr& updateadded_tokens_encoder)r) Zvocabitokenr! r! r- get_vocabd s z PerceiverTokenizer.get_vocabc C s | j S r. )r$ )r) r! r! r-
vocab_sizel s z PerceiverTokenizer.vocab_sizeF token_ids_0 token_ids_1already_has_special_tokensc sf |r t j||ddS |du r dgdgt| dg S dgdgt| dg dgt| dg S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)r7 r8 r9 Nr r )r' get_special_tokens_maskr% )r) r7 r8 r9 r+ r! r- r: p s .z*PerceiverTokenizer.get_special_tokens_maskc C s< |du r| j g| | jg S | j g| | jg | | jg S )af
Build model inputs from a sequence or a pair of sequence for sequence classification tasks. A sequence has the
following format:
- single sequence: `[CLS] X [SEP]`
- pair of sequences: `[CLS] A [SEP] B [SEP]`
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
N)Z cls_token_idZ sep_token_id)r) r7 r8 r! r! r- build_inputs_with_special_tokens s z3PerceiverTokenizer.build_inputs_with_special_tokenstextc C s dd | dD }|S )zPTake as input a string and return a list of strings (tokens) for words/sub-wordsc S s g | ]}t |qS r! )r0 ).0r3 r! r! r-
s z0PerceiverTokenizer._tokenize..utf-8)encode)r) r< tokensr! r! r- _tokenize s z PerceiverTokenizer._tokenizec C s( t |dkr | j}|S t|| j }|S )z0Converts a token (str) in an id using the vocab.r )r% Z unk_token_idordr& )r) r4 Ztoken_idr! r! r- _convert_token_to_id s
z'PerceiverTokenizer._convert_token_to_idc C s t || j }|S )z=Converts an index (integer) in a token (str) using the vocab.)r0 r& )r) indexr4 r! r! r- _convert_id_to_token s z'PerceiverTokenizer._convert_id_to_tokenc C sP d}|D ]}|| j v rt|d}ntt|g}||7 }q|jddd}|S )z:Converts a sequence of tokens (string) in a single string. r? replace)errors)r2 r# r@ bytesrC decode)r) rA bstringr4 Z
tok_stringstringr! r! r- convert_tokens_to_string s
z+PerceiverTokenizer.convert_tokens_to_stringsave_directoryfilename_prefixc C s dS )Nr! r! )r) rO rP r! r! r- save_vocabulary s z"PerceiverTokenizer.save_vocabulary)r r r
r r r r )r N)NFr. )__name__
__module__ __qualname____doc__Zmodel_input_namesr( r r# intr5 propertyr6 r r boolr: r; rB rD rF rN r rQ
__classcell__r! r! r+ r- r
sN
)
(
r
N)rU typingr r r r Ztokenization_utilsr r utilsr Z
get_loggerrR loggerr
__all__r! r! r! r- s
/