fThu | S r SSKJrJrJrJr SSKJrJr SSK J
r
\
R " \ 5 r
" S S\5 rS/rg) z!Tokenization class for Perceiver. )DictListOptionalTuple )
AddedTokenPreTrainedTokenizer)loggingc
6 ^ \ rS rSrSrSS/r S SU 4S jjjrS\\\ 4 4S jr
\ S 5 r SS \
\ S
\\
\ S \S\
\ 4U 4S jjjr SS \
\ S
\\
\ S\
\ 4S
jjrS\S\
\ 4S jrS rS rS rSS\S\\ S\\ 4S jjrSrU =r$ )PerceiverTokenizer a
Construct a Perceiver tokenizer. The Perceiver simply uses raw bytes utf-8 encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
pad_token (`str`, *optional*, defaults to `"[PAD]"`):
The token used for padding, for example when batching sequences of different lengths.
bos_token (`str`, *optional*, defaults to `"[BOS]"`):
The BOS token (reserved in the vocab, but not actually used).
eos_token (`str`, *optional*, defaults to `"[EOS]"`):
The end of sequence token (reserved in the vocab, but not actually used).
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
mask_token (`str`, *optional*, defaults to `"[MASK]"`):
The MASK token, useful for masked language modeling.
cls_token (`str`, *optional*, defaults to `"[CLS]"`):
The CLS token (reserved in the vocab, but not actually used).
sep_token (`str`, *optional*, defaults to `"[SEP]"`):
The separator token, which is used when building a sequence from two sequences.
input_idsattention_maskreturnc , > [ U[ 5 ( a [ USSS9OUn[ U[ 5 ( a [ USSS9OUn[ U[ 5 ( a [ USSS9OUn[ U[ 5 ( a [ USSS9OUn[ U[ 5 ( a [ USSS9OUn[ U[ 5 ( a [ USSS9OUnSU l UUUUUUS.U l [ U R 5 U l [ T U ] " SUUUUUUUS.UD6 g )NF)lstriprstrip )r r ) pad_token bos_token eos_token
mask_token cls_token sep_tokenmodel_max_length )
isinstancestrr _utf_vocab_size_added_tokens_decoderlen_num_special_tokenssuper__init__)
selfr r r r r r r kwargs __class__s
l/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/perceiver/tokenization_perceiver.pyr( PerceiverTokenizer.__init__; s- JTT]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir KUV`beKfKfZ
5Glv
IST]_bIcIcJyuEir IST]_bIcIcJyuEir #
6
" $'t'A'A#B
! -
c 0 n[ U R 5 H n[ U5 nX R - X' M UR U R
5 U$ N)ranger# chrr& updateadded_tokens_encoder)r) vocabitokens r, get_vocab PerceiverTokenizer.get_vocabd sN t++,AFE 7 77EL - T../ r. c U R $ r0 )r# )r) s r,
vocab_size PerceiverTokenizer.vocab_sizel s ###r. token_ids_0 token_ids_1already_has_special_tokensc > U( a [ TU ] XSS9$ Uc S/S/[ U5 - - S/- $ S/S/[ U5 - - S/- S/[ U5 - - S/- $ )ad
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)r= r> r? r r )r' get_special_tokens_maskr% )r) r= r> r? r+ s r, rA *PerceiverTokenizer.get_special_tokens_maskp s $ &72 ']a 3
3!s; ///1#5 5sqcC ,,-3sS=M7MNRSQTTTr. c Uc U R /U- U R /- $ U R /U- U R /- U- U R /- $ )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks. A sequence has the
following format:
- single sequence: `[CLS] X [SEP]`
- pair of sequences: `[CLS] A [SEP] B [SEP]`
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) cls_token_id sep_token_id)r) r= r> s r, build_inputs_with_special_tokens3PerceiverTokenizer.build_inputs_with_special_tokens sb & %%&48I8I7JJ J%%&48I8I7JJ[X\`\m\m[nn nr. textc d UR S5 Vs/ s H n[ U5 PM nnU$ s snf )zPTake as input a string and return a list of strings (tokens) for words/sub-wordsutf-8)encoder2 )r) rH r6 tokenss r, _tokenize PerceiverTokenizer._tokenize s/ "&++g"67"6Q#a&"67
8s -c p [ U5 S:w a U R nU$ [ U5 U R - nU$ )z0Converts a token (str) in an id using the vocab.r )r% unk_token_idordr& )r) r7 token_ids r, _convert_token_to_id'PerceiverTokenizer._convert_token_to_id s: u:?((H 5zD$<$< $ $ sxU 9U3;DI3FUkoU
cU U: JNo 9o3;DI3Fo
co0c d3i
c HSM ]bcf]g r. r N)rl typingr r r r tokenization_utilsr r utilsr
get_loggerrh loggerr __all__r r. r, ry sB ( . . A
H %k, k\
r.