fThK' S r SSKrSSKJrJrJr SSKJrJr SSK J
r
\
R " \ 5 r
" S S\5 rS/rg) z"Tokenization class for model ByT5. N)ListOptionalTuple )
AddedTokenPreTrainedTokenizer)loggingc
r ^ \ rS rSrSrSS/r S SU 4S jjjr\S 5 rS r S S \
\ S
\ \
\ S \
S\
\ 4U 4S jjjrS
\
\ S\
\ 4S jr S S \
\ S
\ \
\ S\
\ 4S jjr S S \
\ S
\ \
\ S\
\ 4S jjrS\S\
\ 4S jrS rS rS rS S\S\ \ S\\ 4S jjrSrU =r$ )
ByT5Tokenizer a7
Construct a ByT5 tokenizer. ByT5 simply uses raw bytes utf-8 encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 125):
Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are
accessible as "" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are
indexed from the end of the vocabulary up to beginning ("" is the last token in the vocabulary
like in ByT5 preprocessing see
[here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)).
additional_special_tokens (`List[str]`, *optional*):
Additional special tokens used by the tokenizer.
input_idsattention_maskreturnc L > US: a# Uc [ U5 Vs/ s H nSU S3PM nnONUS: aH UbE [ U5 S: a6 [ [ [ S U5 5 5 nX:w a [ SU SU S35 e[ U[ 5 ( a [ USSS 9OUn[ U[ 5 ( a [ USSS 9OUn[ U[ 5 ( a [ USSS 9OUnX1US
.U l [ U R 5 U l S U l
[ T U ]0 " S
UUUSUS .UD6 g s snf )Nr z
c 0 [ S[ U 5 ; 5 $ )Nextra_id)boolstr)xs b/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/byt5/tokenization_byt5.py(ByT5Tokenizer.__init__..L s Ds1v9M4N zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to ByT5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensT)lstriprstrip)r ) eos_token unk_token pad_token extra_idsadditional_special_tokens )
rangelensetfilter
ValueError
isinstancer r _added_tokens_decoderoffset_utf_vocab_sizesuper__init__)
selfr r! r" r# r$ kwargsi extra_tokens __class__s
r r0 ByT5Tokenizer.__init__> sT q= 6 >DI)DT(UDTq:aS):DT %(U %
] 8 DMfIgjkIk s6*NPi#j klL( &yk1RSlRm n( ( HRR[]`GaGaJydCgp GQR[]`GaGaJydCgp GQR[]`GaGaJydCgp )2Y%O"$445 #
&?
' )Vs D!c U R $ N)r. )r1 s r
vocab_sizeByT5Tokenizer.vocab_sizee s ###r c [ U R U R - 5 Vs0 s H oR U5 U_M nnUR U R
5 U$ s snf r8 )r& r9 r- convert_ids_to_tokensupdateadded_tokens_encoder)r1 r3 vocabs r get_vocabByT5Tokenizer.get_vocabi sX ;@ SWS^S^A^;_`;_a++A.1;_`
T../ as A token_ids_0 token_ids_1already_has_special_tokensc > U( a [ TU ] XSS9$ Uc S/[ U5 - S/- $ S/[ U5 - S/- S/[ U5 - - S/- $ )ad
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rB rC rD r r )r/ get_special_tokens_maskr' )r1 rB rC rD r5 s r rF %ByT5Tokenizer.get_special_tokens_maskn sw $ &72 ']a 3
C#k**qc1 1c+&&1#-!s;7G1GHA3NNr token_idsc [ U5 S: a9 US U R :X a& [ R " SU R S35 U$ XR /- $ )z.Do not add eos again if user already added it.r zThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)r' eos_token_idwarningswarnr )r1 rH s r _add_eos_if_not_present%ByT5Tokenizer._add_eos_if_not_present s[ y>A )B-43D3D"D MM,T^^,< =+ +
1 1 22 2r c r U R /nUc [ X- 5 S/- $ [ X- U- U- 5 S/- $ )ay
Create a mask from the two sequences passed to be used in a sequence-pair classification task. ByT5 does not
make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
r )rK r' )r1 rB rC eoss r $create_token_type_ids_from_sequences2ByT5Tokenizer.create_token_type_ids_from_sequences sL ! {()QC/ /;${2S89QC??r c X U R U5 nUc U$ U R U5 nX- $ )a"
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:
- single sequence: `X `
- pair of sequences: `A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
)rN )r1 rB rC s r build_inputs_with_special_tokens.ByT5Tokenizer.build_inputs_with_special_tokens s9 & 22;? 66{CK , ,r textc d UR S5 Vs/ s H n[ U5 PM nnU$ s snf )zPTake as input a string and return a list of strings (tokens) for words/sub-wordsutf-8)encodechr)r1 rW r3 tokenss r _tokenizeByT5Tokenizer._tokenize s/ "&++g"67"6Q#a&"67
8s -c \ [ U5 S:w a SnU$ [ U5 U R - nU$ )z0Converts a token (str) in an id using the vocab.r N)r' ordr- )r1 tokentoken_ids r _convert_token_to_id"ByT5Tokenizer._convert_token_to_id s4 u:?H 5zDKK/Hr c 4 [ XR -
5 nU$ )z=Converts an index (integer) in a token (str) using the vocab.)r[ r- )r1 indexra s r _convert_id_to_token"ByT5Tokenizer._convert_id_to_token s EKK'( r c SnU Hk nX0R ; a U R U R S5 nO6X0R ; a UR S5 nO[ [ U5 /5 nX$-
nMm UR SSS9nU$ )z:Converts a sequence of tokens (string) in a single string.r rY ignore)errors)added_tokens_decoderrZ r> bytesr` decode)r1 r\ bstringra
tok_stringstrings r convert_tokens_to_string&ByT5Tokenizer.convert_tokens_to_string s E111 ! 6 6u = D DW M
333 "\\' 2
"CJ< 0
!G 9
r save_directoryfilename_prefixc g)Nr% r% )r1 rt ru s r save_vocabulary ByT5Tokenizer.save_vocabulary s r )r, r. r- )zzz} N)r N)NFr8 )__name__
__module__ __qualname____firstlineno____doc__model_input_namesr0 propertyr9 r@ r intr r rF rN rR rU r r] rc rg rr r rw __static_attributes__
__classcell__)r5 s @r r r sw @ %&67 "&
%
%
%
N $ $ sxO 9O3;DI3FOkoO
cO O8 3c 3tCy 3 JN@ 9@3;DI3F@
c@. JN- 9-3;DI3F-
c-4c d3i
c HSM ]bcf]g r r )r~ rL typingr r r tokenization_utilsr r utilsr
get_loggerrz loggerr __all__r% r r r sE ) ( ( A
H %N' Nb
r