fThLN S r SSKrSSKrSSKrSSKJr SSKJrJrJ r J
r
J r J r SSK
rSSKJr SSKJr SSKJr \( a SS KJr SS
KJr SS KJr \R4 " \5 r S S
0r Sr \" SS9 " S S\5 5 r S/r g)z Tokenization class for model T5. N)copyfile)
TYPE_CHECKINGAnyDictListOptionalTuple )import_protobuf)PreTrainedTokenizer)
AddedToken) TextInput)logging)requires
vocab_filez spiece.modelu ▁)
sentencepiece)backendsc
^ \ rS rSrSr\rSS/r S%S\\ \
\ 4 SS4U 4S jjjr S&S jr
\S
5 r\S 5 rS r S'S
\\ S\\\ S\S\\ 4U 4S jjjrS rS rS\\ S\\ 4S jr S(S
\\ S\\\ S\\ 4S jjr S(S
\\ S\\\ S\\ 4S jjrS r S r SSS\\
4U 4S jjr \S 5 r S r S r!S r"S r#S(S!\
S"\\
S\$\
4S# jjr%S$r&U =r'$ )) T5Tokenizer, a
Construct a T5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are
accessible as "" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be
retrieved by calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids
method
additional_special_tokens (`List[str]`, *optional*):
Additional special tokens used by the tokenizer.
sp_model_kwargs (`dict`, *optional*):
Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
to set:
- `enable_sampling`: Enable subword regularization.
- `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
- `nbest_size = {0,1}`: No sampling is performed.
- `nbest_size > 1`: samples from the nbest_size results.
- `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
- `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
legacy (`bool`, *optional*):
Whether or not the `legacy` behaviour of the tokenizer should be used. Legacy is before the merge of #24622
and #25224 which includes fixes to properly handle tokens that appear after special tokens. A simple
example:
- `legacy=True`:
```python
>>> from transformers import T5Tokenizer
>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-base", legacy=True)
>>> tokenizer.encode("Hello .")
[8774, 32099, 3, 5, 1]
```
- `legacy=False`:
```python
>>> from transformers import T5Tokenizer
>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-base", legacy=False)
>>> tokenizer.encode("Hello .") # the extra space `[3]` is no longer here
[8774, 32099, 5, 1]
```
Checkout the [pull request](https://github.com/huggingface/transformers/pull/24565) for more details.
add_prefix_space (`bool`, *optional*, defaults to `False`):
Whether or not to add an initial space to the input. This allows to treat the leading word just as any
other word.
Attributes:
sp_model (`SentencePieceProcessor`):
The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
input_idsattention_maskNsp_model_kwargsreturnc
> [ U[ 5 ( a
[ USS9OUn[ U[ 5 ( a
[ USS9OUn[ U[ 5 ( a
[ USS9OUnUc 0 OUU l Xl XPl [ R " S0 U R D6U l U R R U5 Ub~ U V s/ s H n S[ U 5 ; d M U PM n n [ U 5 S: a$ U[ U5 V
s/ s H n
SU
S3PM sn
-
nOIUS: a! U[ U 5 :w a [ SU SU S 35 eO![ U5 V
s/ s H n
SU
S3PM n n
U n0 U l
[ [ U 5 5 H> n
[ SU
S3S
SSSS
S 9U R [ U R 5 S-
U- U
-
' M@ Uc% [ R S U R S
35 SnXl U R% U
R' SS
5 5 U l Xl XPl Xl [* TU ]X " SUUUUUU R UU S.U
D6 g s sn f s sn
f s sn
f )NT)specialz
r zBoth extra_ids (z!) and additional_special_tokens (zk) are provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensF) single_wordlstriprstripr
normalizedz2You are using the default legacy behaviour of the a_ . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 from_slow) eos_token unk_token pad_token extra_idsadditional_special_tokensr legacyadd_prefix_space )
isinstancestrr
r r
_extra_idsspmSentencePieceProcessorsp_modelLoadlenrange
ValueError_added_tokens_decoderlogger warning_once __class__r) get_spm_processorpopr* super__init__)selfr r$ r% r& r' r( r r) r* kwargsx extra_tokensir9 s ^/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/t5/tokenization_t5.pyr= T5Tokenizer.__init__ s~ DT^^DT UJ J
F ..vzz+u/MN
$# 0
&? 0 0 -
I \-Z Is II I Ic T [ R " S0 U R D6nU R ( d U( a UR U R
5 U$ [
U R
S5 nUR 5 n[ SU R R S35 nUR R U5 nUR 5 nSUl UR R! U5 UR# 5 nUR% U5 S S S 5 U$ ! , ( d f U$ = f)NrbzThe new behaviour of z (with `self.legacy = False`)Fr+ )r/ r0 r r) r2 r openreadr r9 __name__
ModelProto
FromStringNormalizerSpecadd_dummy_prefixnormalizer_spec MergeFromSerializeToStringLoadFromSerializedProto)r> r# tokenizerfr1 model_pb2modelrN s rC r: T5Tokenizer.get_spm_processor s ..F1E1EF ;;) NN4?? +
$//4
(AvvxH'*?@W@W?XXu(vwI ((33H=E ' 6 6 8O/4O , ! ! + +O < ..0H - -h 7 ) )
( s
,B"D
D'c U [ R ; aH [ R U nUb X!:w a U$ Uc( [ R " SU SU SU SU S3 [ 5 U$ )NzGThis tokenizer was incorrectly instantiated with a model max length of z which will be corrected in Transformers v5.
For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on z( automatically truncating your input to zM when padding/encoding.
- If you want to encode/pad to sequences longer than z you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.)r max_model_input_sizeswarningswarn
FutureWarning) pretrained_model_name_or_pathmax_model_lengthinit_max_model_lengthdeprecated_max_model_lengths rC !_eventually_correct_t5_max_length-T5Tokenizer._eventually_correct_t5_max_length s (K,M,M M*5*K*KLi*j '$05J5^,,&.
34 5 66 734 5$$?#@ Ag
g " c 6 U R R 5 $ N)r1 get_piece_sizer> s rC
vocab_sizeT5Tokenizer.vocab_size s }}++--rb c [ U R 5 Vs0 s H oR U5 U_M nnUR U R 5 U$ s snf rd )r4 rg convert_ids_to_tokensupdateadded_tokens_encoder)r> rB vocabs rC get_vocabT5Tokenizer.get_vocab sL ;@ ;QR;Qa++A.1;QR
T../ Ss A token_ids_0 token_ids_1already_has_special_tokensc > U( a [ TU ] XSS9$ Uc S/[ U5 - S/- $ S/[ U5 - S/- S/[ U5 - - S/- $ )ad
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rp rq rr r r )r< get_special_tokens_maskr3 )r> rp rq rr r9 s rC rt #T5Tokenizer.get_special_tokens_mask sw $ &72 ']a 3
C#k**qc1 1c+&&1#-!s;7G1GHA3NNrb c T [ [ [ S U R 5 5 5 $ )Nc F [ [ R " SU 5 5 S L$ )Nz)boolresearch)r@ s rC 1T5Tokenizer.get_sentinel_tokens.. s bii0A1&E!Fd!Rrb )listsetfilterr( rf s rC get_sentinel_tokens T5Tokenizer.get_sentinel_tokens s& RTXTrTrs t
rb c j U R 5 Vs/ s H oR U5 PM sn$ s snf rd )r convert_tokens_to_idsr> tokens rC get_sentinel_token_ids"T5Tokenizer.get_sentinel_token_ids s. ?C?W?W?YZ?Ye**51?YZZZs 0 token_idsc [ U5 S: a9 US U R :X a& [ R " SU R S35 U$ XR /- $ )z.Do not add eos again if user already added it.r zThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)r3 eos_token_idrY rZ r$ )r> r s rC _add_eos_if_not_present#T5Tokenizer._add_eos_if_not_present s[ y>A )B-43D3D"D MM,T^^,< =+ +
1 1 22 2rb c r U R /nUc [ X- 5 S/- $ [ X- U- U- 5 S/- $ )aw
Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
r )r r3 )r> rp rq eoss rC $create_token_type_ids_from_sequences0T5Tokenizer.create_token_type_ids_from_sequences) sL ! {()QC/ /;${2S89QC??rb c X U R U5 nUc U$ U R U5 nX- $ )a"
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:
- single sequence: `X `
- pair of sequences: `A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
)r )r> rp rq s rC build_inputs_with_special_tokens,T5Tokenizer.build_inputs_with_special_tokens? s9 & 22;? 66{CK , ,rb c D U R R 5 nS US' U$ )Nr1 )__dict__copy)r> states rC __getstate__T5Tokenizer.__getstate__Y s#
""$ j rb c Xl [ U S5 ( d 0 U l [ R " S0 U R D6U l U R
R
U R 5 g )Nr r+ )r hasattrr r/ r0 r1 r2 r )r> ds rC __setstate__T5Tokenizer.__setstate__^ sP
t .//#%D 22JT5I5IJ
4??+rb textr c T > U R ( d [ U5 S:X a [ TU ] " U40 UD6$ UR [
S5 nU R ( a [
U- n[ TU ] " U40 UD6n[ U5 S: a% US [
:X a US U R ; a USS nU$ )z
Converts a string to a list of tokens. If `self.legacy` is set to `False`, a prefix token is added unless the
first token is special.
r r N)r) r3 r< tokenizereplaceSPIECE_UNDERLINEr* all_special_tokens)r> r r? tokensr9 s rC r T5Tokenizer.tokenizeh s
;;#d)q.7#D3F3 3|| ,c2 #d*D!$1&1 v;?vay,< <dNeNeAeABZF
rb c p [ U R R [ U R 5 5 5 $ rd )r3 r1 encoder- r% rf s rC unk_token_length T5Tokenizer.unk_token_lengthz s% 4==''DNN(;<==rb c @ U R ( d UR [ S45 ( d U R R U[
S9$ U R R U R U- [
S9n[ U5 U R : a X0R S $ U$ )u
Returns a tokenized string.
We de-activated the `add_dummy_prefix` option, thus the sentencepiece internals will always strip any
SPIECE_UNDERLINE. For example: `self.sp_model.encode(f"{SPIECE_UNDERLINE}Hey", out_type = str)` will give
`['H', 'e', 'y']` instead of `['▁He', 'y']`. Thus we always encode `f"{unk_token}text"` and strip the
`unk_token`. Here is an example with `unk_token = ""` and `unk_token_length = 4`.
`self.tokenizer.sp_model.encode(" Hey", out_type = str)[4:]`.
r )out_typeN) r)
startswithr r1 r r- r% r3 r )r> r r? r s rC _tokenizeT5Tokenizer._tokenize~ s ;;doo/?.E F F==''s'; ; %%dnnt&;c%J25f+AVAV2Vv++-.b\bbrb c 8 U R R U5 $ )z0Converts a token (str) in an id using the vocab.)r1 piece_to_idr s rC _convert_token_to_id T5Tokenizer._convert_token_to_id s }}((//rb c < U R R U5 nU$ )z=Converts an index (integer) in a token (str) using the vocab.)r1 IdToPiece)r> indexr s rC _convert_id_to_token T5Tokenizer._convert_id_to_token s
''. rb c US R [ 5 ( a U R ( a US SS US' / nSnSnU HW nXPR ; a2 U( d US-
nX0R R U5 U- -
nSn/ nMD UR
U5 SnMY X0R R U5 -
nUR 5 $ )z:Converts a sequence of tokens (string) in a single string.r r N Fr T)r r r* r r1 decodeappendstrip)r> r current_sub_tokens
out_stringprev_is_specialr s rC convert_tokens_to_string$T5Tokenizer.convert_tokens_to_string s !9 0 1 1d6K6K q !"
F1I
E///& #%Jmm 2 23E F NN
"&%'""))%0"' mm**+=>>
!!rb save_directoryfilename_prefixc [ R R U5 ( d [ R SU S35 g [ R R X( a US- OS[ S - 5 n[ R R U R 5 [ R R U5 :w aG [ R R U R 5 ( a [ U R U5 U4$ [ R R U R 5 ( dC [ US5 nU R R 5 nUR U5 S S S 5 U4$ U4$ ! , ( d f U4$ = f)NzVocabulary path (z) should be a directory-r r wb)ospathisdirr7 errorjoinVOCAB_FILES_NAMESabspathr isfiler rG r1 serialized_model_protowrite)r> r r out_vocab_fileficontent_spiece_models rC save_vocabularyT5Tokenizer.save_vocabulary s, ww}}^,, LL,^,<zzd NNNT)F)NFrd )(rI
__module__ __qualname____firstlineno____doc__r vocab_files_namesmodel_input_namesr r r- r r= r: staticmethodr` propertyrg rn r intrx rt r r r r r r r r r r r r r r r __static_attributes__
__classcell__)r9 s @rC r r , s L\ *$&67
"&48 H
"$sCx.1H
H
H
V" * . . sxO 9O3;DI3FOkoO
cO O8
[ 3c 3tCy 3 JN@ 9@3;DI3F@
c@. JN- 9-3;DI3F-
c-4
,[ tCy $ > >c$0
".!c !HSM !]bcf]g ! !rb r )!r r ry rY shutilr typingr r r r r r r r/ convert_slow_tokenizerr tokenization_utilsr tokenization_utils_baser
r utilsr utils.import_utilsr
get_loggerrI r7 r r r __all__r+ rb rC r s ' B B 5 5 1 4 *
H %!>2
%&R!% R! 'R!j /rb