fThg$ 0%SrSSKJrJrJr SSKJrJr SSKJ r \ R"\ 5r Sr SrSrSrS rS rS r\S \S \S\S\S\S0r\\\4\S'\R15VVs0sHupX_M snnr\\\4\S'"SS\5rS/rgs snnf)z Tokenization classes for CANINE.)DictListOptional) AddedTokenPreTrainedTokenizer)loggingiiiiiiz[CLS]z[SEP]z[BOS]z[MASK]z[PAD]z [RESERVED]SPECIAL_CODEPOINTSSPECIAL_CODEPOINTS_BY_NAMEc ^\rSrSrSr\"\5\"\5\"\5\"\5\"\5\"\ 5SS4U4Sjjr \ S\ 4Sj5r SrS \S\\4S jrS \S\ 4S jrS \ S\4SjrSrSS\\ S\\\ S\\ 4SjjrSS\\ S\\\ S\S\\ 4U4SjjjrSS\\ S\\\ S\\ 4SjjrSS\S\\4SjjrSrU=r$)CanineTokenizer:a Construct a CANINE tokenizer (i.e. a character splitter). It turns text into a sequence of characters, and then converts each character into its Unicode code point. [`CanineTokenizer`] inherits from [`PreTrainedTokenizer`]. Refer to superclass [`PreTrainedTokenizer`] for usage examples and documentation concerning parameters. Args: model_max_length (`int`, *optional*, defaults to 2048): The maximum sentence length the model accepts. Fic >[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUn0Ul[R 5HupXRU 'M URR 5V V s0sHupX_M sn n Ul[Ul[UR5Ul [T U]0"SUUUUUUUUS.U D6 gs sn n f)NF)lstriprstripT) bos_token eos_token sep_token cls_token pad_token mask_tokenadd_prefix_spacemodel_max_length) isinstancestrr_special_codepointsr items_special_codepoint_stringsUNICODE_VOCAB_SIZE_unicode_vocab_sizelen_num_special_tokenssuper__init__) selfrrrrrrrrkwargs codepointname __class__s f/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/canine/tokenization_canine.pyr%CanineTokenizer.__init__HszJTT]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir IST]_bIcIcJyuEir KUU_adJeJeZ 4Fku 46 1779OI-6 $ $T * : 483K3K3Q3Q3S; 3SIO3S; '$6 #&t'?'?#@   !--  ; sE3returncUR$N)r!)r&s r+ vocab_sizeCanineTokenizer.vocab_sizevs'''c[UR5Vs0sHn[U5U_M nnURUR5 U$s snfr/)ranger0chrupdateadded_tokens_encoder)r&ivocabs r+ get_vocabCanineTokenizer.get_vocabzsE$)$//$:;$:qQ$:; T../ U(a[TU]XSS9$S/S/[U5--S/-nUbUS/[U5-S/-- nU$)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rSrTr^r)r$get_special_tokens_maskr")r&rSrTr^r[r*s r+ra'CanineTokenizer.get_special_tokens_masksn$ &72']a3 c+../1#5  " sS--!4 4F r2cUR/nUR/n[XA-U-5S/-nUbU[X#-5S/-- nU$)a[ Create a mask from the two sequences passed to be used in a sequence-pair classification task. A CANINE sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). rr`)rVrWr"rXs r+$create_token_type_ids_from_sequences4CanineTokenizer.create_token_type_ids_from_sequencess].  !  !S&,-3  " c++,s2 2F r2save_directoryfilename_prefixcg)Nrr)r&rfrgs r+save_vocabularyCanineTokenizer.save_vocabularysr2)r#rrr!r/)NF)__name__ __module__ __qualname____firstlineno____doc__r5CLSSEPPADMASKr%propertyintr0r:rrr?rGrKrQrr\boolrardri__static_attributes__ __classcell__)r*s@r+r r :sk c(c(c(c(c(t9, \(C(( cd3i:#:#: 5# 5# 5JN93;DI3F c8sx93;DI3Fko c:JN93;DI3F c@cHSMr2r N)rotypingrrrtokenization_utilsrrutilsr get_loggerrkloggerr rrrprqBOSrsRESERVEDr rur__annotations__rr r __all__)r(r)s00r+rs'''A   H %     ( l &DcN  VhUmUmUo-pUo/)doUo-pDcNpw)wt  {.qs'B