o ZhX@sdZddlmZddlZddlZddlmZddlm Z m Z m Z ddl Z ddlmZddlmZmZmZmZdd lmZmZmZmZmZmZdd lmZmZdd lm Z m!Z!m"Z"m#Z#m$Z$m%Z%d d l&m'Z'e$(e)Z*dZ+dZ,gdZ-dZ.dZ/eGddde Z0Gdddej1j2Z3Gdddej1j2Z4Gdddej1j2Z5Gdddej1j2Z6Gdddej1j2Z7Gdddej1j2Z8Gd d!d!ej1j2Z9Gd"d#d#ej1j2Z:Gd$d%d%ej1j2Z;eGd&d'd'ej1j2Zd+Z?e"d,e>Gd-d.d.e=Z@Gd/d0d0ej1j2ZAGd1d2d2ej1j2ZBGd3d4d4ej1j2ZCe"d5e>Gd6d7d7e=ZDe"d8e>Gd9d:d:e=eZEe"d;e>GdZGdS)?zTensorFlow DeiT model.) annotationsN) dataclass)OptionalTupleUnion)get_tf_activation)TFBaseModelOutputTFBaseModelOutputWithPoolingTFImageClassifierOutputTFMaskedImageModelingOutput)TFPreTrainedModelTFSequenceClassificationLossget_initializerkeraskeras_serializable unpack_inputs) shape_liststable_softmax) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) DeiTConfigrz(facebook/deit-base-distilled-patch16-224)riztabby, tabby catc@sNeZdZUdZdZded<dZded<dZded<dZded<dZ ded <dS) -TFDeiTForImageClassificationWithTeacherOutputa Output type of [`DeiTForImageClassificationWithTeacher`]. Args: logits (`tf.Tensor` of shape `(batch_size, config.num_labels)`): Prediction scores as the average of the cls_logits and distillation logits. cls_logits (`tf.Tensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token). distillation_logits (`tf.Tensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token). hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. NzOptional[tf.Tensor]logits cls_logitsdistillation_logitszTuple[tf.Tensor] | None hidden_states attentions) __name__ __module__ __qualname____doc__r__annotations__r r!r"r#r)r)X/var/www/auris/lib/python3.10/site-packages/transformers/models/deit/modeling_tf_deit.pyrCs     rcsJeZdZdZddfd d Zdd d ZdddZ  d d!ddZZS)"TFDeiTEmbeddingszv Construct the CLS token, distillation token, position and patch embeddings. Optionally, also the mask token. Fconfigruse_mask_tokenboolreturnNonec sDtjdi|||_||_t|dd|_tjj|j dd|_ dS)Npatch_embeddings)r,namedropoutr2r)) super__init__r,r-TFDeiTPatchEmbeddingsr1rlayersDropouthidden_dropout_probr3)selfr,r-kwargs __class__r)r*r6gs zTFDeiTEmbeddings.__init__NcCsV|jdd|jjftjddd|_|jdd|jjftjddd|_d|_|j r<|jdd|jjftjddd|_|j j }|jd|d|jjftjddd|_ |j rYdSd|_ t|d ddurt|j j|j dWdn1s|wYt|d ddurt|jj|jdWddS1swYdSdS) NrT cls_token)shapeZ initializerZ trainabler2distillation_token mask_tokenposition_embeddingsr1r3)Z add_weightr, hidden_sizerZ initializersZzerosr?rArBr-r1 num_patchesrDbuiltgetattrtf name_scoper2buildr3)r; input_shaperFr)r)r*rKnsN   "zTFDeiTEmbeddings.build embeddings tf.Tensorheightintwidthc CsF|jdd}|jjdd}||kr||kr|jS|jdddddf}|jdddddf}|jddddddf}|jd} ||jj} ||jj} | d| d} } t|dtt|tt|| f}tj j |t| t| fdd}tj |gdd }t|dd| f}tj tj |dd tj |dd |gdd S) NrrCrg?Zbicubic)sizemethodrrCrrpermaxis)r@rDr, patch_sizerIreshaperPmathsqrtimageresize transposeconcat expand_dims) r;rMrOrQrFZ num_positionsZclass_pos_embedZdist_pos_embedZpatch_pos_embeddimZh0Zw0r)r)r*interpolate_pos_encodings(    z)TFDeiTEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_postf.Tensor | NonetrainingrdcCs|j\}}}}||}t|\} } }|dur:t|j| | dg} tj|dd} tj| | jd} |d| | | }tj |j | dd} tj |j | dd}tj | ||fdd}|j }|rb||||}||}|j||d}|S) NrrRrXdtypeg?r)ZrepeatsrYrh)r@r1rrItilerBrbcastrjrepeatr?rArarDrdr3)r;rerfrhrd_rOrQrM batch_sizeZ seq_lengthZ mask_tokensmaskZ cls_tokensZdistillation_tokensZposition_embeddingr)r)r*calls" zTFDeiTEmbeddings.callF)r,rr-r.r/r0N)rMrNrOrPrQrPr/rN)NFF) rerNrfrgrhr.rdr.r/rN) r$r%r&r'r6rKrdrr __classcell__r)r)r=r*r+bs  'r+cs6eZdZdZdfdd Zdd d Zdd dZZS)r7z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. r,rr/r0c stjdi||j|j}}|j|j}}t|tjj r |n||f}t|tjj r-|n||f}|d|d|d|d}||_||_||_||_ t j j |||dd|_dS)Nrr projection) kernel_sizestridesr2r))r5r6 image_sizerZ num_channelsrE isinstance collectionsabcIterablerFrr8Conv2Drv)r;r,r<ryrZrzrErFr=r)r*r6s  zTFDeiTPatchEmbeddings.__init__rerNcCs^t|\}}}}tr||jkrtd||}t|\}}}}t|||||f}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.)rrIZexecuting_eagerlyrz ValueErrorrvr[)r;rerprOrQrzxr)r)r*rrs zTFDeiTPatchEmbeddings.callNcCsn|jrdSd|_t|dddur5t|jj|jddd|jgWddS1s.wYdSdS)NTrv)rGrHrIrJrvr2rKrzr;rLr)r)r*rK"zTFDeiTPatchEmbeddings.buildr,rr/r0)rerNr/rNrtr$r%r&r'r6rrrKrur)r)r=r*r7s   r7cs@eZdZdfdd Zdd d Z ddddZdddZZS)TFDeiTSelfAttentionr,rc stjd i||j|jdkrtd|jd|jd|j|_t|j|j|_|j|j|_t |j|_ t j j |jt|jdd|_t j j |jt|jdd|_t j j |jt|jdd|_t j j|jd |_||_dS) NrzThe hidden size (z6) is not a multiple of the number of attention heads ()queryunitskernel_initializerr2keyvalueZrater))r5r6rEnum_attention_headsrrPattention_head_size all_head_sizer\r]sqrt_att_head_sizerr8Denserinitializer_rangerrrr9Zattention_probs_dropout_probr3r,r;r,r<r=r)r*r6s,  zTFDeiTSelfAttention.__init__tensorrNrprPr/cCs,tj||d|j|jfd}tj|gddS)NrRrr@rrCrrrV)rIr[rrr`)r;rrpr)r)r*transpose_for_scoressz(TFDeiTSelfAttention.transpose_for_scoresFr" head_maskoutput_attentionsr.rhTuple[tf.Tensor]cCst|d}|j|d}|j|d}|j|d}|||} |||} |||} tj| | dd} tj|j| j d} t | | } t | dd}|j ||d}|durXt ||}t|| }tj|gd d }tj||d|jfd }|rz||f}|S|f}|S) NrinputsT)Z transpose_brirR)rrYrrhrrVr)rrrrrrImatmulrmrrjdividerr3multiplyr`r[r)r;r"rrrhrpZmixed_query_layerZmixed_key_layerZmixed_value_layerZ query_layerZ key_layerZ value_layerZattention_scoresZdkZattention_probsattention_outputoutputsr)r)r*rrs*            zTFDeiTSelfAttention.callNcCs|jrdSd|_t|dddur2t|jj|jdd|jjgWdn1s-wYt|dddur\t|j j|j dd|jjgWdn1sWwYt|dddurt|j j|j dd|jjgWddS1swYdSdS)NTrrr) rGrHrIrJrr2rKr,rErrrr)r)r*rKHs "zTFDeiTSelfAttention.buildr,r)rrNrprPr/rNrs r"rNrrNrr.rhr.r/rrt)r$r%r&r6rrrrKrur)r)r=r*rs   )rcs8eZdZdZdfdd Zddd dZdddZZS)TFDeiTSelfOutputz The residual connection is defined in TFDeiTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r,rc Jtjdi|tjj|jt|jdd|_tjj |j d|_ ||_ dSNdenserrr) r5r6rr8rrErrrr9r:r3r,rr=r)r*r6^  zTFDeiTSelfOutput.__init__Fr"rN input_tensorrhr.r/cCs|j|d}|j||d}|SNrrrr3r;r"rrhr)r)r*rrgs zTFDeiTSelfOutput.callNcCn|jrdSd|_t|dddur5t|jj|jdd|jjgWddS1s.wYdSdSNTr rGrHrIrJrr2rKr,rErr)r)r*rKmrzTFDeiTSelfOutput.buildrrsr"rNrrNrhr.r/rNrtrr)r)r=r*rXs  rcs>eZdZdfdd ZddZ ddddZdddZZS)TFDeiTAttentionr,rc s2tjdi|t|dd|_t|dd|_dS)N attentionr4outputr))r5r6rself_attentionr dense_outputrr=r)r*r6xszTFDeiTAttention.__init__cCtrtNotImplementedError)r;Zheadsr)r)r* prune_heads~szTFDeiTAttention.prune_headsFrrNrrr.rhr/rcCs<|j||||d}|j|d||d}|f|dd}|S)Nr"rrrhrr"rrhr)rr)r;rrrrhZ self_outputsrrr)r)r*rrs zTFDeiTAttention.callNcC|jrdSd|_t|dddur-t|jj|jdWdn1s(wYt|dddurUt|jj|jdWddS1sNwYdSdS)NTrr)rGrHrIrJrr2rKrrr)r)r*rK"zTFDeiTAttention.buildrrs) rrNrrNrr.rhr.r/rrt)r$r%r&r6rrrrKrur)r)r=r*rws  rc2eZdZd fdd Zddd Zdd d ZZS)TFDeiTIntermediater,rc sZtjdi|tjj|jt|jdd|_t |j t r$t |j |_ n|j |_ ||_dS)Nrrr))r5r6rr8rintermediate_sizerrrr{Z hidden_actstrrintermediate_act_fnr,rr=r)r*r6s  zTFDeiTIntermediate.__init__r"rNr/cCs|j|d}||}|S)Nr)rr)r;r"r)r)r*rrs  zTFDeiTIntermediate.callNcCrrrrr)r)r*rKrzTFDeiTIntermediate.buildrr"rNr/rNrtr$r%r&r6rrrKrur)r)r=r*rs rcs4eZdZdfdd Zddd d ZdddZZS) TFDeiTOutputr,rc rrrrr=r)r*r6rzTFDeiTOutput.__init__Fr"rNrrhr.r/cCs&|j|d}|j||d}||}|Srrrr)r)r*rrs zTFDeiTOutput.callNcCrr) rGrHrIrJrr2rKr,rrr)r)r*rKrzTFDeiTOutput.buildrrsrrtrr)r)r=r*rs rcs:eZdZdZdfdd Z ddddZdddZZS) TFDeiTLayerz?This corresponds to the Block class in the timm implementation.r,rc sntjdi|t|dd|_t|dd|_t|dd|_tj j |j dd|_ tj j |j dd|_ ||_dS) Nrr4 intermediaterlayernorm_beforeepsilonr2layernorm_afterr))r5r6rrrrr deit_outputrr8LayerNormalizationlayer_norm_epsrrr,rr=r)r*r6s zTFDeiTLayer.__init__Fr"rNrrr.rhr/rc Csn|j|j||d|||d}|d}||}|j||d}|j||d}|j|||d}|f|dd} | S)Nr)rrrrhr)r"rhrr)rrrrr) r;r"rrrhZattention_outputsrZ layer_outputZintermediate_outputrr)r)r*rrs zTFDeiTLayer.callNcCs|jrdSd|_t|dddur-t|jj|jdWdn1s(wYt|dddurRt|jj|jdWdn1sMwYt|dddurwt|jj|jdWdn1srwYt|dddurt|j j|j dd|j j gWdn1swYt|dddurt|j j|j dd|j j gWddS1swYdSdS)NTrrrrr) rGrHrIrJrr2rKrrrr,rErrr)r)r*rKs0"zTFDeiTLayer.buildrrsrrtrr)r)r=r*rs   rcs6eZdZdfdd Z ddddZdddZZS) TFDeiTEncoderr,rc s0tjdi|fddtjD|_dS)Ncsg|] }td|dqS)zlayer_._r4)r).0ir,r)r* sz*TFDeiTEncoder.__init__..r))r5r6rangenum_hidden_layerslayerrr=rr*r6szTFDeiTEncoder.__init__Fr"rNrrr.output_hidden_states return_dictrhr/*Union[TFBaseModelOutput, Tuple[tf.Tensor]]c Cs|rdnd}|r dnd}t|jD]"\} } |r||f}| ||| ||d} | d}|r3|| df}q|r;||f}|sItdd|||fDSt|||dS)Nr)rrrcss|] }|dur|VqdSrtr))rvr)r)r* Asz%TFDeiTEncoder.call..)last_hidden_stater"r#) enumeratertupler ) r;r"rrrrrhZall_hidden_statesZall_attentionsrZ layer_moduleZ layer_outputsr)r)r*rr!s,   zTFDeiTEncoder.callNc Csj|jrdSd|_t|dddur1|jD]}t|j |dWdn1s+wYqdSdS)NTr)rGrHrrIrJr2rK)r;rLrr)r)r*rKGs  zTFDeiTEncoder.buildrrs)r"rNrrNrr.rr.rr.rhr.r/rrtrr)r)r=r*rs  &rcsjeZdZeZ d#d$fd d Zd%d dZddZddZe        d&d'dd Z d(d!d"Z Z S))TFDeiTMainLayerTFr,radd_pooling_layerr.r-r/r0c sjtjdi|||_t||dd|_t|dd|_tjj |j dd|_ |r0t |dd|_ dSd|_ dS) NrM)r-r2encoderr4 layernormrpoolerr))r5r6r,r+rMrrrr8rrr TFDeiTPoolerrr;r,rr-r<r=r)r*r6Us  zTFDeiTMainLayer.__init__r7cCs|jjSrt)rMr1)r;r)r)r*get_input_embeddingsasz$TFDeiTMainLayer.get_input_embeddingscCr)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel r)r;Zheads_to_pruner)r)r* _prune_headsdszTFDeiTMainLayer._prune_headscCs|durtdg|jj}|Srt)rr,r)r;rr)r)r* get_head_maskkszTFDeiTMainLayer.get_head_maskNrergrfrrOptional[bool]rrrdrh:Union[TFBaseModelOutputWithPooling, Tuple[tf.Tensor, ...]]c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&tdt|d}||}|j||||d} |j | |||||d} | d} |j | |d} |j dur\|j | |dnd} |ss| durh| | fn| f} | | ddSt | | | j | jdS) Nz You have to specify pixel_valuesrU)rfrhrd)rrrrrhrrkr)rZ pooler_outputr"r#)r,rruse_return_dictrrIr`rrMrrrr r"r#)r;rerfrrrrrdrhZembedding_outputZencoder_outputssequence_output pooled_outputZ head_outputsr)r)r*rrssD   zTFDeiTMainLayer.callcCsL|jrdSd|_t|dddur-t|jj|jdWdn1s(wYt|dddurRt|jj|jdWdn1sMwYt|dddur|t|jj|jdd|j j gWdn1swwYt|dddurt|j j|j dWddS1swYdSdS)NTrMrrr) rGrHrIrJrMr2rKrrr,rErrr)r)r*rKs("zTFDeiTMainLayer.buildTFr,rrr.r-r.r/r0)r/r7NNNNNNFF)rergrfrgrrgrrrrrrrdr.rhr.r/rrt) r$r%r&r config_classr6rrrrrrrKrur)r)r=r*rQs$ =rc@seZdZdZeZdZdZdS)TFDeiTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. deitreN)r$r%r&r'rrZbase_model_prefixZmain_input_namer)r)r)r*rs raR This model is a TensorFlow [keras.layers.Layer](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer). Use it as a regular TensorFlow Module and refer to the TensorFlow documentation for all matter related to general usage and behavior. Parameters: config ([`DeiTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`tf.Tensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`DeiTImageProcessor.__call__`] for details. head_mask (`tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the pre-trained position encodings. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare DeiT Model transformer outputting raw hidden-states without any specific head on top.c sfeZdZ ddfd d Zeeeeee e d e d         d d!ddZ d"ddZ ZS)# TFDeiTModelTFr,rrr.r-r/r0c s*tj|fi|t|||dd|_dS)Nrrr-r2)r5r6rrrr=r)r*r6s zTFDeiTModel.__init__Zvision) checkpoint output_typerZmodalityexpected_outputNrergrfrrrrrrdrh*Union[Tuple, TFBaseModelOutputWithPooling]c Cs|j||||||||d} | S)N)rerfrrrrrdrh)r) r;rerfrrrrrdrhrr)r)r*rrs zTFDeiTModel.callcCsd|jrdSd|_t|dddur0t|jj|jdWddS1s)wYdSdS)NTr)rGrHrIrJrr2rKrr)r)r*rK!s"zTFDeiTModel.buildrrr)rergrfrgrrgrrrrrrrdr.rhr.r/rrt)r$r%r&r6rrDEIT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErrrKrur)r)r=r*rs,  rcr)rr,rc s<tjdi|tjj|jt|j|jdd|_ ||_ dS)Nr)rrZ activationr2r)) r5r6rr8rZpooler_output_sizerrZ pooler_actrr,rr=r)r*r6,s zTFDeiTPooler.__init__r"rNr/cCs |dddf}|j|d}|S)Nrr)r)r;r"Zfirst_token_tensorrr)r)r*rr7s zTFDeiTPooler.callNcCrrrrr)r)r*rK?rzTFDeiTPooler.buildrrrtrr)r)r=r*r+s rcs,eZdZdZd fdd Zd d d ZZS)TFDeitPixelShufflez0TF layer implementation of torch.nn.PixelShuffleupscale_factorrPr/r0c s<tjdi|t|tr|dkrtd|||_dS)NrCz1upscale_factor must be an integer value >= 2 got r))r5r6r{rPrr)r;rr<r=r)r*r6Ks zTFDeitPixelShuffle.__init__rrNcs~|}t|\}}}}|jdt|tfddtDg}tj|t||dgdd}tjj ||jdd}|S) NrCcs&g|]}tD]}||qqSr))r)rrjZblock_size_squaredZ output_depthr)r*r[s&z+TFDeitPixelShuffle.call..rrR)paramsindicesZ batch_dimsZNHWC) block_sizeZ data_format) rrrPrIZconstantrgatherrlnnZdepth_to_space)r;rr"rproZnum_input_channelsZ permutationr)rr*rrQs  zTFDeitPixelShuffle.call)rrPr/r0)rrNr/rN)r$r%r&r'r6rrrur)r)r=r*rHsrcs4eZdZdfdd Zddd d ZdddZZS) TFDeitDecoderr,rr/r0c sLtjdi|tjj|jd|jddd|_t|jdd|_ ||_ dS)NrCr0)filtersrwr21r4r)) r5r6rr8rZencoder_striderzconv2dr pixel_shuffler,rr=r)r*r6cs  zTFDeitDecoder.__init__FrrNrhr.cCs|}||}||}|Srt)rr)r;rrhr"r)r)r*rrks  zTFDeitDecoder.callNcCs|jrdSd|_t|dddur3t|jj|jddd|jjgWdn1s.wYt|dddur[t|j j|j dWddS1sTwYdSdS)NTrr) rGrHrIrJrr2rKr,rErrr)r)r*rKqs"zTFDeitDecoder.buildrrs)rrNrhr.r/rNrtrr)r)r=r*r bs r zvDeiT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://arxiv.org/abs/2111.09886).cs\eZdZdfdd Zeeeeee d       ddddZ dddZ Z S)TFDeiTForMaskedImageModelingr,rr/r0cs0t|t|dddd|_t|dd|_dS)NFTrrdecoderr4)r5r6rrr rr;r,r=r)r*r6s z%TFDeiTForMaskedImageModeling.__init__rrNFrergrfrrrrrrdr.rh)Union[tuple, TFMaskedImageModelingOutput]c  Cs|dur|n|jj}|j||||||||d} | d} | ddddf} t| \} } } t| d}}t| | ||| f} |j| |d}t|d}d}|dur|jj |jj }t|d||f}t ||jj d}t ||jj d }t |d}t |tj}tjt|d t|d }t |d}t||}t|d |jj}||}t|d }|s|f| dd}|dur|f|S|St||| j| jd S)a bool_masked_pos (`tf.Tensor` of type bool and shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, TFDeiTForMaskedImageModeling >>> import tensorflow as tf >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = TFDeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="tf").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = tf.cast(tf.random.uniform((1, num_patches), minval=0, maxval=2, dtype=tf.int32), tf.bool) >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```N)rfrrrrrdrhrrrRg?rk)rrrrCrC)rrCrrgh㈵>)r)lossZreconstructionr"r#)r,rrrrPrIr[rr`ryrZrnrbrmZfloat32rZlossesZmean_absolute_errorZ reduce_sumrzr r"r#)r;rerfrrrrrdrhrrrpZsequence_lengthrzrOrQZreconstructed_pixel_valuesZmasked_im_lossrSrqZreconstruction_lossZ total_lossZnum_masked_pixelsrr)r)r*rrsX+       z!TFDeiTForMaskedImageModeling.callcCr)NTrr)rGrHrIrJrr2rKrrr)r)r*rKrz"TFDeiTForMaskedImageModeling.buildrr)rergrfrgrrgrrrrrrrdr.rhr.r/rrt) r$r%r&r6rrrrr rrrrKrur)r)r=r*r}s crz DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. cs\eZdZdfdd Zeeeeee d        ddddZ dddZ Z S)TFDeiTForImageClassificationr,rcs\t||j|_t|ddd|_|jdkr tjj|jddntjjddd|_ ||_ dS)NFrrr2r classifierr4linear) r5r6 num_labelsrrrr8r Activationrr,rr=r)r*r6s   z%TFDeiTForImageClassification.__init__rNFrergrlabelsrrrrrdr.rhr/)Union[tf.Tensor, TFImageClassifierOutput]c  Cs|dur|n|jj}|j|||||||d} | d} || dddddf} |dur.dn||| } |sJ| f| dd} | durH| f| S| St| | | j| jdS)a labels (`tf.Tensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, TFDeiTForImageClassification >>> import tensorflow as tf >>> from PIL import Image >>> import requests >>> keras.utils.set_random_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # note: we are loading a TFDeiTForImageClassificationWithTeacher from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = TFDeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> inputs = image_processor(images=image, return_tensors="tf") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = tf.math.argmax(logits, axis=-1)[0] >>> print("Predicted class:", model.config.id2label[int(predicted_class_idx)]) Predicted class: little blue heron, Egretta caerulea ```Nrrrrrdrhrr)rrr"r#)r,rrrZhf_compute_lossr r"r#)r;rerrrrrrdrhrrrrrr)r)r*rrs,/ z!TFDeiTForImageClassification.callcCs|jrdSd|_t|dddur-t|jj|jdWdn1s(wYt|dddurZt|jj|jdd|jj gWddS1sSwYdSdS)NTrr) rGrHrIrJrr2rKrr,rErr)r)r*rK^s"z"TFDeiTForImageClassification.buildrr)rergrrgrrgrrrrrrrdr.rhr.r/rrt) r$r%r&r6rrrrr rrrrKrur)r)r=r*rs Jra DeiT Model transformer with image classification heads on top (a linear layer on top of the final hidden state of the [CLS] token and a linear layer on top of the final hidden state of the distillation token) e.g. for ImageNet. .. warning:: This model supports inference-only. Fine-tuning with distillation (i.e. with a teacher) is not yet supported. cs^eZdZdfdd Zeeeeee e e d      ddddZ dddZ ZS)'TFDeiTForImageClassificationWithTeacherr,rr/r0cst||j|_t|ddd|_|jdkr tjj|jddntjjddd|_ |jdkr7tjj|jddntjjddd|_ ||_ dS) NFrrrcls_classifierr4rdistillation_classifier) r5r6rrrrr8rrr!r"r,rr=r)r*r6ws    z0TFDeiTForImageClassificationWithTeacher.__init__)rrrrNFrergrrrrrrdr.rh;Union[tuple, TFDeiTForImageClassificationWithTeacherOutput]c Cs|dur|n|jj}|j|||||||d}|d} || dddddf} || dddddf} | | d} |sK| | | f|dd} | St| | | |j|jdS)NrrrrC)rr r!r"r#)r,rrr!r"rr"r#)r;rerrrrrdrhrrr r!rrr)r)r*rrs0  z,TFDeiTForImageClassificationWithTeacher.callcCs |jrdSd|_t|dddur-t|jj|jdWdn1s(wYt|dddurWt|jj|jdd|jj gWdn1sRwYt|dddurt|j j|j dd|jj gWddS1s}wYdSdS)NTrr!r") rGrHrIrJrr2rKr!r,rEr"rr)r)r*rKs "z-TFDeiTForImageClassificationWithTeacher.buildr)NNNNNFF)rergrrgrrrrrrrdr.rhr.r/r#rt)r$r%r&r6rrrr_IMAGE_CLASS_CHECKPOINTrr_IMAGE_CLASS_EXPECTED_OUTPUTrrrKrur)r)r=r*r js& *r )rr rrr)Hr' __future__rcollections.abcr|r\ dataclassesrtypingrrrZ tensorflowrIZactivations_tfrZmodeling_tf_outputsr r r r Zmodeling_tf_utilsr rrrrrZtf_utilsrrutilsrrrrrrZconfiguration_deitrZ get_loggerr$loggerrrrr$r%rr8ZLayerr+r7rrrrrrrrrZDEIT_START_DOCSTRINGrrrrr rrr __all__r)r)r)r*sv       m.[(D6r  4yh  U