o Zh|@sdZddlmZmZmZmZmZddlZddlZddlm Z ddl m Z m Z ddl mZddlmZmZmZdd lmZmZdd lmZmZdd lmZmZmZd d lmZee Z!Gddde j"Z#Gddde j"Z$ d5de j"dej%dej%dej%deej%de&de&fddZ'Gddde j"Z(Gddde j"Z)Gd d!d!e j"Z*Gd"d#d#e j"Z+Gd$d%d%e j"Z,Gd&d'd'e j"Z-Gd(d)d)e j"Z.Gd*d+d+e j"Z/eGd,d-d-eZ0eGd.d/d/e0Z1ed0d1Gd2d3d3e0Z2gd4Z3dS)6zPyTorch ViViT model.)CallableOptionalSetTupleUnionN)nn)CrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging torch_int) VivitConfigcs0eZdZdZfddZddefddZZS) VivitTubeletEmbeddingsa Construct Vivit Tubelet embeddings. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size[0]) * (height // tubelet_size[1]) * (width // tubelet_size[2]). cs|t|j|_|j|_|j|_|j|jd|j|jd|j|jd|_|j|_t j |j |j|j|jd|_ dS)Nrr)Z kernel_sizeZstride) super__init__ num_frames image_size tubelet_size patch_size num_patches hidden_sizeZ embed_dimrConv3d num_channels projectionselfconfig __class__W/var/www/auris/lib/python3.10/site-packages/transformers/models/vivit/modeling_vivit.pyr.s  zVivitTubeletEmbeddings.__init__Finterpolate_pos_encodingc Cs|j\}}}}}|s+||jks||jkr+td|d|d|jdd|jdd |ddddd }||}|ddd}|S) NzImage image size (*z) doesn't match model (rrz).rr )shaper ValueErrorpermuter$flatten transpose) r& pixel_valuesr, batch_sizerr#heightwidthxr*r*r+forward>s( zVivitTubeletEmbeddings.forwardF)__name__ __module__ __qualname____doc__rboolr9 __classcell__r*r*r(r+r#s rcsNeZdZdZfddZdejdededejfdd Zdd e fd d Z Z S)VivitEmbeddingsz Vivit Embeddings. Creates embeddings from a video using VivitTubeletEmbeddings, adds CLS token and positional embeddings. csttttdd|j|_t||_ ttd|j j d|j|_ t |j |_|jdd|_||_dS)Nr)rrr ParametertorchZzerosr! cls_tokenrpatch_embeddingsr position_embeddingsDropouthidden_dropout_probdropoutrrr'r%r(r*r+rVs   zVivitEmbeddings.__init__ embeddingsr6r7returnc Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||jd} ||jd} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNrg?r rZbicubicF)sizemodeZ align_cornersdim)r/rFrCZjit is_tracingrrreshaper1r functionalZ interpolateviewcat) r&rJr6r7r Z num_positionsZclass_pos_embedZpatch_pos_embedrPZ new_heightZ new_widthZsqrt_num_positionsr*r*r+r,ds(   z(VivitEmbeddings.interpolate_pos_encodingFr,c Csr|j\}}}}}|j||d}|j|ddg} tj| |fdd}|r-|||||}n||j}||}|S)Nr,rrO) r/rErDZtilerCrUr,rFrI) r&r4r,r5rr#r6r7rJZ cls_tokensr*r*r+r9s  zVivitEmbeddings.forwardr:) r;r<r=r>rrCTensorintr,r?r9r@r*r*r(r+rAOs  (rAmodulequerykeyvalueattention_maskscalingrIc Ks|t||dd|}tjj|dtjd|j}tjj |||j d}|dur,||}t||} | dd } | |fS)NrL)rPdtype)ptrainingrr) rCmatmulr3rrSZsoftmaxZfloat32torarIrc contiguous) rZr[r\r]r^r_rIkwargsZ attn_weightsZ attn_outputr*r*r+eager_attention_forwards  rhc sveZdZdeddffdd ZdejdejfddZ dd eejd e de e ejejfe ejffd d Z Z S)VivitSelfAttentionr'rKNcst|j|jdkrt|dstd|jd|jd||_|j|_t|j|j|_|j|j|_ |j |_ |jd|_ d|_ tj|j|j |jd|_tj|j|j |jd|_tj|j|j |jd|_dS) NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .gF)bias)rrr!num_attention_headshasattrr0r'rXattention_head_size all_head_sizeZattention_probs_dropout_prob dropout_probr_ is_causalrLinearZqkv_biasr[r\r]r%r(r*r+rs"   zVivitSelfAttention.__init__r8cCs6|dd|j|jf}||}|ddddS)NrLrrrr )rMrlrnrTr1)r&r8Z new_x_shaper*r*r+transpose_for_scoress z'VivitSelfAttention.transpose_for_scoresF head_maskoutput_attentionsc Cs|||}|||}|||}t}|jjdkr4|jjdkr.|r.tdnt |jj}|||||||j |j |j sCdn|j d\}} |dd|jf} || }|rc|| f} | S|f} | S)NeagerZsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rY)rqr_rIr`)rsr\r]r[rhr'Z_attn_implementationloggerZ warning_oncerrqr_rcrprMrorR) r& hidden_statesrtruZ key_layerZ value_layerZ query_layerZattention_interfaceZ context_layerZattention_probsZnew_context_layer_shapeoutputsr*r*r+r9s4     zVivitSelfAttention.forwardNF)r;r<r=rrrCrWrsrr?rrr9r@r*r*r(r+risricsFeZdZdZdeddffdd Zdejdejdejfd d ZZ S) VivitSelfOutputz The residual connection is defined in VivitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r'rKNcs.tt|j|j|_t|j|_dSN) rrrrrr!denserGrHrIr%r(r*r+r zVivitSelfOutput.__init__rx input_tensorcCs||}||}|Sr|r}rIr&rxrr*r*r+r9s  zVivitSelfOutput.forward) r;r<r=r>rrrCrWr9r@r*r*r(r+r{s$r{c s~eZdZdeddffdd ZdeeddfddZ  dd ej d e ej d e de e ej ej fe ej ffd dZZS)VivitAttentionr'rKNcs*tt||_t||_t|_dSr|)rrri attentionr{outputset pruned_headsr%r(r*r+rs    zVivitAttention.__init__headscCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrO)lenrrrlrnrrr[r\r]rr}rounion)r&rindexr*r*r+ prune_headss zVivitAttention.prune_headsFrxrtrucCs4||||}||d|}|f|dd}|S)Nrr)rr)r&rxrtruZ self_outputsattention_outputryr*r*r+r9)szVivitAttention.forwardrz)r;r<r=rrrrXrrCrWrr?rrr9r@r*r*r(r+rsrc$eZdZfddZddZZS)VivitIntermediatecsRtt|j|j|_t|j|_ t |j t r#t |j |_dS|j |_dSr|)rrrrrr!intermediate_sizer}rGrHrI isinstanceZ hidden_actstrr intermediate_act_fnr%r(r*r+r8s   zVivitIntermediate.__init__cCs"||}||}||}|Sr|)r}rrI)r&rxr*r*r+r9As   zVivitIntermediate.forwardr;r<r=rr9r@r*r*r(r+r7s  rcr) VivitOutputcs.tt|j|j|_t|j|_ dSr|) rrrrrrr!r}rGrHrIr%r(r*r+rJr~zVivitOutput.__init__cCs ||}||}||}|Sr|rrr*r*r+r9Os  zVivitOutput.forwardrr*r*r(r+rI rcs*eZdZdZfddZdddZZS) VivitLayerzNThis corresponds to the EncoderBlock class in the scenic/vivit implementation.csbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)NrZeps)rrZchunk_size_feed_forwardZ seq_len_dimrrr intermediaterrr LayerNormr!layer_norm_epslayernorm_beforelayernorm_afterr%r(r*r+r\s    zVivitLayer.__init__NFcCs`|j||||d}|d}|dd}||}||}||}|||}|f|}|S)N)rurr)rrrrr)r&rxrtruZself_attention_outputsrryZ layer_outputr*r*r+r9fs     zVivitLayer.forwardrz)r;r<r=r>rr9r@r*r*r(r+rYs  rcs.eZdZfddZ    dddZZS) VivitEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr*)r).0_r'r*r+ sz)VivitEncoder.__init__..F) rrr'rZ ModuleListrangenum_hidden_layerslayergradient_checkpointingr%r(rr+rs   zVivitEncoder.__init__NFTc Cs|rdnd}|r dnd}t|jD]8\}} |r||f}|dur$||nd} |jr6|jr6|| j|| |} n| || |} | d}|rI|| df}q|rQ||f}|s_tdd|||fDSt|||dS)Nr*rrcss|] }|dur|VqdSr|r*)rvr*r*r+ sz'VivitEncoder.forward..)last_hidden_staterx attentions) enumeraterrrcZ_gradient_checkpointing_func__call__tupler ) r&rxrtruoutput_hidden_states return_dictZall_hidden_statesZall_self_attentionsiZ layer_moduleZlayer_head_maskZ layer_outputsr*r*r+r9s6      zVivitEncoder.forward)NFFTrr*r*r(r+rs  rcr) VivitPoolercs*tt|j|j|_t|_dSr|)rrrrrr!r}ZTanh activationr%r(r*r+rs zVivitPooler.__init__cCs(|dddf}||}||}|S)Nr)r}r)r&rxZfirst_token_tensor pooled_outputr*r*r+r9s  zVivitPooler.forwardrr*r*r(r+rrrc@s0eZdZeZdZdZdZgZdZ dZ ddZ dS)VivitPreTrainedModelvivitr4TcCst|tjtjfr#|jjjd|jjd|j dur!|j j dSdSt|tj rF|jjjd|jjd|j durD|jj|j  dSdSt|tj r[|j j |jjddSt|trn|jj |jj dSdS)zInitialize the weightsrY)meanZstdNg?)rrrrr"weightdataZnormal_r'Zinitializer_rangerkZzero_Z EmbeddingZ padding_idxrZfill_rArDrF)r&rZr*r*r+ _init_weightss"       z"VivitPreTrainedModel._init_weightsN) r;r<r=rZ config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_sdpaZ_supports_flash_attn_2rr*r*r*r+rs rcseZdZdfdd ZddZddZe     dd eej d eej d ee d ee de dee de e ej e ffddZZS) VivitModelTcsXt|||_t||_t||_tj|j |j d|_ |r#t |nd|_ |dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)rrr'rArJrencoderrrr!r layernormrpooler post_init)r&r'add_pooling_layerr(r*r+rs    zVivitModel.__init__cCs|jjSr|)rJrE)r&r*r*r+get_input_embeddingsszVivitModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. Args: heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrrrr)r&Zheads_to_prunerrr*r*r+ _prune_headsszVivitModel._prune_headsNFr4rtrurr,rrKc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|||jj}|j||d}|j|||||d}|d} | | } |j durR| | nd} |s`| | f|ddSt | | |j |j dS)a Examples: ```python >>> import av >>> import numpy as np >>> from transformers import VivitImageProcessor, VivitModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitModel.from_pretrained("google/vivit-b-16x2-kinetics400") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 3137, 768] ```Nz You have to specify pixel_valuesrV)rtrurrrr)rZ pooler_outputrxr)r'ruruse_return_dictr0Z get_head_maskrrJrrrr rxr) r&r4rtrurr,rZembedding_outputZencoder_outputssequence_outputrr*r*r+r9s4T zVivitModel.forward)T)NNNNFN)r;r<r=rrrrrrC FloatTensorr?rrr r9r@r*r*r(r+rs4 ra ViViT Transformer model with a video classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for Kinetics-400. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. )Z custom_introcseZdZfddZe       ddeejdeejdeejdee d ee d e d ee d e e eje ffd dZ ZS)VivitForVideoClassificationcsRt||j|_t|dd|_|jdkrt|j|jnt|_ | dS)NF)rr) rr num_labelsrrrrrr!ZIdentity classifierrr%r(r*r+rs $ z$VivitForVideoClassification.__init__NFr4rtlabelsrurr,rrKcCs|dur|n|jj}|j||||||d}|d} || dddddf} d} |durQ|jdkrAt} | | d|d} nt} | | d|j|d} |sg| f|dd} | dure| f| S| St| | |j |j dS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import av >>> import numpy as np >>> import torch >>> from transformers import VivitImageProcessor, VivitForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=4, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitForVideoClassification.from_pretrained("google/vivit-b-16x2-kinetics400") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) LABEL_116 ```N)rtrurr,rrrrLr)losslogitsrxr) r'rrrrr rTrrrxr)r&r4rtrrurr,rryrrrZloss_fctrr*r*r+r9s6]  z#VivitForVideoClassification.forward)NNNNNFN)r;r<r=rrrrCrZ LongTensorr?rrrr9r@r*r*r(r+rxs6   r)rrr)rY)4r>typingrrrrrrCZtorch.utils.checkpointrZtorch.nnrr Z activationsr Zmodeling_outputsr r rZmodeling_utilsrrZ pytorch_utilsrrutilsrrrZconfiguration_vivitrZ get_loggerr;rwModulerrArWfloatrhrir{rrrrrrrrr__all__r*r*r*r+sd    ,W ?''3