fTh|~SrSSKJrJrJrJrJr SSKrSSKrSSKJ r SSK J r J r SSK Jr SSKJrJrJr SS KJrJr SS KJrJr SS KJrJrJr S S KJr \R>"\ 5r!"SS\ RD5r#"SS\ RD5r$S3S\ RDS\RJS\RJS\RJS\\RJS\&S\&4Sjjr'"SS\ RD5r("SS\ RD5r)"SS\ RD5r*"S S!\ RD5r+"S"S#\ RD5r,"S$S%\ RD5r-"S&S'\ RD5r."S(S)\ RD5r/\"S*S+\55r0\"S,S-\055r1\"S.S/9"S0S1\055r2/S2Qr3g)4zPyTorch ViViT model.)CallableOptionalSetTupleUnionN)nn)CrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging torch_int) VivitConfigc>^\rSrSrSrU4SjrSS\4SjjrSrU=r $)VivitTubeletEmbeddings#az Construct Vivit Tubelet embeddings. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size[0]) * (height // tubelet_size[1]) * (width // tubelet_size[2]). c>[TU]5 URUlURUlURUlURUR S-URUR S--URUR S--UlURUl[R"URURURURS9Ul g)Nrr) kernel_sizestride) super__init__ num_frames image_size tubelet_size patch_size num_patches hidden_size embed_dimrConv3d num_channels projectionselfconfig __class__s `/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/vivit/modeling_vivit.pyr!VivitTubeletEmbeddings.__init__.s  ++ ++ -- __ 2 2$//!"44 6$//!"44 6   ++))   !3!3ATAT]c]p]p interpolate_pos_encodingc bURup4pVnU(dPX`R:wdXpR:wa2[SUSUSURSSURSS3 5eURSSSSS 5nUR U5nUR S5R SS5nU$) NzImage image size (*z) doesn't match model (rrz).rr )shaper# ValueErrorpermuter+flatten transpose) r- pixel_valuesr3 batch_sizer"r*heightwidthxs r0forwardVivitTubeletEmbeddings.forward>s>J>P>P; e'V-F%SbSbJb$VHAeW4KDOO\]L^K__`aeapapqras`ttvw  $++Aq!Q: OOL ) IIaL " "1a (r2)r(r#r"r&r%r+F) __name__ __module__ __qualname____firstlineno____doc__r!boolrA__static_attributes__ __classcell__r/s@r0rr#s dr2rc^\rSrSrSrU4SjrS\RS\S\S\R4Sjr S S \ 4S jjr S r U=r $) VivitEmbeddingsOz| Vivit Embeddings. Creates embeddings from a video using VivitTubeletEmbeddings, adds CLS token and positional embeddings. c>[TU]5 [R"[R "SSUR 55Ul[U5Ul [R"[R "SURRS-UR 55Ul [R"UR5UlURSSUlXlg)Nr)r r!r Parametertorchzerosr' cls_tokenrpatch_embeddingsr&position_embeddingsDropouthidden_dropout_probdropoutr$r%r.r,s r0r!VivitEmbeddings.__init__Vs ekk!Q8J8J&KL 6v >#%<< KK400<r?returncURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R S-n X0R S-n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNrg?r rbicubicF)sizemode align_cornersdim)r7rVrRjit is_tracingr%rreshaper9r functional interpolateviewcat) r-r[r>r?r& num_positionsclass_pos_embedpatch_pos_embedrd new_height new_widthsqrt_num_positionss r0r3(VivitEmbeddings.interpolate_pos_encodingds]!&&q)A- 0066q9A= yy##%%+*F6?++ +221bqb59221ab59r"q11 __Q// &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr2r3c"URup4pVnURXS9nURRUSS/5n [R "X4SS9nU(aXR XU5-nOXR-nURU5nU$)Nr3rrc) r7rUrTtilerRrkr3rVrY) r-r<r3r=r"r*r>r?r[ cls_tokenss r0rAVivitEmbeddings.forwards>J>P>P; e**<*k ^^((*a);< YY 7Q?  $#&C&CJX]&^^J#&>&>>J\\*- r2)rTr.rYrUr%rVrC)rDrErFrGrHr!rRTensorintr3rIrArJrKrLs@r0rNrNOsQ &D5<<&D&DUX&D]b]i]i&DPdr2rNmodulequerykeyvalueattention_maskscalingrYc [R"XRSS55U-n[RR US[R S9RUR5n[RRXURS9nUbX-n[R"X5n U RSS5R5n X4$)Nr^)rddtype)ptrainingrr) rRmatmulr;rrhsoftmaxfloat32torrYr contiguous) rzr{r|r}r~rrYkwargs attn_weights attn_outputs r0eager_attention_forwardrs<<}}R'<=GL==((2U]](SVVW\WbWbcL==((6??([L!#4 ,,|3K''1-88:K  $$r2c ^\rSrSrS\SS4U4SjjrS\RS\R4SjrS S\ \RS \ S\ \ \R\R4\ \R44S jjr S rU=r$) VivitSelfAttentionr.r\Nc0>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul URUl URS-Ul SUl [R"URURUR S9Ul[R"URURUR S9Ul[R"URURUR S9Ulg) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .gF)bias)r r!r'num_attention_headshasattrr8r.ryattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalrLinearqkv_biasr{r|r}r,s r0r!VivitSelfAttention.__init__sG     : : :a ?PVXhHiHi"6#5#5"67334A7   #)#=#= #&v'9'9FO##L '>dkk>^>^&_#)<     nnLL#}}C$2C2C * & #0"4"4"6s";t?Q?Q>S"S%--.EF 6G=2O\M]r2) rrr.rrr|rr{rr}NF)rDrErFrGrr!rRrxrrrIrrrArJrKrLs@r0rrs]{]t](%ell%u||% bg!(0(>!Z^! uU\\5<</0% 2EE F!!r2rc^\rSrSrSrS\SS4U4SjjrS\RS\RS\R4S jr S r U=r $) VivitSelfOutputz The residual connection is defined in VivitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r.r\Nc>[TU]5 [R"URUR5Ul[R "UR5UlgN) r r!rrr'denserWrXrYr,s r0r!VivitSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r2r input_tensorcJURU5nURU5nU$rrrYr-rrs r0rAVivitSelfOutput.forwards$ =1  ]3 r2r) rDrErFrGrHrr!rRrxrArJrKrLs@r0rrsI >{>t> U\\RWR^R^r2rc ^\rSrSrS\SS4U4SjjrS\\SS4SjrS S\ RS \ \ RS \ S\ \\ R\ R4\\ R44S jjrS rU=r$)VivitAttentionir.r\Nc>[TU]5 [U5Ul[ U5Ul[ 5Ulgr)r r!r attentionroutputset pruned_headsr,s r0r!VivitAttention.__init__s0 +F3%f- Er2headsc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrrrc)lenrrrrrrr{r|r}rrrunion)r-rindexs r0 prune_headsVivitAttention.prune_headss u:? 7 >>55t~~7Y7Y[_[l[l  2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r2rrrcfURXU5nURUSU5nU4USS-nU$)Nrr)rr)r-rrr self_outputsattention_outputrs r0rAVivitAttention.forward)sC ~~m@QR ;;|A F#% QR(88r2)rrrr)rDrErFrGrr!rryrrRrxrrIrrrArJrKrLs@r0rrs"{"t" ;S;d;*-1"' || ELL)   uU\\5<</0% 2EE F  r2rc.^\rSrSrU4SjrSrSrU=r$)VivitIntermediatei7c^>[TU]5 [R"URUR 5Ul[R"UR5Ul [UR[5(a[URUlgURUlgr)r r!rrr'intermediate_sizerrWrXrY isinstance hidden_actstrr intermediate_act_fnr,s r0r!VivitIntermediate.__init__8sv YYv1163K3KL zz&"<"<= f'' - -'-f.?.?'@D $'-'8'8D $r2clURU5nURU5nURU5nU$r)rrrY)r-rs r0rAVivitIntermediate.forwardAs4 =1 00?  ]3 r2)rrYrrDrErFrGr!rArJrKrLs@r0rr7s9r2rc.^\rSrSrU4SjrSrSrU=r$) VivitOutputiIc>[TU]5 [R"URUR 5Ul[R"UR5Ul gr) r r!rrrr'rrWrXrYr,s r0r!VivitOutput.__init__JsB YYv779K9KL zz&"<"<= r2cRURU5nURU5nX-nU$rrrs r0rAVivitOutput.forwardOs, =1  ]3 %4 r2rrrLs@r0rrIs> r2rc6^\rSrSrSrU4SjrSSjrSrU=r$) VivitLayeriYzNThis corresponds to the EncoderBlock class in the scenic/vivit implementation.cj>[TU]5 URUlSUl[ U5Ul[ U5Ul[U5Ul [R"URURS9Ul[R"URURS9Ulg)Nreps)r r!chunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormr'layer_norm_epslayernorm_beforelayernorm_afterr,s r0r!VivitLayer.__init__\s '-'E'E$'/-f5!&) " V-?-?VEZEZ [!||F,>,>FDYDYZr2cURURU5UUS9nUSnUSSnXQ-nURU5nURU5nUR Xq5nU4U-nU$)N)rrr)rrrrr)r-rrrself_attention_outputsrr layer_outputs r0rAVivitLayer.forwardfs!%  ! !- 0 / "0"  2!4(,)8 ++M: ((6 {{<? /G+r2)rrrrrrrr) rDrErFrGrHr!rArJrKrLs@r0rrYsX[r2rc:^\rSrSrU4SjrSSjrSrU=r$) VivitEncoderic>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfr) r r!r.r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r-r._r/s r0r!VivitEncoder.__init__sR  ]]fF^F^@_#`@_1Jv$6@_#`a &+#$asA&cU(aSOSnU(aSOSn[UR5HzupU(aXa4-nUbX(OSn UR(a0UR(aUR U R UU U5n O U "XU5n U SnU(dMrX{S4-nM| U(aXa4-nU(d[ SXU455$[UUUS9$)Nrrc3.# UH ocMUv M g7frr).0vs r0 'VivitEncoder.forward..sm$[q$[s )last_hidden_stater attentions) enumeraterrr_gradient_checkpointing_func__call__tupler ) r-rrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r0rAVivitEncoder.forwards#7BD$5b4(4OA#$58H$H!.7.CilO**t}} $ A A ))!#% ! !-]M^ _ )!,M  &91=M[TU]5 [R"URUR5Ul[R "5Ulgr)r r!rrr'rTanh activationr,s r0r!VivitPooler.__init__s9 YYv1163E3EF '')r2c\USS2S4nURU5nURU5nU$)Nr)rr)r-rfirst_token_tensor pooled_outputs r0rAVivitPooler.forwards6+1a40 #56  6 r2)rrrrLs@r0rrs$ r2rc6\rSrSr\rSrSrSr/r Sr Sr Sr Sr g)VivitPreTrainedModelivivitr<Tc[U[R[R45(akURR R SURRS9 URb%URR R5 gg[U[R5(axURR R SURRS9 URb2URR URR5 gg[U[R5(aJURR R5 URR RS5 g[U[5(aIUR R R5 UR"R R5 gg)zInitialize the weightsr)meanstdNg?)rrrr)weightdatanormal_r.initializer_rangerzero_ Embedding padding_idxrfill_rNrTrV)r-rzs r0 _init_weights"VivitPreTrainedModel._init_weightssU fryy"))4 5 5 MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S )  0 0    ! ! ' ' )  & & + + 1 1 31r2rN)rDrErFrGr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_2r1rJrr2r0r$r$s.L$O&*#N!4r2r$c^\rSrSrSU4SjjrSrSr\SS\\ RS\\ RS\\ S\\ S \ S \\ S \ \ \ R\44S jj5rS rU=r$) VivitModelic>[TU]U5 Xl[U5Ul[ U5Ul[R"URURS9Ul U(a [U5OSUl UR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r r!r.rNr[rencoderrrr'r layernormrpooler post_init)r-r.add_pooling_layerr/s r0r!VivitModel.__init__si   )&1#F+ f&8&8f>S>ST->k&)D  r2c.URR$r)r[rU)r-s r0get_input_embeddingsVivitModel.get_input_embeddingss///r2cUR5H7up#URRURR U5 M9 g)zy Prunes heads of the model. Args: heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsr=rrr)r-heads_to_prunerrs r0 _prune_headsVivitModel._prune_headss<+002LE LL  u % / / ; ;E B3r2r<rrrr3rr\cUbUOURRnUbUOURRnUbUOURRnUc [ S5eUR X RR 5nURXS9nURUUUUUS9nUSn URU 5n URbURU 5OSn U(d X4USS-$[U U URURS9$)a( Examples: ```python >>> import av >>> import numpy as np >>> from transformers import VivitImageProcessor, VivitModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitModel.from_pretrained("google/vivit-b-16x2-kinetics400") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 3137, 768] ```Nz You have to specify pixel_valuesrt)rrrrrr)r  pooler_outputrr )r.rruse_return_dictr8 get_head_maskrr[r=r>r?rrr ) r-r<rrrr3rembedding_outputencoder_outputssequence_outputr!s r0rAVivitModel.forwards#h2C1N-TXT_T_TqTq$8$D $++JjJj &1%r?)T)NNNNFN)rDrErFrGr!rDrIrrrR FloatTensorrIrrrrArJrKrLs@r0r;r;s"0 C5915,0/3).&*s u001s E--.s $D> s 'tn s #' s d^s  uU&&')CC Ds s r2r;a ViViT Transformer model with a video classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for Kinetics-400. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. ) custom_introc^\rSrSrU4Sjr\S S\\RS\\RS\\RS\\ S\\ S\ S \\ S \ \ \R\ 44S jj5rS rU=r$)VivitForVideoClassificationixc.>[TU]U5 URUl[USS9UlURS:a+[ R "URUR5O[ R"5Ul UR5 g)NF)rAr) r r! num_labelsr;r%rrr'Identity classifierr@r,s r0r!$VivitForVideoClassification.__init__ss   ++%@ OUN_N_bcNc"))F$6$68I8IJikititiv r2r<rlabelsrrr3rr\c UbUOURRnURUUUUUUS9nUSn URU SS2SSS245n Sn Ub~URS:Xa2[ 5n U "U R S5UR S55n O<[5n U "U R SUR5UR S55n U(dU 4USS-n U bU 4U -$U $[U U URURS9$)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import av >>> import numpy as np >>> import torch >>> from transformers import VivitImageProcessor, VivitForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=4, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitForVideoClassification.from_pretrained("google/vivit-b-16x2-kinetics400") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) LABEL_116 ```N)rrrr3rrrr^r)losslogitsrr ) r.rMr%rZrXr rjr rrr )r-r<rr\rrr3rrrQr_r^loss_fctrs r0rA#VivitForVideoClassification.forwardsz&1% ~ 'tn ~ #'~ d^~  uU&&')>> ?~ ~ r2rV)r;r$rV)r)4rHtypingrrrrrrRtorch.utils.checkpointrtorch.nnr r activationsr modeling_outputsr rrmodeling_utilsrr pytorch_utilsrrutilsrrrconfiguration_vivitr get_loggerrDrModulerrNrxfloatrrrrrrrrrr$r;rV__all__rr2r0rps88 .!bbFQ77,   H %)RYY)XLbiiLn% II% <<% % << % U\\* %  %%>;;~bii&$RYY$N $ "))  $$N0 2990 f ")) 4?44:T %T T n  L "6L  L ^ Pr2