a «º”hî€ã @s¾ddlZddlmZddlmZddlmZddlmZm Z m Z ddl Z ddl Z ddl mZddlmZmZddlmZdd lmZd d lmZmZmZd d lmZd d lmZmZgd¢ZeGdd„dƒƒZ ee!e!dœdd„Z"e j#e!e!e$e j#e!fdœdd„Z%e j#e!e!e!e j#dœdd„Z&e j' (d¡e j' (d¡Gdd„dej)ƒZ*e j#e!e j#dœdd„Z+e j#e j#e$e!e!e!fe$e!e!e!fe j#e j#e j#e j#dœd d!„Z,e j#e j#e-d"œd#d$„Z.e j' (d!¡e j' (d$¡Gd%d&„d&ej)ƒZ/Gd'd(„d(ej)ƒZ0Gd)d*„d*ej)ƒZ1Gd+d,„d,ej)ƒZ2e3e e4e ee-ee2d-œd.d/„Z5Gd0d1„d1eƒZ6Gd2d3„d3eƒZ7eƒed4e6j8fd5dd6d7œe e6e-ee2d8œd9d:„ƒƒZ9eƒed4e7j8fd5dd6d7œe e7e-ee2d8œd;d<„ƒƒZ:dS)=éN)ÚSequence)Ú dataclass)Úpartial)ÚAnyÚCallableÚOptionalé)ÚMLPÚStochasticDepth)ÚVideoClassification)Ú_log_api_usage_onceé)Úregister_modelÚWeightsÚ WeightsEnum)Ú_KINETICS400_CATEGORIES)Ú_ovewrite_named_paramÚhandle_legacy_interface)ÚMViTÚMViT_V1_B_WeightsÚ mvit_v1_bÚMViT_V2_S_WeightsÚ mvit_v2_sc@sVeZdZUeed<eed<eed<eeed<eeed<eeed<eeed<dS) Ú MSBlockConfigÚ num_headsÚinput_channelsÚoutput_channelsÚkernel_qÚ kernel_kvÚstride_qÚ stride_kvN)Ú__name__Ú __module__Ú __qualname__ÚintÚ__annotations__Úlist©r'r'úK/var/www/auris/lib/python3.9/site-packages/torchvision/models/video/mvit.pyrs    r)ÚsÚreturncCsd}|D] }||9}q|S©Nér')r)ÚproductÚvr'r'r(Ú_prod's r/)ÚxÚ target_dimÚ expand_dimr*cCs@| ¡}||dkr | |¡}n||kr8td|j›ƒ‚||fS)Nr,zUnsupported input dimension )ÚdimÚ unsqueezeÚ ValueErrorÚshape©r0r1r2Ú tensor_dimr'r'r(Ú _unsqueeze.s   r9)r0r1r2r8r*cCs||dkr| |¡}|Sr+)Úsqueezer7r'r'r(Ú_squeeze7s  r;csleZdZd ejeejeejeddœ‡fdd„ Zej e e e e fe ej e e e e ffdœdd„Z ‡Z S) ÚPoolNF)ÚpoolÚnormÚ activationÚnorm_before_poolr*csVtƒ ¡||_g}|dur&| |¡|dur8| |¡|rFtj|Žnd|_||_dS)N)ÚsuperÚ__init__r=ÚappendÚnnÚ SequentialÚnorm_actr@)Úselfr=r>r?r@Úlayers©Ú __class__r'r(rBBs   z Pool.__init__©r0Úthwr*c Csöt|ddƒ\}}tj|ddd\}}| dd¡}|jdd…\}}}| |||f|¡ ¡}|jrx|jdurx| |¡}|  |¡}|jdd…\}} } | |||d¡ dd¡}tj ||fdd}|jsÚ|jdurÚ| |¡}t |dd|ƒ}||| | ffS) Nér,©r,r )Úindicesr3réÿÿÿÿ©r3) r9ÚtorchZ tensor_splitÚ transposer6ÚreshapeÚ contiguousr@rFr=Úcatr;) rGr0rLr8Ú class_tokenÚBÚNÚCÚTÚHÚWr'r'r(ÚforwardSs    z Pool.forward)NF)r!r"r#rDÚModulerÚboolrBrRÚTensorÚtupler$r^Ú __classcell__r'r'rIr(r<Asûúr<)Ú embeddingÚdr*cCs@|jd|kr|Stjj| dd¡ d¡|dd d¡ dd¡S)Nrr,Zlinear)ÚsizeÚmode)r6rDZ functionalZ interpolateÚpermuter4r:)rdrer'r'r(Ú _interpolatemsýûúÿri)ÚattnÚqÚq_thwÚk_thwÚ rel_pos_hÚ rel_pos_wÚ rel_pos_tr*c% Cs6|\}}} |\} } } tdt|| ƒdƒ} tdt| | ƒdƒ}tdt|| ƒdƒ}t| |dƒ}t|| dƒ}t |¡dd…df|t | ¡ddd…fd| |}t| | dƒ}t| | dƒ}t | ¡dd…df|t | ¡ddd…fd| |}t| |dƒ}t|| dƒ}t |¡dd…df|t | ¡ddd…fd| |}t|| ƒ}t||ƒ}t||ƒ}|| ¡}|| ¡}|| ¡}|j\}}}}|dd…dd…dd…f ||||| |¡} t d| |¡}!t d| |¡}"|   dddddd ¡ ||||| |¡} t  | |  dd¡¡  dd¡}#|#  |||| || ¡  dddddd ¡}#|!dd…dd…dd…dd…dd…ddd…df|"dd…dd…dd…dd…dd…dddd…f|#dd…dd…dd…dd…dd…dd…ddf ||||| | | | ¡}$|dd…dd…dd…dd…f|$7<|S) Nr r,çð?zbythwc,hkc->bythwkzbythwc,wkc->bythwkrrrMé) r$ÚmaxrRZarangeriÚlongr6rTZeinsumrhÚmatmulrSÚview)%rjrkrlrmrnrorpZq_tZq_hZq_wZk_tZk_hZk_wZdhZdwÚdtZ q_h_ratioZ k_h_ratioZdist_hZ q_w_ratioZ k_w_ratioZdist_wZ q_t_ratioZ k_t_ratioZdist_tZRhZRwZRtrXZn_headÚ_r3Zr_qZrel_h_qZrel_w_qZrel_q_tZrel_posr'r'r(Ú _add_rel_pos|sJ  <<<      **$..ÿ.þÿü(ry©r0ZshortcutÚresidual_with_cls_embedc CsX|r| |¡nD|dd…dd…dd…dd…f|dd…dd…dd…dd…f7<|Sr+)Úadd_rzr'r'r(Ú _add_shortcut¸s Dr}cs”eZdZdejfeeeeeeeeeeeeeeeeee dej fddœ‡fdd„ Z e j eeeefee j eeeeffdœdd „Z‡ZS) ÚMultiscaleAttentionç.N)Ú input_sizeÚ embed_dimÚ output_dimrrrrr Ú residual_poolr{Ú rel_pos_embedÚdropoutÚ norm_layerr*c sttƒ ¡||_||_||_|||_dt |j¡|_| |_ | |_ t   |d|¡|_ t   ||¡g}| dkr€| t j| dd¡t j|Ž|_d|_t|ƒdksªt|ƒdkrädd„|Dƒ}tt j|j|j||||jd d | |jƒƒ|_d|_d|_t|ƒdks t|ƒdkrrd d„|Dƒ}tt j|j|j||||jd d | |jƒƒ|_tt j|j|j||||jd d | |jƒƒ|_d|_d|_d|_| rpt|dd…ƒ}t|ƒd kr´||dn|}t|ƒd krÒ||dn|}d t||ƒd}d |d d}t  t ||j¡¡|_t  t ||j¡¡|_t  t ||j¡¡|_t j j!|jddt j j!|jddt j j!|jdddS)NrqrrT©Úinplacer,cSsg|]}t|dƒ‘qS©r ©r$)Ú.0rkr'r'r(Ú æóz0MultiscaleAttention.__init__..F)ÚstrideÚpaddingÚgroupsÚbiascSsg|]}t|dƒ‘qSr‰rŠ)r‹Úkvr'r'r(rŒ÷rrr ç{®Gáz”?©Ústd)"rArBrr‚rÚhead_dimÚmathÚsqrtÚscalerrƒr{rDÚLinearÚqkvrCÚDropoutrEÚprojectÚpool_qr/r<ÚConv3dÚpool_kÚpool_vrnrorprsÚlenÚ ParameterrRÚzerosÚinitÚ trunc_normal_)rGr€rr‚rrrrr rƒr{r„r…r†rHZ padding_qZ padding_kvrfZq_sizeZkv_sizeZ spatial_dimZ temporal_dimrIr'r(rBÅsŽ   ù ö ù ö ù ö zMultiscaleAttention.__init__rKc Cs<|j\}}}| |¡ ||d|j|j¡ dd¡jdd\}}}|jdurZ| ||¡\}} n|} |jdurx| ||¡d}|j dur’|  ||¡\}}t   |j || dd¡¡} |j durä|jdurä|jdurät| ||| |j |j|jƒ} | jdd} t   | |¡}|jrt|||jƒ| dd¡ |d|j¡}| |¡}||fS)Nrr,r rQrrP)r6r›rTrr–rSZunbindr r¡ržrRrur™rnrorpryZsoftmaxrƒr}r{r‚r) rGr0rLrXrYrZrkÚkr.rmrjr'r'r(r^!s6 2   ù   zMultiscaleAttention.forward)r!r"r#rDÚ LayerNormr&r$r`Úfloatrr_rBrRrarbr^rcr'r'rIr(r~Äs&ò ñ\r~c s~eZdZddejfeeeeeeee e e dej fddœ ‡fdd„ Z e jeeeefee jeeeeffdœdd „Z‡ZS) ÚMultiscaleBlockr.N) r€Úcnfrƒr{r„Úproj_after_attnr…Ústochastic_depth_probr†r*c stƒ ¡||_d|_t|jƒdkr\dd„|jDƒ} dd„| Dƒ} ttj| |j| ddƒ|_|rf|j n|j } | |j ƒ|_ | | ƒ|_ t |j tjƒ|_t||j | |j|j|j|j|j||||| d |_t| d| |j gtj|dd|_t|d ƒ|_d|_|j |j krt |j |j ¡|_dS) Nr,cSs g|]}|dkr|dn|‘qSrNr')r‹r)r'r'r(rŒVrz,MultiscaleBlock.__init__..cSsg|]}t|dƒ‘qSr‰rŠ)r‹r§r'r'r(rŒWr)rŽr) rrrr r„rƒr{r…r†rM)Zactivation_layerr…rˆÚrow)rArBr¬Ú pool_skipr/rr<rDZ MaxPool3drrÚnorm1Únorm2Ú isinstanceZ BatchNorm1dÚneeds_transposalr~rrrr rjr ZGELUÚmlpr Ústochastic_depthrrš) rGr€r«rƒr{r„r¬r…r­r†Z kernel_skipZ padding_skipZattn_dimrIr'r(rBEsN ÿ  ó û zMultiscaleBlock.__init__rKc Csä|jr | | dd¡¡ dd¡n| |¡}| ||¡\}}|jdusJ|jsN|n| |¡}|jdurf|n| ||¡d}|| |¡}|jr¤| | dd¡¡ dd¡n| |¡}|jdus¾|jrÂ|n| |¡}|| |  |¡¡|fS)Nr,r r) r³r°rSrjrr¬r¯rµr±r´) rGr0rLZx_norm1Zx_attnZthw_newZx_skipZx_norm2Zx_projr'r'r(r^s**zMultiscaleBlock.forward)r!r"r#rDr¨r&r$rr`r©rr_rBrRrarbr^rcr'r'rIr(rªDs  ö õ:rªcsFeZdZeeeefeeddœ‡fdd„ Zejejdœdd„Z ‡Z S)ÚPositionalEncodingN)Ú embed_sizeÚ spatial_sizeÚ temporal_sizer„r*csŽtƒ ¡||_||_t t |¡¡|_d|_ d|_ d|_ |sŠt t |jd|jd|¡¡|_ t t |j|¡¡|_ t t |¡¡|_ dS)Nrr,) rArBr¸r¹rDr£rRr¤rWÚ spatial_posÚ temporal_posÚ class_pos)rGr·r¸r¹r„rIr'r(rBs $zPositionalEncoding.__init__©r0r*cCs¼|j | d¡d¡ d¡}tj||fdd}|jdur¸|jdur¸|jdur¸|jj \}}tj |j|dd}|  |j d¡ |j dd¡  d|¡¡tj|j d¡|fdd d¡}|  |¡|S)NrrPr,rQ)rWÚexpandrfr4rRrVrºr»r¼r6Zrepeat_interleaver|r¹rT)rGr0rWZhw_sizer·Z pos_embeddingr'r'r(r^›s &  zPositionalEncoding.forward) r!r"r#r$rbr`rBrRrar^rcr'r'rIr(r¶Œs"r¶csžeZdZdeeefeeeeeeeeeeee e de j fe e de j feeeefeeeefeeeefdd œ‡fd d „ Z ejejd œd d„Z‡ZS)rçà?réN©rér©r rMrM©r,rr.)r¸r¹Ú block_settingrƒr{r„r¬r…Úattention_dropoutr­Ú num_classesÚblockr†Úpatch_embed_kernelÚpatch_embed_strideÚpatch_embed_paddingr*cs*tƒ ¡t|ƒt|ƒ}|dkr*tdƒ‚| dur6t} | durLttjdd} tj d|dj |||d|_ dd „t |f||j j ƒDƒ}t|dj |d |d f|d|d |_t ¡|_t|ƒD]^\}}| ||d }|j | ||||||| || d ¡t|jƒdkrÀdd „t ||jƒDƒ}qÀ| |djƒ|_t tj|ddt |dj| ¡¡|_| ¡D]Æ}t|tjƒr®tjj|jddt|tjƒr"|j dur"tj !|j d¡ntt|tjƒrö|jdurØtj !|jd ¡|j dur"tj !|j d¡n,t|tƒr^| "¡D]}tjj|ddq q^dS)aÄ MViT main class. Args: spatial_size (tuple of ints): The spacial size of the input as ``(H, W)``. temporal_size (int): The temporal size ``T`` of the input. block_setting (sequence of MSBlockConfig): The Network structure. residual_pool (bool): If True, use MViTv2 pooling residual connection. residual_with_cls_embed (bool): If True, the addition on the residual connection will include the class embedding. rel_pos_embed (bool): If True, use MViTv2's relative positional embeddings. proj_after_attn (bool): If True, apply the projection after the attention. dropout (float): Dropout rate. Default: 0.0. attention_dropout (float): Attention dropout rate. Default: 0.0. stochastic_depth_prob: (float): Stochastic depth rate. Default: 0.0. num_classes (int): The number of classes. block (callable, optional): Module specifying the layer which consists of the attention and mlp. norm_layer (callable, optional): Module specifying the normalization layer to use. patch_embed_kernel (tuple of ints): The kernel of the convolution that patchifies the input. patch_embed_stride (tuple of ints): The stride of the convolution that patchifies the input. patch_embed_padding (tuple of ints): The padding of the convolution that patchifies the input. rz+The configuration parameter can't be empty.Ngíµ ÷Æ°>)Zepsr)Z in_channelsZ out_channelsZ kernel_sizerŽrcSsg|]\}}||‘qSr'r'©r‹rfrŽr'r'r(rŒìrz!MViT.__init__..r,r )r·r¸r¹r„rq) r€r«rƒr{r„r¬r…r­r†cSsg|]\}}||‘qSr'r'rÌr'r'r(rŒ rrPTr‡r“r”r)#rArBr r¢r5rªrrDr¨rŸrÚ conv_projÚziprŽr¶Ú pos_encodingZ ModuleListÚblocksÚ enumeraterCrrr>rErœršÚheadÚmodulesr²r¥r¦Zweightr‘Z constant_Ú parameters)rGr¸r¹rÅrƒr{r„r¬r…rÆr­rÇrÈr†rÉrÊrËZtotal_stage_blocksr€Zstage_block_idr«Zsd_probÚmÚweightsrIr'r(rBªsv) ûü ÷ÿ þ     z MViT.__init__r½cCsŠt|ddƒd}| |¡}| d¡ dd¡}| |¡}|jjf|jj}|jD]}|||ƒ\}}qN| |¡}|dd…df}|  |¡}|S)Nrrr rr,) r9rÍÚflattenrSrÏr¹r¸rÐr>rÒ)rGr0rLrÈr'r'r(r^"s     z MViT.forward) r¿rrrÀNNrÁrÃrÄ)r!r"r#rbr$rrr`r©rrrDr_rBrRrar^rcr'r'rIr(r©s: ï    îxr)rÅr­rÖÚprogressÚkwargsr*c KsÚ|durbt|dt|jdƒƒ|jdd|jddks>J‚t|d|jdƒt|d|jdƒ| dd ¡}| dd ¡}tf|||| d d ¡| d d¡| dd ¡| dd ¡|dœ|¤Ž}|durÖ| |j|dd¡|S)NrÇÚ categoriesÚmin_sizerr,r¸r¹Úmin_temporal_size©éàrÞérƒFr{Tr„r¬)r¸r¹rÅrƒr{r„r¬r­)rØZ check_hash)rr¢ÚmetaÚpoprZload_state_dictZget_state_dict)rÅr­rÖrØrÙr¸r¹Úmodelr'r'r(Ú_mvit9s,       ø ÷ rãc @sJeZdZedeedddddddedd d d d d dœidddœ dZeZdS)rz:https://download.pytorch.org/models/mvit_v1_b-dbeb1030.pthrÝ©é©çÍÌÌÌÌÌÜ?rçrç©çÍÌÌÌÌÌÌ?réré©Z crop_sizeZ resize_sizeÚmeanr•rßzShttps://github.com/facebookresearch/pytorchvideo/blob/main/docs/source/model_zoo.mdúœThe weights were ported from the paper. The accuracies are estimated on video-level with parameters `frame_rate=7.5`, `clips_per_video=5`, and `clip_len=16`ip¢.ú Kinetics-400gJ +‡žS@gh‘í|?eW@©zacc@1zacc@5gu“V¦Q@gœÄ °rxa@© rÛrÜrÚZrecipeZ_docsZ num_paramsZ_metricsZ_opsÚ _file_size©ÚurlZ transformsràN© r!r"r#rrr rÚKINETICS400_V1ÚDEFAULTr'r'r'r(rZs2ûþÿï÷rc @sJeZdZedeedddddddedd d d d d dœidddœ dZeZdS)rz:https://download.pytorch.org/models/mvit_v2_s-ae3be167.pthrÝrärærèrêrßzChttps://github.com/facebookresearch/SlowFast/blob/main/MODEL_ZOO.mdrìirígœÄ °r0T@gÃõ(\ªW@rîgu“VP@g?5^ºI|`@rïrñNrór'r'r'r(r{s2ûþÿï÷rZ pretrained)rÖT)rÖrØ)rÖrØrÙr*cKsÚt |¡}gd¢gd¢gd¢ggd¢ggd¢gggggggggggd¢gggd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gggd¢ggd¢gggggggggggd¢gggd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd œ}g}tt|d ƒƒD]Z}| t|d ||d ||d ||d ||d||d||d|d ¡qVtfdd|dd| dd¡||dœ|¤ŽS)a¿ Constructs a base MViTV1 architecture from `Multiscale Vision Transformers `__. .. betastatus:: video module Args: weights (:class:`~torchvision.models.video.MViT_V1_B_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.video.MViT_V1_B_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.video.MViT`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.video.MViT_V1_B_Weights :members: ©r,r r rMrMrMrMrMrMrMrMrMrMrMér÷©é`éÀrúé€rûrûrûrûrûrûrûrûrûrûérü)rúrúrûrûrûrûrûrûrûrûrûrûrûrürürü©rrr©r,r r ©r,r÷r÷©r,rMrM©r,r,r,©rrrrrrr rrrrrrr rÝrßFr­çš™™™™™É?)r¸r¹rÅrƒr{r­rÖrØ)rÚverifyÚranger¢rCrrãrá©rÖrØrÙÚconfigrÅÚir'r'r(rœs„ .ð.ðè,       ùÿ  ø ÷rcKsFt |¡}gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢ggd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢ggd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢ggd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd œ}g}tt|d ƒƒD]Z}| t|d ||d ||d ||d ||d||d||d|d ¡q¾tfdd|dddd| dd¡||dœ |¤ŽS)aCConstructs a small MViTV2 architecture from `Multiscale Vision Transformers `__ and `MViTv2: Improved Multiscale Vision Transformers for Classification and Detection `__. .. betastatus:: video module Args: weights (:class:`~torchvision.models.video.MViT_V2_S_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.video.MViT_V2_S_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.video.MViT`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.video.MViT_V2_S_Weights :members: rö)rùrùrúrúrûrûrûrûrûrûrûrûrûrûrûrürørýrrþrÿrrrrrrrrr rÝrßTFr­r) r¸r¹rÅrƒr{r„r¬r­rÖrØ)rrrr¢rCrrãrárr'r'r(rþsÈ ððððÆN       ùÿ  ö õr);r—Úcollections.abcrZ dataclassesrÚ functoolsrÚtypingrrrrRZtorch.fxZtorch.nnrDÚopsr r Ztransforms._presetsr Úutilsr Z_apirrrZ_metarÚ_utilsrrÚ__all__rr$r/rarbr9r;ZfxÚwrapr_r<riryr`r}r~rªr¶rr&r©rãrrrôrrr'r'r'r(Úsp            ,  ø <  Hú !!!$`