o Zh @sdZddlZddlZddlmZmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddlmZmZddlmZdd lmZmZdd lmZd d lmZeeZGd dde jZe j j!ddZ"ddZ#Gddde jZ$d9de j%de&de'de j%fddZ(Gddde jZ)Gdd d e jZ*Gd!d"d"e jZ+Gd#d$d$e jZ,d%d&Z-d'd(Z.Gd)d*d*e jZ/Gd+d,d,e jZ0d-e jddfd.d/Z1eGd0d1d1eZ2eGd2d3d3e2Z3ed4d5Gd6d7d7e2eZ4gd8Z5dS):zPyTorch ViTDet backbone.N)DictListOptionalTupleUnion)nn)ACT2FN)BackboneOutputBaseModelOutput)PreTrainedModel)auto_docstringlogging) BackboneMixin) VitDetConfigcs>eZdZdZfddZddZdejdejfdd ZZ S) VitDetEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_ ||_||_||_ |j r]|d}t td||j|_nd|_t j||||d|_dS)Nrr)Z kernel_sizeZstride)super__init__Zpretrain_image_size patch_size num_channels hidden_size isinstance collectionsabcIterable image_size num_patchesZ use_absolute_position_embeddingsr Parametertorchzerosposition_embeddingsConv2d projection)selfconfigrrrrrZ num_positions __class__Y/var/www/auris/lib/python3.10/site-packages/transformers/models/vitdet/modeling_vitdet.pyr*s  zVitDetEmbeddings.__init__cCs|r |ddddf}|jd}tt|}|||kr"tdtjs/||ks/||krOtj j | d||d dddd||fdd d }| ddddS| d||dS) a Calculate absolute positional embeddings. If needed, resize embeddings and remove cls_token dimension for the original embeddings. Args: abs_pos_embeddings (`torch.Tensor`): Absolute positional embeddings with (1, num_position, num_channels). has_cls_token (`bool`): If true, has 1 embedding in abs_pos_embeddings for cls token. height (`int`): Height of input image tokens. width (`int`): Width of input image tokens. Returns: Absolute positional embeddings after processing with shape (1, height, width, num_channels) Nrz5Absolute position embeddings must be a square number.rrZbicubicF)sizemodeZ align_corners) shapeintmathsqrt ValueErrorrjit is_tracingr functional interpolatereshapepermute)r$Zabs_pos_embeddingsZ has_cls_tokenheightwidthZ num_positionr,Znew_abs_pos_embeddingsr(r(r)get_absolute_positions@s  z'VitDetEmbeddings.get_absolute_positions pixel_valuesreturncCs|jd}||jkrtd|jd|d||}|jdurA|dddd}|||jd|jd|jd}|dddd}|S) NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rr+rT)r.rr2r#r!r8r;)r$r<r embeddingsr(r(r)forwardfs"    zVitDetEmbeddings.forward) __name__ __module__ __qualname____doc__rr;rTensorr@ __classcell__r(r(r&r)r$s  &rcCstdt||d}|jd|kr3tjj|d|jddddd|dd}|d|dd}n|}t |dddft||d}t |dddft||d}|||dt||d}|| S) a Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (`int`): Size of query q. k_size (`int`): Size of key k. rel_pos (`torch.Tensor`): Relative position embeddings (num_embeddings, num_channels). Returns: Extracted positional embeddings according to relative positions. r+rrr*Zlinear)r,r-N?) r/maxr.rr5r6r7r8rZarangelong)q_sizek_sizeZrel_posZ max_rel_distZrel_pos_resizedZq_coordsZk_coordsZrelative_coordsr(r(r) get_rel_pos|s$$ rLcCs|\}}|\}} t|||} t|| |} |j\} } }|| |||}td|| } td|| }|| |||| | dddddddddf|dddddddddf| |||| }|S)a Calculate decomposed Relative Positional Embeddings as introduced in [MViT2](https://github.com/facebookresearch/mvit/blob/19786631e330df9f3622e5402b4a419a263a2c80/mvit/models/attention.py). Args: attn (`torch.Tensor`): Attention map. queries (`torch.Tensor`): Query q in the attention layer with shape (batch_size, queries_height * queries_width, num_channels). rel_pos_h (`torch.Tensor`): Relative position embeddings (Lh, num_channels) for height axis. rel_pos_w (`torch.Tensor`): Relative position embeddings (Lw, num_channels) for width axis. q_size (`Tuple[int]`): Spatial sequence size of query q with (queries_height, queries_width). k_size (`Tuple[int]`): Spatial sequence size of key k with (keys_height, keys_width). Returns: attn (Tensor): attention map with added relative positional embeddings. zbhwc,hkc->bhwkzbhwc,wkc->bhwkN)rLr.r7rZeinsumview)Zattnqueries rel_pos_h rel_pos_wrJrKZqueries_heightZ queries_widthZ keys_heightZ keys_widthZrelative_heightZrelative_width batch_size_dimZr_qZrelative_weightr(r(r)!add_decomposed_relative_positionss      rTcs,eZdZdZdfdd Zd ddZZS) VitDetAttentionz=Multi-head Attention block with relative position embeddings.Ncst|j}|j}||_||}|d|_tj||d|jd|_ t|||_ |j |_ |j rSt t d|dd||_t t d|dd||_dSdS)z Args: config (`VitDetConfig`): Model configuration. input_size (`Tuple[int]`, *optional*): Input resolution, only required in case relative position embeddings are added. grbiasr+rrN)rrrZnum_attention_heads num_headsscalerLinearZqkv_biasqkvproj use_relative_position_embeddingsrrr rOrP)r$r% input_sizerSrXZhead_dimr&r(r)rs   $zVitDetAttention.__init__FcCs&|j\}}}}|||||d|jdddddd}|d||j||dd\}} } ||j| dd} |jrMt | ||j |j ||f||f} | j dd} | | }| ||j||d}|ddddd}||||d}||}|r| ||j| jd| jd} || f} | S|f} | S) Nrr*r+rr)rS)r.r[r7rXr8ZunbindrYZ transposer]rTrOrPZsoftmaxrMr\)r$ hidden_stateoutput_attentionsrQr9r:rRr[rNkeysvaluesZattention_scoresZattention_probsoutputsr(r(r)r@s,,&  zVitDetAttention.forwardN)FrArBrCrDrr@rFr(r(r&r)rUsrUFinput drop_probtrainingr=cCsd|dks|s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rhrr)r)dtypedevice)r.ndimrZrandrlrmZfloor_div)rirjrkZ keep_probr.Z random_tensoroutputr(r(r) drop_path s rqcsTeZdZdZd deeddffdd Zdejdejfdd Z de fd d Z Z S) VitDetDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nrjr=cst||_dSrf)rrrj)r$rjr&r(r)r!s  zVitDetDropPath.__init__ hidden_statescCst||j|jSrf)rqrjrk)r$rsr(r(r)r@%szVitDetDropPath.forwardcCs d|jS)Nzp={})formatrjr$r(r(r) extra_repr(s zVitDetDropPath.extra_reprrf) rArBrCrDrfloatrrrEr@strrvrFr(r(r&r)rrs rrcs*eZdZdZdfdd ZddZZS)VitDetLayerNormaL A LayerNorm variant, popularized by Transformers, that performs point-wise mean and variance normalization over the channel dimension for inputs that have shape (batch_size, channels, height, width). https://github.com/facebookresearch/ConvNeXt/blob/d1fa8f6fef0a165b27399986cc2bdacc92777e40/models/convnext.py#L119 ư>cs@ttt||_tt||_||_ |f|_ dSrf) rrrrrZonesweightr rWepsnormalized_shape)r$r}r|r&r(r)r3s  zVitDetLayerNorm.__init__cCsn|jddd}||djddd}||t||j}|jddddf||jddddf}|S)NrT)Zkeepdimr+)meanpowrr1r|r{rW)r$xusr(r(r)r@:s ,zVitDetLayerNorm.forward)rzrgr(r(r&r)ry,srycs(eZdZdZfddZddZZS)VitDetResBottleneckBlockz The standard bottleneck residual block without the last activation layer. It contains 3 conv layers with kernels 1x1, 3x3, 1x1. csttj||ddd|_t||_t|j|_ tj||dddd|_ t||_ t|j|_ tj||ddd|_ t||_dS)ar Args: config (`VitDetConfig`): Model configuration. in_channels (`int`): Number of input channels. out_channels (`int`): Number of output channels. bottleneck_channels (`int`): Number of output channels for the 3x3 "bottleneck" conv layers. rFrVr)paddingrWN)rrrr"conv1rynorm1r hidden_actZact1conv2norm2Zact2conv3norm3)r$r% in_channels out_channelsbottleneck_channelsr&r(r)rHs     z!VitDetResBottleneckBlock.__init__cCs&|}|D]}||}q||}|Srf)children)r$routlayerr(r(r)r@`s   z VitDetResBottleneckBlock.forwardrgr(r(r&r)rBs rcs@eZdZdededdffdd Zdejdejfdd ZZS) VitDetMlp in_featureshidden_featuresr=NcsDtt|||_t|j|_t|||_t |j |_ dSrf) rrrrZfc1r ractfc2ZDropoutZ dropout_probdrop)r$r%rrr&r(r)rjs  zVitDetMlp.__init__rcCs6||}||}||}||}||}|Srf)rrrr)r$rr(r(r)r@qs     zVitDetMlp.forward) rArBrCr/rrrEr@rFr(r(r&r)risrc Cs|j\}}}}||||}||||}tj|ddd|d|f}||||}} |||||| |||}|ddddddd|||} | || ffS)a  Partition into non-overlapping windows with padding if needed. Args: hidden_state (`torch.Tensor`): Input tokens with [batch_size, height, width, num_channels]. window_size (`int`): Window size. Returns: `tuple(torch.FloatTensor)` comprising various elements: - windows: windows after partition with [batch_size * num_windows, window_size, window_size, num_channels]. - (padded_height, padded_width): padded height and width before partition rrrr+r_r*)r.rr5padrMr8 contiguous) ra window_sizerQr9r:rZ pad_heightZ pad_width padded_height padded_widthwindowsr(r(r)window_partition{s$ rc Cs|\}}|\}}|jd||||}||||||||d} | dddddd} | |||d} | ddd|d|ddf} | S) aB Window unpartition into original sequences and removing padding. Args: windows (`torch.Tensor`): Input tokens with [batch_size * num_windows, window_size, window_size, num_channels]. window_size (`int`): Window size. pad_height_width (`Tuple[int]`): Padded height and width (padded_height, padded_width). height_width (`Tuple[int]`): Original height and width before padding. Returns: hidden_state: unpartitioned sequences with [batch_size, height, width, num_channels]. rr*rrr+r_rN)r.rMr8r) rrpad_height_widthZ height_widthrrr9r:rQrar(r(r)window_unpartitions$rc s|eZdZdZ ddededededd f fd d Z dd e j d e e j dede e e j e j fe e j ffddZZS) VitDetLayerzCThis corresponds to the Block class in the original implementation.rFr%drop_path_rateruse_residual_blockr=Nc st|j}|j}t|ttfr|n||f}|j}t|ttfr$|n||f}|d|d|d|df}tj ||j d|_ t ||dkrI|n||fd|_ |dkrXt|nt|_tj ||j d|_t||t||jd|_||_||_|jrt||||dd|_dSdS) Nrr)r|)r^rh)r%rrr+)r%rrr)rrrrrlisttuplerr LayerNormZlayer_norm_epsrrU attentionrrZIdentityrqrrr/Z mlp_ratiomlprrrresidual) r$r%rrrrSrrr^r&r(r)rs0   zVitDetLayer.__init__rs head_maskrbc Cs|dddd}|}||}|jdkr'|jd|jd}}t||j\}}|j||d}|d}|dd} |jdkrGt||j|||f}|||}|||| |}|dddd}|j rk| |}|f| } | S)Nrr+rr)rb) r8rrr.rrrrqrrrr) r$rsrrbZshortcutr9r:rZself_attention_outputsrer(r(r)r@s*      zVitDetLayer.forward)rrF)NF)rArBrCrDrrwr/boolrrrErrrr@rFr(r(r&r)rs2&rcsbeZdZdeddffdd Z    ddejd eejd ed ed ede e e ff d dZ Z S) VitDetEncoderr%r=Nc st||_|j}ddtjd|j|ddD}g}t|D]}|t |||||j vr1|j nd||j vdq t ||_d|_dS)NcSsg|]}|qSr()item).0rr(r(r) sz*VitDetEncoder.__init__..rcpu)rm)rrrF)rrr%num_hidden_layersrZlinspacerrangeappendrZwindow_block_indicesrZresidual_block_indicesrZ ModuleListrgradient_checkpointing)r$r%depthrZlayersir&r(r)r s   zVitDetEncoder.__init__FTrsrrboutput_hidden_states return_dictc Cs|rdnd}|r dnd}t|jD]8\}} |r||f}|dur$||nd} |jr6|jr6|| j|| |} n| || |} | d}|rI|| df}q|rQ||f}|s_tdd|||fDSt|||dS)Nr(rrcss|] }|dur|VqdSrfr()rvr(r(r) Fsz(VitDetEncoder.forward..Zlast_hidden_staters attentions) enumeraterrrkZ_gradient_checkpointing_func__call__rr ) r$rsrrbrrZall_hidden_statesZall_self_attentionsrZ layer_moduleZlayer_head_maskZ layer_outputsr(r(r)r@"s6      zVitDetEncoder.forward)NFFT)rArBrCrrrrErrrrr r@rFr(r(r&r)r s& rmodulecCs6tjj|jddd|jdurtj|jddSdS)a Initialize `module.weight` using the "MSRAFill" implemented in Caffe2. Also initializes `module.bias` to 0. Source: https://detectron2.readthedocs.io/en/latest/_modules/fvcore/nn/weight_init.html. Args: module (torch.nn.Module): module to initialize. Zfan_outZrelu)r-Z nonlinearityNr)rinitZkaiming_normal_r{rWZ constant_)rr(r(r)caffe2_msra_fillNs rc@sBeZdZeZdZdZdZgZde e j e j e j fddfddZdS) VitDetPreTrainedModelZvitdetr<Trr=NcCst|tjtjfr0tjj|jjt j d|j j d|jj |j_|jdur.|jjdSdSt|tjrE|jj|jjddSt|trdtjj|jjt j d|j j d|jj |j_dSt|tr|j jrtjj|jjt j d|j j d|j_tjj|jjt j d|j j d|j_dSt|tr|j|j|jfD]}t|q|j|jfD]}|jjd|jjq|jjj|jjjdSdS)zInitialize the weightsrh)r~ZstdNrG) rrrZr"rZ trunc_normal_r{datatorZfloat32r%Zinitializer_rangerlrWZzero_rZfill_rr!rUr]rOrPrrrrrrrr)r$rrr(r(r) _init_weightsdsP          z#VitDetPreTrainedModel._init_weights)rArBrCrZ config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingZ_no_split_modulesrrrZr"rrr(r(r(r)r\s&rcseZdZdeffdd ZdefddZdeee efddfd d Z e     dd e e jd e e jd e ede ede edeeeff ddZZS) VitDetModelr%cs2t|||_t||_t||_|dSrf)rrr%rr?rencoder post_initr$r%r&r(r)rs    zVitDetModel.__init__r=cC|jjSrfr?r#rur(r(r)get_input_embeddingsz VitDetModel.get_input_embeddingsheads_to_pruneNcCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrZ prune_heads)r$rrZheadsr(r(r) _prune_headsszVitDetModel._prune_headsr<rrbrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|||jj}||}|j|||||d}|d}|sL|f|ddSt ||j |j dS)a Examples: ```python >>> from transformers import VitDetConfig, VitDetModel >>> import torch >>> config = VitDetConfig() >>> model = VitDetModel(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 768, 14, 14] ```Nz You have to specify pixel_values)rrbrrrrr) r%rbruse_return_dictr2Z get_head_maskrr?rr rsr) r$r<rrbrrembedding_outputZencoder_outputsZsequence_outputr(r(r)r@s. zVitDetModel.forward)NNNNN)rArBrCrrrrrr/rrr rrrErrrr r@rFr(r(r&r)rs.  rzF ViTDet backbone, to be used with frameworks like Mask R-CNN. )Z custom_introc sbeZdZfddZdefddZe   d dejde e d e e d e e de f d d Z Z S)VitDetBackbonecsVttt|_t|_fddtjdD|_ | dS)Ncsg|]}jqSr()r)rrRr%r(r)rsz+VitDetBackbone.__init__..r) rrZ_init_backbonerr?rrrrZ num_featuresrrr&rr)rs     zVitDetBackbone.__init__r=cCrrfrrur(r(r)rrz#VitDetBackbone.get_input_embeddingsNr<rrbrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}||}|j|d||d}|r1|jn|d}d}t|j|D]\} } | |j vrK|| f7}q=|sf|r[|f|dd} | S|f|dd} | St ||rm|jnd|j dS)a Examples: ```python >>> from transformers import VitDetConfig, VitDetBackbone >>> import torch >>> config = VitDetConfig() >>> model = VitDetBackbone(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```NT)rrbrrr(r+) feature_mapsrsr) r%rrrbr?rrszipZ stage_namesZ out_featuresr r) r$r<rrbrrrersrZstagerarpr(r(r)r@s8    zVitDetBackbone.forward)NNN)rArBrCrrrr rrErrr r@rFr(r(r&r)rs$  r)rrr)rhF)6rDcollections.abcrr0typingrrrrrrZtorch.utils.checkpointrZ activationsr Zmodeling_outputsr r Zmodeling_utilsr utilsr rZutils.backbone_utilsrZconfiguration_vitdetrZ get_loggerrAloggerModulerr3Zscript_if_tracingrLrTrUrErwrrqrrryrrrrrrrrrr__all__r(r(r(r)sL      X $) ?' QC3W N