a hn} @s\ddlZddlmZddlmZddlmZddlmZm Z m Z ddl Z ddl Z ddlmmZddl mZmZddlmZmZmZddlmZdd lmZmZdd lmZmZdd lm Z dd l!m"Z"m#Z#dd l$m%Z%gdZ&e'e(e(fe(e(e(e'e(e(fdddZ)e'e(e(fe(e*e'e(e(fdddZ+e(e(e jdddZ,Gdddej-Z.Gdddej-Z/Gdddej-Z0Gdddej-Z1Gd d!d!ej-Z2Gd"d#d#ej-Z3Gd$d%d%ej-Z4Gd&d'd'ej-Z5Gd(d)d)ej-Z6d7e(e*e(e*e(e7e(e(e ee8ee6d+ d,d-Z9Gd.d/d/eZ:eed0e:j;fd1dd2d3e e:e8ee6d4d5d6Z B G H I Jz!B G H I J, B G H J D -> B G H I D)shapercr]rhr,chunkreshaper/riZeinsumrrFZsoftmaxrj)rLrPBGPDHZDHZqkvqkvZdot_prodZpos_biasoutr r r!rSs      z,RelativePositionalMultiHeadAttention.forward) rTrUrVrWrIrDr,r rrrSrZr r rMr!r[s r[cs>eZdZdZeeddfdd ZejejdddZZ S) SwapAxeszPermute the axes of a tensor.N)abrcst||_||_dSN)rCrDrr)rLrrrMr r!rDs zSwapAxes.__init__rOcCst||j|j}|Sr)r,ZswapaxesrrrQr r r!rSszSwapAxes.forward) rTrUrVrWrIrDr,r rSrZr r rMr!rsrcs8eZdZdZddfdd ZeeedddZZS) WindowPartitionzB Partition the input tensor into non-overlapping windows. NrpcstdSrrCrDrLrMr r!rDszWindowPartition.__init__)rPprcCsf|j\}}}}|}|||||||||}|dddddd}|||||||||}|S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. p (int): Number of partitions. Returns: Tensor: Output tensor with expected layout of [B, H/P, W/P, P*P, C]. rrrsr$rrtrvr/)rLrPrrxCr|Wrzr r r!rSs  zWindowPartition.forward rTrUrVrWrDr rIrSrZr r rMr!rsrcs<eZdZdZddfdd ZeeeeedddZZS) WindowDepartitionzo Departition the input tensor of non-overlapping windows into a feature volume of layout [B, C, H, W]. NrpcstdSrrrrMr r!rDszWindowDepartition.__init__)rPr h_partitions w_partitionsrc Cs`|j\}}}}|} ||} } ||| | | | |}|dddddd}|||| | | | }|S)ar Args: x (Tensor): Input tensor with expected layout of [B, (H/P * W/P), P*P, C]. p (int): Number of partitions. h_partitions (int): Number of vertical partitions. w_partitions (int): Number of horizontal partitions. Returns: Tensor: Output tensor with expected layout of [B, C, H, W]. rrrr$rrsr) rLrPrrrrxryZPPrrzZHPZWPr r r!rSs  zWindowDepartition.forwardrr r rMr!rsrc sheZdZdZeeeeeeefeedej fedej fe e e dd fdd Z e e ddd Z ZS) PartitionAttentionLayera Layer for partitioning the input tensor into non-overlapping windows and applying attention to each window. Args: in_channels (int): Number of input channels. head_dim (int): Dimension of each attention head. partition_size (int): Size of the partitions. partition_type (str): Type of partitioning to use. Can be either "grid" or "window". grid_size (Tuple[int, int]): Size of the grid to partition the input tensor into. mlp_ratio (int): Ratio of the feature size expansion in the MLP layer. activation_layer (Callable[..., nn.Module]): Activation function to use. norm_layer (Callable[..., nn.Module]): Normalization function to use. attention_dropout (float): Dropout probability for the attention layer. mlp_dropout (float): Dropout probability for the MLP layer. p_stochastic_dropout (float): Probability of dropping out a partition. .N) r5r]partition_sizepartition_type grid_size mlp_ratior9r:attention_dropout mlp_dropoutr;rc s(t|||_||_|d||_||_||_|dvrDtd|dkr^||j|_|_ n|j||_|_ t |_ t |_ |dkrtddnt|_|dkrtddnt|_t||t|||dt| |_tt|t||||t|||t| |_t| d d |_dS) Nr)gridwindowz0partition_type must be either 'grid' or 'window'rrrr>r?)rCrDrcr]Z n_partitionsrrrbrgr partition_oprdepartition_oprrrHpartition_swapdepartition_swaprFr[ZDropout attn_layer LayerNormrg mlp_layerrstochastic_dropout) rLr5r]rrrrr9r:rrr;rMr r!rD-s8    z PartitionAttentionLayer.__init__rOcCs|jd|j|jd|j}}t|jd|jdkoL|jd|jdkd|j|j|||j}||}||||}||| |}| |}| ||j||}|S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. Returns: Tensor: Output tensor with expected layout of [B, C, H, W]. rrz[Grid size must be divisible by partition size. Got grid size of {} and partition size of {}) rrr,Z_assertformatrrrrrrr)rLrPZghZgwr r r!rSgs "&  zPartitionAttentionLayer.forward)rTrUrVrWrIstrtuplerrrYrXrDr rSrZr r rMr!rs   :rcsneZdZdZeeeeeedejfedejfeeeeeee eefddfdd Z e e ddd Z Z S) MaxVitLayera MaxVit layer consisting of a MBConv layer followed by a PartitionAttentionLayer with `window` and a PartitionAttentionLayer with `grid`. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. stride (int): Stride of the depthwise convolution. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Ratio of the MLP layer. mlp_dropout (float): Dropout probability for the MLP layer. attention_dropout (float): Dropout probability for the attention layer. p_stochastic_dropout (float): Probability of stochastic depth. partition_size (int): Size of the partitions. grid_size (Tuple[int, int]): Size of the input feature grid. .N)r5r6r8r7rr:r9r]rrrr;rrrc stt}t|||||||| d|d<t||| d|| |tj| | | d |d<t||| d|| |tj| | | d |d<t||_dS)N)r5r6r7r8rr9r:r;ZMBconvr) r5r]rrrrr9r:rrr;Zwindow_attentionrZgrid_attention) rCrDrr3rrrrFrK)rLr5r6r8r7rr:r9r]rrrr;rrrKrMr r!rDsN    zMaxVitLayer.__init__rOcCs||}|Sz Args: x (Tensor): Input tensor of shape (B, C, H, W). Returns: Tensor: Output tensor of shape (B, C, H, W). rK)rLrPr r r!rSs zMaxVitLayer.forward)rTrUrVrWrIrXrrrYrrDr rSrZr r rMr!rs$   ArcsreZdZdZeeeeedejfedejfeeeeee eefee eddfdd Z e e ddd Z ZS) MaxVitBlocka( A MaxVit block consisting of `n_layers` MaxVit layers. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Ratio of the MLP layer. mlp_dropout (float): Dropout probability for the MLP layer. attention_dropout (float): Dropout probability for the attention layer. p_stochastic_dropout (float): Probability of stochastic depth. partition_size (int): Size of the partitions. input_grid_size (Tuple[int, int]): Size of the input feature grid. n_layers (int): Number of layers in the block. p_stochastic (List[float]): List of probabilities for stochastic depth for each layer. .N)r5r6r8r7r:r9r]rrrrinput_grid_sizen_layers p_stochasticrcstt|| ks,td| d|dt|_t| dddd|_t |D]V\}}|dkrddnd}|jt |dkr||n|||||||||| | | |j|d g7_qPdS) Nz'p_stochastic must have length n_layers=z, got p_stochastic=.r$rrr=r)r5r6r8r7rr:r9r]rrrrrr;) rCrDlenrbr ModuleListrKr"r enumerater)rLr5r6r8r7r:r9r]rrrrrrridxrrrMr r!rDs2   zMaxVitBlock.__init__rOcCs|jD] }||}q|Srr)rLrPZlayerr r r!rS-s  zMaxVitBlock.forward)rTrUrVrWrIrXrrrYrlistrDr rSrZr r rMr!rs$   3rcseZdZdZdejddddddfeeefeeeeeeee e e dej fe dej fe e ee e eddfd d Z eed d d ZddZZS)ray Implements MaxVit Transformer from the `MaxViT: Multi-Axis Vision Transformer `_ paper. Args: input_size (Tuple[int, int]): Size of the input image. stem_channels (int): Number of channels in the stem. partition_size (int): Size of the partitions. block_channels (List[int]): Number of channels in each block. block_layers (List[int]): Number of layers in each block. stochastic_depth_prob (float): Probability of stochastic depth. Expands to a list of probabilities for each layer that scales linearly to the specified value. squeeze_ratio (float): Squeeze ratio in the SE Layer. Default: 0.25. expansion_ratio (float): Expansion ratio in the MBConv bottleneck. Default: 4. norm_layer (Callable[..., nn.Module]): Normalization function. Default: None (setting to None will produce a `BatchNorm2d(eps=1e-3, momentum=0.01)`). activation_layer (Callable[..., nn.Module]): Activation function Default: nn.GELU. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Expansion ratio of the MLP layer. Default: 4. mlp_dropout (float): Dropout probability for the MLP layer. Default: 0.0. attention_dropout (float): Dropout probability for the attention layer. Default: 0.0. num_classes (int): Number of classes. Default: 1000. Ng?rsr4i.)r stem_channelsrblock_channels block_layersr]stochastic_depth_probr:r9r8r7rrr num_classesrcstt|d}|dur.ttjddd}t|t|}t|D]J\}}|d|dksl|d|dkrDt d|d|d |d |d qDt t ||dd || d ddt ||dddddd|_ t |dd dd}||_t|_|g|dd}|}td|t|}d}t|||D]X\}}}|jt||| | || || | ||||||||d|jdj}||7}q t tdtt|dt|d|dttj|d|d d|_|dS)Nr$gMbP?g{Gz?)ZepsZmomentumrrz Input size z of block z$ is not divisible by partition size zx. Consider changing the partition size or the input size. Current configuration yields the following block input sizes: rrF)rr:r9r<rAT)rr:r9r<r=r+)r5r6r8r7r:r9r]rrrrrrr)r<) rCrDrrr BatchNorm2dr(rrrbrFrstemr"rrblocksnpZlinspacer1tolistzipr&rrZAdaptiveAvgPool2dZFlattenrrgZTanh classifier _init_weights)rLrrrrrr]rr:r9r8r7rrrrZinput_channelsZblock_input_sizesrZblock_input_sizer5r6rZp_idxZ in_channelZ out_channelZ num_layersrMr r!rDNs        zMaxVit.__init__rOcCs,||}|jD] }||}q||}|Sr)rrr)rLrPblockr r r!rSs     zMaxVit.forwardcCs|D]}t|tjrDtjj|jdd|jdurtj|jqt|tj rrtj |jdtj |jdqt|tj rtjj|jdd|jdurtj|jqdS)Nr`rarr) modules isinstancerrEroZnormal_Zweightr<Zzeros_rZ constant_rg)rLmr r r!rs      zMaxVit._init_weights)rTrUrVrWrZGELUrrIrrXrrrYrDr rSrrZr r rMr!r9s:%  vrF) rrrrrr]weightsprogresskwargsrc Ks|durPt|dt|jd|jdd|jddks>Jt|d|jd|dd} tf||||||| d|} |dur| |j|d d | S) Nr categoriesmin_sizerrrr)rrrrr]rrT)rZ check_hash)rrmetapoprZload_state_dictZget_state_dict) rrrrrr]rrrrmodelr r r!_maxvits&   rc @sHeZdZedeeddejdedddddd d id d d ddZ e Z dS)rz9https://download.pytorch.org/models/maxvit_t-bc5ab103.pthr)Z crop_sizeZ resize_size interpolationirzLhttps://github.com/pytorch/vision/tree/main/references/classification#maxvitz ImageNet-1KgT@g|?5.X@)zacc@1zacc@5gZd;@gK7]@zThese weights reproduce closely the results of the paper using a similar training recipe. They were trained with a BatchNorm2D momentum of 0.99 instead of the more correct 0.01.)rZ num_paramsrZrecipeZ_metricsZ_ops _file_sizeZ_docs)urlZ transformsrN) rTrUrVr rrrZBICUBICr IMAGENET1K_V1DEFAULTr r r r!rs( rZ pretrained)rT)rr)rrrrc Ks2t|}tfdgdgdddd||d|S)a Constructs a maxvit_t architecture from `MaxViT: Multi-Axis Vision Transformer `_. Args: weights (:class:`~torchvision.models.MaxVit_T_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.MaxVit_T_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.maxvit.MaxVit`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.MaxVit_T_Weights :members: @)ri)rrrr g?)rrrr]rrrr)rverifyr)rrrr r r!rs  r)NF)=rd collectionsrcollections.abcr functoolsrtypingrrrnumpyrr,Ztorch.nn.functionalrZ functionalrwr Ztorchvision.models._apir r r Ztorchvision.models._metar Ztorchvision.models._utilsrrZtorchvision.ops.miscrrZ torchvision.ops.stochastic_depthrZtorchvision.transforms._presetsrrZtorchvision.utilsr__all__rrIr"rr(r2rYr3r[rrrrrrrrXboolrrrrr r r r!s^      &&  WI haU. *