a h@sddlZddlmZddlmZmZmZddlZddlm m Z ddlm Z m Z ddl mZmZddlmZddlmZmZdd lmZd d lmZmZmZd d lmZd d lmZmZgdZ ej ej dddZ!ej"#dej ej e$e%ej dddZ&ej"#dGddde j'Z(Gddde j'Z)dHe e e e e$e%e%e$e%e*e*ee ee eej e+e dddZ,ej"#dGddde j'Z-Gd d!d!e-Z.Gd"d#d#e j'Z/Gd$d%d%e/Z0Gd&d'd'e j'Z1e$e%e%e$e%e$e%e$e%e*eee+ee1d( d)d*Z2d+eiZ3Gd,d-d-eZ4Gd.d/d/eZ5Gd0d1d1eZ6Gd2d3d3eZ7Gd4d5d5eZ8Gd6d7d7eZ9eed8e4j:fd9ddd:ee4e+ee1d;dd?Zeed8e8j:fd9ddd:ee8e+ee1d;dDdEZ?eed8e9j:fd9ddd:ee9e+ee1d;dFdGZ@dS)IN)partial)AnyCallableOptional)nnTensor)MLPPermute)StochasticDepth)ImageClassificationInterpolationMode)_log_api_usage_once)register_modelWeights WeightsEnum)_IMAGENET_CATEGORIES)_ovewrite_named_paramhandle_legacy_interface) SwinTransformerSwin_T_WeightsSwin_S_WeightsSwin_B_WeightsSwin_V2_T_WeightsSwin_V2_S_WeightsSwin_V2_B_Weightsswin_tswin_sswin_b swin_v2_t swin_v2_s swin_v2_bxreturnc Cs|jdd\}}}t|ddd|dd|df}|dddddddddf}|dddddddddf}|dddddddddf}|dddddddddf}t||||gd}|S)Nrr.r)shapeFpadtorchcat)r$HW_Zx0x1Zx2Zx3r1Q/var/www/auris/lib/python3.9/site-packages/torchvision/models/swin_transformer.py_patch_merging_pad#s     r3)relative_position_bias_tablerelative_position_index window_sizer%cCsB|d|d}||}|||d}|dddd}|S)Nrrr'r)viewpermute contiguous unsqueeze)r4r5r6Nrelative_position_biasr1r1r2_get_relative_position_bias1s r=csFeZdZdZejfeedejfdfdd Z e dddZ Z S) PatchMergingzPatch Merging Layer. Args: dim (int): Number of input channels. norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. .dim norm_layercsDtt|||_tjd|d|dd|_|d||_dSNrFbiassuper__init__rr@rLinear reductionnormselfr@rA __class__r1r2rHEs  zPatchMerging.__init__r$cCs t|}||}||}|Sz Args: x (Tensor): input tensor with expected layout of [..., H, W, C] Returns: Tensor with layout of [..., H/2, W/2, 2*C] )r3rKrJrMr$r1r1r2forwardLs  zPatchMerging.forward __name__ __module__ __qualname____doc__r LayerNormintrModulerHrrS __classcell__r1r1rNr2r>>s$r>csFeZdZdZejfeedejfdfdd Z e dddZ Z S) PatchMergingV2zPatch Merging Layer for Swin Transformer V2. Args: dim (int): Number of input channels. norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. .r?csDtt|||_tjd|d|dd|_|d||_dSrBrFrLrNr1r2rH`s  zPatchMergingV2.__init__rPcCs t|}||}||}|SrQ)r3rJrKrRr1r1r2rSgs  zPatchMergingV2.forwardrTr1r1rNr2r]Ys$r]T)input qkv_weight proj_weightr<r6 num_heads shift_sizeattention_dropoutdropoutqkv_bias proj_bias logit_scaletrainingr%c $ Cs|j\} }}}|d||d|d}|d||d|d}t|ddd|d|f}|j\}}}}|}|d|krd|d<|d|krd|d<t|dkrtj||d |d fdd}||d||d}|| ||d|d||d|d|}|dddddd | ||d|d|}| d urv| d urv| } | d}| |d| t ||| }| |d|dd|||ddddd}|d|d|d}}}| d ur tj|d d tj|d d d d }tj| td d} || }n"|||d}||d d }||}t|dkr|||f}d|d f|d |d f|d d ff}d|d f|d |d f|d d ff} d}!|D]>}"| D]2}#|!||"d|"d|#d|#df<|!d7}!qҐq|||d|d||d|d}|dddd ||d|d}|d|d}||dktd|dktd}||d||||d|d}||dd}|d ||d|d}tj|d d }tj||| d}||dd |d|d|}t ||| }tj||| d}|| ||d||d|d|d|}|dddddd | |||}t|dkrtj||d|dfdd}|d d d |d |d d f}|S)a Window based multi-head self attention (W-MSA) module with relative position bias. It supports both of shifted and non-shifted window. Args: input (Tensor[N, H, W, C]): The input tensor or 4-dimensions. qkv_weight (Tensor[in_dim, out_dim]): The weight tensor of query, key, value. proj_weight (Tensor[out_dim, out_dim]): The weight tensor of projection. relative_position_bias (Tensor): The learned relative position bias added to attention. window_size (List[int]): Window size. num_heads (int): Number of attention heads. shift_size (List[int]): Shift size for shifted window attention. attention_dropout (float): Dropout ratio of attention weight. Default: 0.0. dropout (float): Dropout ratio of output. Default: 0.0. qkv_bias (Tensor[out_dim], optional): The bias tensor of query, key, value. Default: None. proj_bias (Tensor[out_dim], optional): The bias tensor of projection. Default: None. logit_scale (Tensor[out_dim], optional): Logit scale of cosine attention for Swin Transformer V2. Default: None. training (bool, optional): Training flag used by the dropout parameters. Default: True. Returns: Tensor[N, H, W, C]: The output tensor after shifted window attention. rr)rr)ZshiftsdimsrrCNr')r@gY@)maxggYr^)pri)r(r)r*copysumr+Zrollr7r8Zreshapeclonenumelzero_Zlinearsize normalizeZ transposeclampmathlogexpmatmulZ new_zerosr:Z masked_fillfloatZsoftmaxrer9)$r_r`rar<r6rbrcrdrerfrgrhriBr-r.CZpad_rZpad_br$r/Zpad_HZpad_WZ num_windowslengthqkvqkvattnZ attn_maskZh_slicesZw_slicescounthwr1r1r2shifted_window_attentiontsp#    ,. 0 $ ..$($$((, $rc sneZdZdZdeeeeeeeeeedfdd ZddZ d d Z e j d d d Z e e dddZZS)ShiftedWindowAttentionz/ See :func:`shifted_window_attention`. Tr^r@r6rcrbrfrgrdrec stt|dks"t|dkr*td||_||_||_||_||_t j ||d|d|_ t j |||d|_ | |dS)Nrz.window_size and shift_size must be of length 2rkrD)rGrHlen ValueErrorr6rcrbrdrerrIrproj#define_relative_position_bias_tabledefine_relative_position_index) rMr@r6rcrbrfrgrdrerNr1r2rHs zShiftedWindowAttention.__init__cCsLttd|jddd|jdd|j|_tjj|jdddS)Nrrr{Gz?Zstd) r Parameterr+zerosr6rbr4init trunc_normal_rMr1r1r2r s,z:ShiftedWindowAttention.define_relative_position_bias_tablecCst|jd}t|jd}ttj||dd}t|d}|dddddf|dddddf}|ddd}|dddddf|jdd7<|dddddf|jdd7<|dddddfd|jdd9<|d}| d|dS)NrrijZindexingrr'r5) r+aranger6stackmeshgridflattenr8r9rqregister_buffer)rMZcoords_hZcoords_wZcoordsZcoords_flattenZrelative_coordsr5r1r1r2rs ,((,z5ShiftedWindowAttention.define_relative_position_indexr%cCst|j|j|jSN)r=r4r5r6rr1r1r2get_relative_position_biass z1ShiftedWindowAttention.get_relative_position_biasr#cCsD|}t||jj|jj||j|j|j|j|j |jj |jj |j d S) Args: x (Tensor): Tensor with layout of [B, H, W, C] Returns: Tensor with same layout as input, i.e. [B, H, W, C] )rcrdrerfrgri) rrrweightrr6rbrcrdrerErirMr$r<r1r1r2rS#szShiftedWindowAttention.forward)TTr^r^)rUrVrWrXrZlistboolr|rHrrr+rrrSr\r1r1rNr2rs& rc sdeZdZdZdeeeeeeeeeedfdd ZddZ e j d d d Z e d d dZ ZS)ShiftedWindowAttentionV2z2 See :func:`shifted_window_attention_v2`. Tr^rc stj||||||||dttdt|ddf|_ttj ddddtj ddtj d|d d|_ |r|j j d } |j j | d| jdS) N)rfrgrdre rriTrD)inplaceFrk)rGrHrrr+ryZonesrh SequentialrIZReLUcpb_mlprrErsdatart) rMr@r6rcrbrfrgrdrerrNr1r2rH@s"  "&z!ShiftedWindowAttentionV2.__init__cCstj|jdd |jdtjd}tj|jdd |jdtjd}ttj||gdd}|dddd}|dddddddf|jdd<|dddddddf|jdd<|d9}t |t t |dd }| d |dS) Nrr)Zdtyperrrg?g@relative_coords_table) r+rr6float32rrr8r9r:signlog2absr)rMZrelative_coords_hZrelative_coords_wrr1r1r2r_s$$.. z