a hB6@sUddlZddlZddlmZmZddlZddlmZejdedZiZ e ej j efe d<ehdZeeeegefdd d Zed d d'd dddejejejeejeeeejdddZeeeedddZejeeejdddZejejeeeeeejdddZeeddddZejejeeeeejddd Zed!d d(d ddddd"dd#ejejejeejeejeejeeeeeeeeeeejejejejfd$d%d&ZdS))N)CallableOptional)_dtype_mappings_T)boundONNX_ATEN_DECOMP_TABLE> )op_type opset_versionreturncsttdfdd }|S)zDDecorator to register an ONNX operator with a custom implementation.)funcrcsPd}tjjdd|dd|}|ttttjj|<|||S)NZopsetzonnx::.)Z mutates_args)torchlibraryZ custom_oprgetattropsZonnxZ register_fake)roverloadZtorch_opr r rB/var/www/auris/lib/python3.9/site-packages/torch/onnx/ops/_impl.py decorators  z_onnx_op..decorator)r)r r rrrr_onnx_ops rZRotaryEmbeddingF) interleaved num_headsrotary_embedding_dim)x cos_cache sin_cache position_idsrrrrcsPjd}jd}tjdkrdjd} t|dkfdd| |} |||| g} t| ttjdkddjd} |dkr| }d d d d d d d |f} d d d d d d |d f} |d}|d ur||}||}n|}|}|d d d d d |f}|d d d d d |f}t|d}t|d}|r| d d d d d d dd df}| d d d d d d dd df}ntj| dd d \}}||||}||||}|r t|d }t|d }tj||fd d }t|| j} ntj||fd d } tj| | fd d }tjdkrLt|j}|S) z_RotaryEmbedding-23 https://onnx.ai/onnx/operators/onnx__RotaryEmbedding.html#rotaryembedding-23rr cs djS)NzKnum_heads must be provided for 3D inputs. Received input tensor with shape )shaperrrr>z%rotary_embedding_23..cSsdS)Nzx should be a 4D tensor by nowrrrrrr'Cr(NZdim)r%lenr_checkZreshapeZ unsqueezechunkcat)rr r!r"rrr batch_sizesequence_length hidden_size head_sizeZ new_shapeZx_rotateZ x_not_rotateZrotary_embedding_dim_halfcossinx1Zx2realimagZx_rotate_concatoutputrr&rrotary_embedding_23+sn         "$  r:)scaler3rcCs|dur |Sdt|S)z/Get the scale factor for attention computation.Ng?)mathsqrt)r;r3rrr_get_scale_factorsr>)tensorr0rrcCs:|jd|jd}}||}|||||ddS)z1Reshape 3D tensor to 4D for multi-head attention.r r$)r%view transpose contiguous)r?r0rr1r2r3rrr_reshape_3d_to_4ds rC)QKcurrent_q_num_headscurrent_kv_num_headsr;qk_matmul_output_moderc Cs6|dkrt|||||Stt||ddSdS)z1Get QK output tensor based on the specified mode.rr*N)_compute_qk_output_for_mode_0rZ zeros_likematmulrA)rDrErFrGr;rHrrr_get_qk_output_for_aten_spdas  rL)rFrGrcs"tdkfdddS)z-Validate Group Query Attention configuration.rcsdddS)Nz q_num_heads (z%) must be divisible by kv_num_heads (z ) for GQArrrGrFrrr'r(z-_validate_gqa_configuration..N)rr-)rFrGrrMr_validate_gqa_configurations  rN)rDrErFrGr;rc Cs`|}||kr"||}|j|dd}t||jd}t|}||} ||} t| | ddS)zDHelper function to compute QK output for qk_matmul_output_mode == 0.r r+r#rIr*)repeat_interleaver>r%r<r=rrKrA) rDrErFrGr;ZK_for_qk repeat_factor scale_factor sqrt_scaleQ_scaledK_scaledrrrrJs  rJZ Attention) is_causal kv_num_heads q_num_headsrHr;softcapsoftmax_precision)rDrEV attn_maskpast_key past_valuerVrWrXrHr;rYrZrc( Csd\} }}t|j}|jd}t|jdkrvt|dko>|dkdd|jd}t|||}t|||}t|||}tt|jdkot|jdkot|jdkdd|j|}t| |} |d urtj||g|d n|}|d urtj||g|d n|}||}}|j| }|j| }|j|}|j|}| d koh| dkoh| d uoh|d uph|jtj k}t |||rd }|d ur|jtj kr|n|}tj j j ||||d || t ||kd }t||||| | }n||kr ||}|j|| d }|j|| d }tj|||j|jd }|rht|d uddttj||tj |jd }||td}|d ur|jtj kr||td}n||}t| |jd} t| }!||!}"||!}#t|"|#dd}$|$}|$|}%| dkr|%}| d kr| t|%| }%| dkr|%}| d urr| tvrb|%j}&|%tj| }%tj|%dd }'|'|&}'ntj|%dd }'ntj|%dd }'| dkr|'}t|'|}|dkr|dd !||d}||||fS)zMAttention-23 https://onnx.ai/onnx/operators/onnx__Attention.html#attention-23)r r$r#rr#cSsdS)Nz;q_num_heads and kv_num_heads must be provided for 3D inputsrrrrrr'r(zattention_23..r r)cSsdS)Nz'Q, K, and V should be 4D tensors by nowrrrrrr'r(Nr+rU)r\Z dropout_prVr;Z enable_gqa)dtypedevicecSsdS)Nz'Cannot use both is_causal and attn_maskrrrrrr'@r(z-infrIr*r$)"r,r%rr-rCr>r/cloner_boolrNnnZ functionalZscaled_dot_product_attentionrLrOzerosr`ZtrilZonesZ masked_fillfloatr<r=rKrAtanh-_ATTENTION_23_ALLOWED_INTERMEDIATE_PRECISIONStorZONNX_DTYPE_TO_TORCH_DTYPEZsoftmaxrBr@)(rDrEr[r\r]r^rVrWrXrHr;rYrZZ num_head_dimZ sequence_dimZhead_dimZinput_shape_lenr0Zq_sequence_lengthZ q_head_sizeZ present_keyZ present_valuerFrGZkv_sequence_lengthZ can_use_sdpaZsdpa_attn_maskr9Z qk_outputrPZ attn_biasZ causal_maskrQrRrSrTZqk_matmul_outputZ qk_with_biasZoriginal_dtypeZ qk_softmaxrrr attention_23s       (                         ri)N)NNN)r<typingrrrZtorch.onnx.opsrTypeVarrrdictZ_opsZ OpOverload__annotations__ frozensetrgstrintrZTensorrbr:rer>rCrLrNrJtuplerirrrrs   U