o wZh@sddlZddlmZddlZddlZddlmmZddl m Z m Z m Z m Z mZmZmZmZddlmZddlmZeeZ    dCd ejd ejd ejd eejfd dZdDdedefddZdDdedefddZdDdedefddZ dDdejdedefddZ dDdefddZ!dDdedefddZ"dDdedefdd Z#dDdedefd!d"Z$dDdedefd#d$Z%d%d&Z&d'ejde'eje(e(ffd(d)Z)d*ejfd+d,Z*d*ejd-e(d.e(d/e(dejf d0d1Z+d2d3Z,d*ejd4e(d5edejfd6d7Z-d8d9Z.d:ejfd;d<Z/d=d>Z0d ejd ejd ejd eejde'ejejejeejff d?d@Z1     dEd ejd ejd ejd eejfdAdBZ2dS)FN)Optional)can_use_cudnn_attentioncan_use_efficient_attentioncan_use_flash_attentioncudnn_sdp_enabledflash_sdp_enabledmath_sdp_enabledmem_efficient_sdp_enabled SDPAParams) SDPBackend) NestedTensorFquerykeyvalue attn_maskcCsNt|trt|trt|ts td|jd|jd|jd|j|jks,|j|jkr=td|jd|jd|jd|j|jksI|j|jkrZtd|jd |jd |jd|d ksl|d ksl|d krtd |d |d|d|j|jks|j|jkrtd|jd|jd|jd|durtddS)NzNExpected query, key, and value to be nested tensors, but got query.is_nested: z, key.is_nested: z, and value.is_nested: z instead.zLExpected query, key, and value to have the same dtype, but got query.dtype: z , key.dtype: z, and value.dtype: zSExpected query, key, and value to have the same device type, but got query.device: z, key.device: z, and value.device: zUExpected query, key, and value to all be at least 3 dimensional, but got query.dim: z , key.dim: z and value.dim: z[Expected query, key, and value to all be ragged on the same dimension, but got ragged dims z, z, and z, respectively.zMasks are not yet supported!) isinstancer ValueError is_nesteddtypedevicedim _ragged_idxtorchbool)rrrr dropout_p is_causalscaler J/var/www/auris/lib/python3.10/site-packages/torch/nested/_internal/sdpa.py_validate_sdpa_inputst $r"paramsreturncCs4|jd}|jd}|jd}||ko||kS)Nr)rsizerr)r#debug q_batch_size k_batch_size v_batch_sizer r r!_check_batch_size_nestedMs   r*cCld}|jd}|jd}|jd}||ko||k}|r(|ddkr(||ks4|r2td|||dSdS)NrzFor NestedTensor inputs, Flash attention requires q,k,v to have the same last dimension and to be a multiple of 8 and less than or equal to 256. Got Query.size(-1): %d, Key.size(-1): %d, Value.size(-1): %d instead.FTrr%rrlogwarningr#r&max_sizeZquery_size_lastZ key_size_lastZvalue_size_lastZsame_head_dim_sizer r r!!_check_head_dim_size_flash_nestedZ&    r4cCr+)Nr-r.rzFor NestedTensor inputs, cuDNN attention requires q,k,v to have the same last dimension and to be a multiple of 8 and less than or equal to 128. Got Query.size(-1): %d, Key.size(-1): %d, Value.size(-1): %d instead.FTr/r2r r r!!_check_head_dim_size_cudnn_nestedtr5r7param param_namecCsTt|ts Jd|jdkr|rtd|dS|dkr(|r&td|dSdS)Nzparam should be a jagged NTr zMFused kernels do not support ragged num_head_dims, %s has a ragged num_heads.FrzAFused kernels do not support seq_len == 0, %s has a seq len of 0.T)rr rr0r1_get_min_seqlen)r8r9r&r r r!:_check_for_seq_len_0_and_consistent_head_dim_nested_helpers   r;c Cs`t|||}||kr|dks||kr|dks||kr.|dkr.|r,td|||||||dSdS)Nr zzBoth fused kernels require query, key and value to have broadcastable %s, got Query %s %d, Key %s %d, Value %s %d instead.FT)maxr0r1)Zq_sizeZk_sizeZv_sizer9r&r3r r r!_try_broadcast_param_sizes"  r=c Cs|jjr t|jd|nd}|sdS|jjrt|jd|nd}|s"dS|jjr-t|jd|nd}|s3dS|jd}|jd}|jd}||koL||k}|sl|jjs[|jjs[|jjrd|rbtddSt |||d|SdS) NrTFrrr zFBoth fused kernels do not support training with broadcasted NT inputs.z num heads) rrr;rrr% requires_gradr0r1r=) r#r&Z q_is_safeZ k_is_safeZ v_is_safe q_num_heads k_num_heads v_num_headsZsame_num_headsr r r!_check_for_seq_len_0_nestedsX    rBcCs(tttf}|D] }|||sdSqdSNFT)r*r4rBr#r& constraints constraintr r r!_can_use_flash_sdpa_jaggeds rGcCs&ttf}|D] }|||sdSqdSrC)r*rBrDr r r!_can_use_efficient_sdpa_jaggeds rHcCsd|jddr|jddr|jdds$|r"tddS|jr0|r.tddSdS)Nr zGIf inputs are nested tensors they must be contiguous after transposing.FzENested tensors for query / key are not supported when is_causal=True.T)r transpose is_contiguousrrr0r1r)r#r&r r r!_can_use_math_sdpa_jaggeds$rLc CsLtststststjStjtjtjtj f}t |||||||}|D]E} | tj kr5t |r5tj S| tjkrGt |rGt |rGtjS| tjkrYt|rYt|rYtjS| tjkrjtrjt|rjtjSq%tdt|ddt|ddtdt |ddt |ddtdt|ddtdt |ddtjS)Nz)Memory efficient kernel not used because:T)r&z(Flash attention kernel not used because:z'Math attention kernel not used because:z(cuDNN attention kernel not used because:)rr rrr ERRORFLASH_ATTENTIONEFFICIENT_ATTENTIONMATHCUDNN_ATTENTIONr rrrGrrHrLr0r1) rrrrZdropoutr enable_gqaZorderingr#backendr r r!_select_sdp_backend%sT                   rTqkvcCst|ts td|dur&|jtj|jd}| }| j d}n| djtj|jd}| }t |d}|||fS)Nzs  z+_is_computing_meta_flops..F) rZjitZ is_scriptingrrrZ_python_dispatchZ _get_current_dispatch_mode_stackany)rZtorch_dispatch_mode_stackr r r!_is_computing_meta_flopss rcsN|jjt|s ts||||fSfdd}||||||||fS)a* [Autocasting SDPA for NJT] Normal autocasting doesn't work for NJT+SDPA right now: * NJT intercepts the __torch_function__ call for scaled_dot_product_attention, which happens before we get to any aten ops or dispatcher logic; then the torch_function logic calls into efficient attention or flash attention. So, autocasting on the scaled_dot_product_attention op won't work because we never see that aten op. * If we put autocasting on `_flash_attention_forward`, then we'll get autocasting to run, but the kernel selection logic in torch_function handling (ie. jagged_scaled_dot_product_attention) won't work correctly: the kernel selection logic will run before autocasting, and choose a kernel based on the un-autocasted dtypes; but then autocasting will run and the actual attention computation will happen in a different dtype. An alternative is to just change the backend selection logic for SDPA+NJT to be autocast-aware and rely on autocasting to do the actual conversions for flash attention / efficient attention. However, by manually doing the actual autocast before the backend selection, we ensure that the autocast handling for backend selection doesn't diverge from the autocast handling for the actual dtype conversions. csB|dur|St}|jjr|j|ks|jtjkr|S||Srf)rZget_autocast_dtyperZis_floating_pointZfloat64rX)rZ target_dtypeZ device_typer r!cvts    z_autocast..cvt)rrrrZis_autocast_enabled)rrrrrr rr! _autocasts    rc2Cst||||\}}}}t|||||||t|tr$t|tr$t|ts&Jddlm}|dkrm|dkrm|dkrm|jdkrmtj | | | t|trW| n||||d} || | | |j |jdS|jpu|jpu|j} t|||||||} t|rtj} | tjkr|d} t|dd } t|dd }t|dd }t||}t| ||\}}}}}}}}tjjj|||||||||d |d \}}}}}||fi|dd }t|| S| tjkr%t|||\}}} }}}}}tjjj| d| d| dd|||||t!|| |d \}}!}"}#}$}||"dfi|dd S| tj#krbt|||\}}} }}}}}tjjj$||| |||||| ||d |d \ }}%}&}'}$}(}"}#})||fi|dd S| tj%kr| }*| }+|j },|j}-|j&d}.|j&d}/d d }0|0|}|0|}|0|}tj'|||||||d d}1|1dd ( }1|1)d|.|/}1||1|*|+|,|-ddd }1|1St*d)Nr)'nested_view_from_values_offsets_lengthsrr )rrrr)rir^r-r.F)rrIcSsd|jdd|jdd}t|dd}|jt|dd}tjt|}|dd}|S)Nr r-rIr)r) _offsetsrrJrZsplitlistnestedZas_nested_tensorrk)Zjagged_layout_ntrVrJZ tensor_listZ strided_ntr r r! get_strided_layout_nested_tensors zMjagged_scaled_dot_product_attention..get_strided_layout_nested_tensor)rVrir^z=No viable backend for scaled_dot_product_attention was found.)+rr"rr Z$torch.nested._internal.nested_tensorrrrFZscaled_dot_product_attentionrZrWrVZ_maybe_min_seqlenZ_maybe_max_seqlenr>rTrr rNr%rzr{rtropsZatenZ_flash_attention_forwardrJr~rOZ_efficient_attention_forwardZ unsqueezer\ZsqueezerQZ_cudnn_attention_forwardrP_sizeZ"_scaled_dot_product_attention_mathrkrgrj)2rrrrrrrrRroutputZcompute_logsumexpZbackend_choicer}Z query_paddedZ key_paddedZ value_paddedZog_scalerprqrrrlrnrmrorsZ attentionZ _logsumexpZ _philox_seedZ_philox_offsetZ_debug_attn_maskZquery_reshapedZ key_reshapedZvalue_reshapedZ log_sumexpseedoffsetZ max_seqlen_qZ logsumexpZ cum_seqlen_qZ cum_seqlen_kvZ max_seqlen_kv_rWZ q_lengthsrir^Zd1Zd2rZattn_outr r r!#jagged_scaled_dot_product_attentionsx  .                  r)NrFN)F)NrFNF)3loggingtypingrrZtorch.nnZtorch.nn.functionalrwrxrZtorch.backends.cudarrrrrrr r Ztorch.nn.attentionr Z nested_tensorr getLogger__name__r0ZTensorr"rr*r4r7strr;r=rBrGrHrLrTtupler\r_rbrhrtrzr{r~rrrr r r r!s (   3  ;   / J  2