JThZ4SrSSKJrJr SSKJr SSKJr SSKrSSK J s J r SSK JrJrJrJr SSKJr SSKJrJrJrJr /S Qr\R4R7\5 \R4R7\5 \R4R7\5 \R4R7\5 "S S \5r"S S \R:5rS\4SjrS\4Sjr g)zCDefines bias subclasses that work with scaled_dot_product_attention)autoIntEnum)Optional)warnN)can_use_efficient_attentioncan_use_flash_attentionis_flash_attention_available SDPAParams)_raise_kernel_warnings)_calculate_scale_input_requires_grad_postprocess_flash_output_validate_sdpa_input)causal_upper_leftcausal_lower_right CausalVariant CausalBiasc4\rSrSrSr\"5r\"5rSrg)r!a Enum for causal variants used in attention mechanisms. Defines two types of causal biases: `UPPER_LEFT`: Represents upper-left triangular bias for standard causal attention. The equivalent pytorch code for constructing this bias is: .. code-block:: python torch.tril(torch.ones(size, dtype=torch.bool)) For instance, with `shape=(3,4)`, the materialized bias tensor will be: .. code-block:: text [[1, 0, 0, 0], [1, 1, 0, 0], [1, 1, 1, 0]] `LOWER_RIGHT`: Represents lower-right triangular bias, the include values are aligned to the lower right corner of the matrix. The equivalent pytorch code for constructing this bias is: .. code-block:: python diagonal_offset = size[1] - size[0] torch.tril( torch.ones(size, dtype=torch.bool), diagonal=diagonal_offset, ) For instance, with `shape=(3,4)`, the materialized bias tensor will be: .. code-block:: text [[1, 1, 0, 0], [1, 1, 1, 0], [1, 1, 1, 1]] Note that these variants are equivalent to each other when the sequence lengths of the query and key/value tensors are equal since the triangular matrix is square. .. warning:: This enum is a prototype and subject to change. N) __name__ __module__ __qualname____firstlineno____doc__r UPPER_LEFT LOWER_RIGHT__static_attributes__rO/var/www/auris/envauris/lib/python3.13/site-packages/torch/nn/attention/bias.pyrr!s.`J&Krrc\rSrSrSrS\S\S\4SjrS\RS\R4S jr S\RS\R4S jr SS\ \RS\R4S jjr\SS \RS\RS\RSSS\S\S\ \S\S\R4Sjj5r\SSj5rSrSrg )rVa A bias representing causal attention patterns. For an overview of the bias structure, see the :class:`CausalVariant` enum. This class is used for defining causal (triangular) attention biases. For construing the bias, there exist two factory functions: :func:`causal_upper_left` and :func:`causal_lower_right`. Example: .. code-block:: python from torch.nn.attention.bias import causal_lower_right bsz, num_heads, seqlen_q, seqlen_kv, head_dim = 32, 8, 4, 12, 8 # Create a lower-right causal bias attn_bias = causal_lower_right(seqlen_q, seqlen_kv) q = torch.randn(bsz, num_heads, seqlen_q, head_dim, device="cuda", dtype=torch.float16) k = torch.randn(bsz, num_heads, seqlen_kv, head_dim, device="cuda", dtype=torch.float16) v = torch.randn(bsz, num_heads, seqlen_kv, head_dim, device="cuda", dtype=torch.float16) out = F.scaled_dot_product_attention(q, k, v, attn_bias) .. warning:: This class is a prototype and subject to change. variant seq_len_q seq_len_kvc[U[5(deXlX lX0lX#:a!U[R :Xa [ S5 ggg)a Initializes the CausalBias instance with a specified variant and sequence lengths. Args: variant (CausalVariant): The type of causal bias to use (either UPPER_LEFT or LOWER_RIGHT). seq_len_q (int): The sequence length of the query tensor. seq_len_kv (int): The sequence length of the key/value tensor. Raises a warning if the LOWER_RIGHT variant is used with seq_len_q > seq_len_kv, as it may produce NaNs. zTLower right causal bias will produce NaNs in the output when seq_len_q > seq_len_kv!N) isinstancerr#r$r%rr)selfr#r$r%s r __init__CausalBias.__init__qsM'=1111 "$  !g1J1J&J f 'K !rdevicereturnc [R"[R"URURU[R S95$)zUpper left causal biasr+dtype)torchtrilonesr$r%boolr(r+s r _upper_leftCausalBias._upper_lefts1zz JJt~~tvUZZ X  rc URUR- n[R"[R"URURU[R S9US9$)zLower right causal biasr.)diagonal)r%r$r0r1r2r3)r(r+diagonal_offsets r _lower_rightCausalBias._lower_rightsK//DNN:zz JJejj %   rNcUc[R"S5nUR[R:XaUR U5$UR[R :XaURU5$g)aX Materializes the causal bias into a tensor form. Depending on the variant, this method generates either an upper-left or lower-right triangular matrix to represent the causal bias. Args: device (Optional[torch.device]): The device on which to create the tensor. Defaults to CPU. Returns: torch.Tensor: The materialized bias tensor. Ncpu)r0r+r#rrr5rr:r4s r _materializeCausalBias._materializesb >\\%(F <<=33 3##F+ + \\]66 6$$V, ,7rquerykeyvalue attn_mask dropout_p is_causalscale enable_gqacU(a [S5eURUR:XdUR[R :Xa[ R"UUUSUSUUS9$UR[R:XGaT[XUSXEU5 [XUSXEU5n[U5(Ga8URS5S-S:gn URS5n [X5n U (a[RR R#USSURS5S-- 45n[RR R#USSURS5S-- 45n[RR R#USSURS5S-- 45n[R$R&R)UUUUSSU S 9Sn [+X5$[-U5(aSn [/XU5(aSn [R$R&R1UR3S S 5UR3S S 5UR3S S 5SSSSSU[5UR5U USS 9 SR3S S 5$[7U5 [ R"UUUUR9UR:5USUUS9$[S UR35e)a Handles the logic for computing attention with the specified causal bias. Args: query (Tensor): Query tensor; shape :math:`(N, ..., L, E)`. key (Tensor): Key tensor; shape :math:`(N, ..., S, E)`. value (Tensor): Value tensor; shape :math:`(N, ..., S, Ev)`. attn_mask (CausalBias): The type of causal attention to apply. A boolean mask where a value of True indicates that the element *should* take part in attention. A float mask of the same type as query, key, value that is added to the attention score. dropout_p (float): Dropout probability; if greater than 0.0, dropout is applied is_causal (bool): If true, assumes upper left causal attention masking and errors if both attn_mask and is_causal are set. scale (optional float): Scaling factor applied prior to softmax. If None, the default value is set to :math:`\frac{1}{\sqrt{E}}`. enable_gqa (optional bool): If set to True, Grouped Query Attention (GQA) is enabled, by default it is set to False. Returns: output (Tensor): Attention output; shape :math:`(N, ..., L, Ev)`. Raises: ValueError: If the causal bias variant is not a CausalVariant type. z.CausalBias should not be used with causal=TrueNT)rCrDrErFrGrF)rEreturn_debug_maskrF) bias cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_krDcustom_mask_typecompute_log_sumexprFseqlen_kzr+)r@rArBrCrDrErFrG sdpa_params needs_padding og_head_sizeog_scaleoutrTs r _dispatchCausalBias._dispatchsF MN N   9#7#7 7  M$<$<<11#%    -";"; ; UD)PU V$E4zK'{33 % 2 2a 7 $zz"~ +L@ !HH//33EAq5::b>TUCU?U;VWE((--11#1sxx|a?O;O7PQC!HH//33EAq5::b>TUCU?U;VWEiinnHH"&+"I1CC*;77%*"'E::)-&yy~~BBOOAq)MM!Q'OOAq)!%!%!%!%'%():):%;'9!CYq!_% '{355'44U\\B'#)  NyO`O`Nab rcUc0nU[RRR:wa [ S5eUR "U0UD6$)zjDefines the behavior of torch.nn.functional.scaled_dot_product_attention when the attn_bias is an AttnBiasz5CausalBias only supports scaled_dot_product_attention)r0rZr[rXNotImplementedErrorrh)clsfunctypesargskwargss r __torch_function__CausalBias.__torch_function__sL >F 588&&CC C%G }}d-f--rc>UR5R5$N)r>__repr__)r(s r ruCausalBias.__repr__$s  "++--r)r%r$r#rt)gFNF)rN)rrrrrrrbr)r0r+Tensorr5r:rr> staticmethodfloatr3rh classmethodrqrurrrr rrVs(4 #3( %,, 5<<  5<< ELL -8ELL#9-U\\-( !% m||m \\m||m m  m  mmm mm^...rrr,cj[U5S:XdS5eUup[[RX5$)a Creates an upper-left triangular causal bias. This function generates a upper-left triangular matrix to represent causal attention bias with a diagonal offset set so that the inclusive values are aligned to the upper left corner of the matrix. This equivalent to the `is_causal=True` argument in `scaled_dot_product_attention`. The equivalent pytorch code for constructing this bias is: .. code-block:: python torch.tril(torch.ones(size, dtype=torch.bool)) For instance, with `shape=(3,4)`, the materialized bias tensor will be: .. code-block:: text [[1, 0, 0, 0], [1, 1, 0, 0], [1, 1, 1, 0]] Args: size: The size of the bias matrix. Returns: CausalBias: The UPPER_LEFT triangular causal bias variant. rMz*causal_upper_left only supports 2D tensors)lenrrrrYr$r%s r rr(s58 t9>GGG> I m.. FFrcj[U5S:XdS5eUup[[RX5$)a Creates a lower-right triangular causal bias. This function generates a lower-right triangular matrix to represent causal attention bias with a diagonal offset set so that the inclusive values are aligned to the lower right corner of the matrix. The equivalent pytorch code for constructing this bias is: .. code-block:: python diagonal_offset = size[1] - size[0] torch.tril( torch.ones(size, dtype=torch.bool), diagonal=diagonal_offset, ) For instance, with `shape=(3,4)`, the materialized bias tensor will be: .. code-block:: text [[1, 1, 0, 0], [1, 1, 1, 0], [1, 1, 1, 1]] Args: size: The size of the bias matrix. Returns: CausalBias: The LOWER_RIGHT triangular causal bias variant. rMz+causal_lower_right only supports 2D tensors)r|rrrr}s r rrIs5> t9>HHH> I m// GGr)!renumrrtypingrwarningsrr0torch.nn.functionalrZr[rWtorch.backends.cudarrr r torch.nn.attentionr torch.nn.attention._utilsr r rr__all___dynamoallow_in_graphrrwrrrrrr rsI  6 U 9: 45 89 Z(2G2jO.O.dG GB!H!Hr