eThRdSSKJr SSKJrJr SSKrSSKJr \"SS55rSS\\RS \\R\ \ 4S \RS \ S \\ 4 S jjrSS\\RS \\R\ \ 4S \RS \ S \\ 4 SjjrSS\RS\R S\\ 4SjjrSS\RS\R S\\ 4SjjrSS \\R\ \ 4S\R S\R&S \ S \\ S\\R4 Sjjrg)) dataclass)OptionalUnionN)is_torchdynamo_compilingc\rSrSr%Sr\\S'\\S'SS\S\\4Sjjr S S\S\S \S \ RS \ \ RS 4S \\ R4 SjjrSS\ RS\S \ RS \\S \ R4 Sjjr\S!S\ R$S \ RS \ RS\S\\4 Sjj5r\SS\ RS \ RS\\4Sjj5r\S\ R*S\4Sj5r\S"S\\ RS\ RS\S\\S\S \4 Sjj5rSrg)#AttentionMaskConvertera A utility attention mask class that allows one to: - Create a causal 4d mask - Create a causal 4d mask with slided window - Convert a 2d attention mask (batch_size, query_length) to a 4d attention mask (batch_size, 1, query_length, key_value_length) that can be multiplied with attention scores Examples: ```python >>> import torch >>> from transformers.modeling_attn_mask_utils import AttentionMaskConverter >>> converter = AttentionMaskConverter(True) >>> converter.to_4d(torch.tensor([[0, 0, 0, 1, 1]]), 5, key_value_length=5, dtype=torch.float32) tensor([[[[-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, 0.0000e+00, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, 0.0000e+00, 0.0000e+00]]]]) ``` Parameters: is_causal (`bool`): Whether the attention mask should be a uni-directional (causal) or bi-directional mask. sliding_window (`int`, *optional*): Optionally, the sliding window masks can be created if `sliding_window` is defined to a positive integer. is_causalsliding_windowNcXlX lURb*URS::a[SURS35egg)NrzaMake sure that when passing `sliding_window` that its value is a strictly positive integer, not ``)r r ValueError)selfr r s ]/var/www/auris/envauris/lib/python3.13/site-packages/transformers/modeling_attn_mask_utils.py__init__AttentionMaskConverter.__init__9sd",    *t/B/Ba/GstxuHuHtIIJK 0H * batch_size query_lengthkey_value_lengthdtypedevicestrreturncUR(d[SURS35eX4nX2- nSnUSS:d URbUR UUUUURS9nU$)z Creates a causal 4D mask of (bsz, head_dim=1, query_length, key_value_length) shape and adds large negative bias to upper right hand triangular matrix (causal mask). z"Please use `to_causal_4d` only if z has `is_causal` set to True.Nrrpast_key_values_lengthr )r r __class__r _make_causal_mask) rrrrrr input_shapercausal_4d_masks r to_causal_4d#AttentionMaskConverter.to_causal_4dBs~~A$..AQQnop p"0 !1!@ r?Q $"5"5"A!33'=#22 4Nrattention_mask_2dcURSU4nSnUSS:d URbKUR(a:Uc [S5eXB- nUR UUUR UURS9nOURb [ S5eURXUSS9RUR 5nUb>URUR5[R"U5R5nUn U $) z Converts 2D attention mask to 4D attention mask by expanding mask to (bsz, head_dim=1, query_length, key_value_length) shape and by adding a large negative bias to not-attended positions. If attention_mask is causal, a causal mask will be added. rNrrzpThis attention mask converter is causal. Make sure to pass `key_value_length` to correctly create a causal mask.rz?Sliding window is currently only implemented for causal masking)tgt_len)shaper r rr!rNotImplementedError _expand_maskto masked_fillbooltorchfinfomin) rr&rrrr"r#rexpanded_attn_maskexpanded_4d_masks rto_4dAttentionMaskConverter.to_4dcs%)..q1<@  Oa 4#6#6#B' G&6%D "!33(//'=#22 4N ,%&gh h"../@Q\]_Q`.add  $ $   %!/!;!;' zz7,ekk%.@.D.DVTLL2v>  )1}&:&:499R=!&LLaPwwu~ !A %99ekk'QVfhlmsuvD  %->BH ::eood%**&MX`aL())zz|   lEKK,>,B,B CD$1$%,,S!WH^>^__rrGr(c2UR5up4UbUOUnUSS2SSSS24RUSX$5RU5nSU- nURUR[R 5[R "U5R5$)zW Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. Nr?)r=rEr,r-r/r.r0r1)rGrr(rFsrc_len expanded_mask inverted_masks rr+#AttentionMaskConverter._expand_masks yy{ $0'gQdA-.55c1gORRSXY m+ (()9)9%**)Eu{{SXGYG]G]^^rrN min_dtypecUR[R:Xa [S5eUR [R "X:HSSS9)5$)a Attend to all tokens in masked rows from the expanded attention mask, for example the relevant first rows when using left padding. This is required by F.scaled_dot_product_attention memory-efficient attention path. Details: https://github.com/pytorch/pytorch/issues/110213 `expanded_mask` is [bsz, num_masks, tgt_seq_len, src_seq_len] or [bsz, tgt_seq_len, src_seq_len]. `attention_mask` is [bsz, src_seq_len]. The dimension num_masks of `expanded_mask` is most often 1, but it can also be the number of heads in the case of alibi attention bias. For example, if `expanded_mask` is (e.g. here left-padding case) ``` [[[[0, 0, 0], [0, 0, 0], [0, 0, 1]]], [[[1, 0, 0], [1, 1, 0], [1, 1, 1]]], [[[0, 0, 0], [0, 1, 0], [0, 1, 1]]]] ``` then the modified `expanded_mask` will be ``` [[[[1, 1, 1], <-- modified [1, 1, 1], <-- modified [0, 0, 1]]], [[[1, 0, 0], [1, 1, 0], [1, 1, 1]]], [[[1, 1, 1], <-- modified [0, 1, 0], [0, 1, 1]]]] ``` z\AttentionMaskConverter._unmask_unattended expects a float `expanded_mask`, got a BoolTensor.rT)r9keepdim)rr/r.rmulall)rNrQs r_unmask_unattended)AttentionMaskConverter._unmask_unattendedsNT   %** ,n   %))M,FBX\"]!]^^rattention_mask inputs_embeds is_trainingcURSURSpeXb-n[RR5=(d5 [ U[R R 5=(d [5nSn Uc%U(dU(dUS:XdXv:Xa UbXs:aSn U $UbXs:aL[UR5S:XagU(d+[R"US:H5(a US:XdXv:XaSn U $)a Detects whether the optional user-specified attention_mask & the automatically created causal mask can be ignored in case PyTorch's SDPA is used, rather relying on SDPA's `is_causal` argument. In case no token is masked in the `attention_mask` argument, if `query_length == 1` or `key_value_length == query_length`, we rather rely on SDPA `is_causal` argument to use causal/non-causal masks, allowing to dispatch to the flash attention kernel (that can otherwise not be used if a custom `attn_mask` is passed). rrFT) r)r/jit is_tracing isinstancefxProxyrlenrU) rXrYrr rZ_rrr^ignore_causal_masks r_ignore_causal_mask_sdpa/AttentionMaskConverter._ignore_causal_mask_sdpas$(--a0-2E2Ea2H<'@YY))+vz-/Xv\t\v "  !J!Q&*:*J#+/?/P%)""! #'7'H>''(A-EIIn.A$B$B1$(8(H)-&"!rr r N)cpurN)NF)__name__ __module__ __qualname____firstlineno____doc__r.__annotations__intrrr/rrrTensorr$r4 staticmethodSizer!r+ FloatTensorfloatrVre__static_attributes__rrr r s<O$ .3   {{  ellE)*  %,, L+/ -  <<- - {{ - #3- -  - ^ '((, ``{{` `!$ ` ! ``B _5<< _ _hsm _ _._((._._._` )-! 7" .7"||7"!$7"! 7"  7"  7"7"rr rXr"rYrr c[SUS9nUSU-nUb8[UR5S:XaURXSXbRS9nU$Ub[UR5S:XaUSSUSU4n[ UR5U:wa%[ S [ UR5S US 35eS U- nURUR[R5[R"UR5R5nU$URUSUSXbRURS 9nU$)a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)` Args: attention_mask (`torch.Tensor` or `None`): A 2D attention mask of shape `(batch_size, key_value_length)` input_shape (`tuple(int)` or `list(int)` or `torch.Size`): The input shape should be a tuple that defines `(batch_size, query_length)`. inputs_embeds (`torch.Tensor`): The embedded inputs as a torch Tensor. past_key_values_length (`int`): The length of the key value cache. sliding_window (`int`, *optional*): If the model uses windowed attention, a sliding window should be passed. Trgr)rrr\rrz#Incorrect 4D attention_mask shape: z ; expected: .rLr8)r rbr)r4rtuplerr-r,r/r.r0r1r$r) rXr"rYrr attn_mask_converterrexpected_shaperOs r!_prepare_4d_causal_attention_maskr.sg.14P^_"2)??!c.*>*>&?1&D,22 O>NViVi3 ( #  #N,@,@(AQ(F%a.![^=MN %% &. 85eNrs"" 8 T"T" T"x%) 1U\\*1uzz5$./1<<1 1 SM 1t%) 7U\\*7uzz5$./7<<7 7 SM 7t XU\\ X%++ XPXY\P] X \ell\5;;\YabeYf\:#$$( uzz5$./ ;; LL  SM  ell r