fThzpSrSSKrSSKJr SSKJrJrJrJr SSK r SSK J r SSK J s J r SSKJr SSKJr SSKJrJr SS KJr SS KJrJrJr S S KJr S S KJr SSK J!r! \RD"\#5r$\"SS\55r%\"SS\55r&"SS\ RN5r("SS\ RN5r)"SS\5r*"SS\ RN5r+"SS\ RN5r,"SS\ RN5r-\"S S!\55r.\"S"S#\.55r/"S$S%\.5r0/S&Qr1g)'zPyTorch TimesFM model.N) dataclass)CallableOptionalSequenceUnion)FlashAttentionKwargs)BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringcan_return_tuplelogging) LlamaRMSNorm)simple_eager_attention_forward) TimesFmConfigcj\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr g) TimesFmOutput&z Args: loc (`torch.Tensor` of shape `(batch_size, )`): The mean of the time series inputs. scale (`torch.Tensor` of shape `(batch_size,)`): The scale of the time series inputs. Nlocscale) __name__ __module__ __qualname____firstlineno____doc__rrtorchTensor__annotations__r__static_attributes__rc/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/timesfm/modular_timesfm.pyrr&s/#'C%,, &$(E8ELL !(r%rc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \R\ 4\ S'Srg)TimesFmOutputForPrediction4a Args: mean_predictions (`torch.Tensor` of shape `(batch_size, sequence_length)`): The mean predictions of the time series. full_predictions (`torch.Tensor` of shape `(batch_size, sequence_length)`): The full predictions of the time series including the mean and the quantiles. loss (`torch.Tensor` of shape `(1,)`, *optional*, returned when `future_values` is provided): The loss of the TimesFM model. Nmean_predictionsfull_predictionslossr)rrrrr r*rr!r"r#r+r,rfloatr$rr%r&r(r(4sN04hu||,3/3hu||,315D(5u,- .5r%r(c>^\rSrSrSrS\4U4SjjrSSjrSrU=r $) TimesFmMLPEzPax MLP in pytorch.configc>[TU]5 URnURn[R "X#5Ul[R "X25Ul[R"USS9Ul g)Ngư>)normalized_shapeeps) super__init__ hidden_sizeintermediate_sizennLinear gate_proj down_proj LayerNorm layer_norm)selfr1r7r8 __class__s r&r6TimesFmMLP.__init__HsX (( "44;B#4B,, Nr%cURU5nURU5n[R"U5nUR U5nUbUSUSS2SS2S4- -nXQ-$)N?)r>r;Frelur<)r?xpaddingsgate_inpgateoutputss r&forwardTimesFmMLP.forwardQsa??1%~~h'vvd|..&  x1d ';!;N rrrrr rr6rKr$ __classcell__r@s@r&r/r/EsO}Or%r/c2^\rSrSrSrU4SjrSrSrU=r$)TimesFmResidualBlock[zTimesFM residual block.c>[TU]5 XlX lX0l[ R "X5Ul[ R"5Ul [ R "X#5Ul [ R "X5Ul grM) r5r6 input_dims hidden_dims output_dimsr9r: input_layerSiLU activation output_layerresidual_layer)r?rUrVrWr@s r&r6TimesFmResidualBlock.__init__^s_ $&&99Z='')IIk? ii @r%cURU5nURU5nURU5nURU5nX4-$rM)rXrZr[r\)r?rFhiddenoutputresiduals r&rKTimesFmResidualBlock.forwardisI!!!$(""6*&&q)  r%)rZrVrUrXrWr[r\) rrrrr r6rKr$rOrPs@r&rRrR[s! A!!r%rRc\rSrSrSrg)TimesFmRMSNormqrN)rrrrr$rr%r&rdrdqsr%rdc>^\rSrSrSrS\4U4SjjrSSjrSrU=r $)TimesFmPositionalEmbeddinguz6Generates position embedding for a given 1-d sequence.r1c >[TU]5 URnURnURUlUR S-n[ R"[U5[U5- 5[US- S5- nURSU[R"[R"U[RS9U*-5-5 g)Nrrinv_timescalesdtype)r5r6 min_timescale max_timescaler7embedding_dimsmathlogr-maxregister_bufferr!exparangefloat32)r?r1rmrnnum_timescaleslog_timescale_incrementr@s r&r6#TimesFmPositionalEmbedding.__init__xs ,, ,, $00,,1"&((5+?% BV+V"WZ]^lop^prsZt"t   EIIell>&W[rZr&rs s r%cTUcUc [S5eUcH[R"U[RURR S9R S5nO(URS:wa[SUR35eUR"/URQSP76URRSSS5-n[R"[R"U5[R"U5/SS9n[R"USSSURS-45nU$) aGenerates a Tensor of sinusoids with different frequencies. Args: seq_length: an optional Python int defining the output sequence length. if the `position` argument is specified. position: [B, seq_length], optional position for each token in the sequence, only required when the sequence is packed. Returns: [B, seqlen, D] if `position` is specified, else [1, seqlen, D] z.Either position or seq_length must be providedrldevicerrz*position must be 2-dimensional, got shape rdim) ValueErrorr!rurvrjr| unsqueezendimshapeviewcatsincosrDpadro)r? seq_lengthposition scaled_timesignals r&rK"TimesFmPositionalEmbedding.forwards   2MN N  ||JemmDL_L_LfLfgqqrstH ]]a I(..IYZ[ [mm7X^^7Q7$:M:M:R:RSTVWY[:\\ EIIk2EIIk4JKQRSv1a)<)[TU]5 XlSUlURUlX lUR UlURUlURUl URUR-Ul URUR-Ul [R"[R"UR455Ul[R""URURUR-5Ul[R""URURUR-5Ul[R""URURUR-5Ul[R""URUR-UR5Ulg)NT)r5r6r1 is_causalattention_dropoutrnum_attention_heads num_headsr7head_dimq_sizekv_sizer9 Parameterr!emptyscalingr:q_projk_projv_projo_projr?r1rr@s r&r6TimesFmAttention.__init__s/  !'!9!9"33!-- nnt}}4 ~~ 5 ||EKK0@$AB ii 0 0$..4==2PQ ii 0 0$..4==2PQ ii 0 0$..4==2PQ ii >@P@PQ r%queryreturnc[R"UR5RS[R "UR 5- 5nXSSSSS24-$)Ng^$3eG?)rDsoftplusrmulrpsqrtr)r?rrs r& _scale_queryTimesFmAttention._scale_querysH 4<<(,,[499T]];S-STT4q0111r% hidden_statesattention_maskkwargsc ZURSSn/UQSPURP7nURU5RU5R SS5nUR U5nUR U5RU5R SS5nURU5RU5R SS5n[n URRS:wadURRS:Xa-URSS5(a[RS5 O[URRn U "UUUUU4UR(dS O UR S S .UD6upU R""/UQSP76R%5n UR'U 5n X4$) Nr}rreagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rC)dropoutr)rrrr transposerrrrr1_attn_implementationgetlogger warning_oncer trainingrreshape contiguousr) r?rrr input_shape hidden_shape query_states key_states value_statesattention_interface attn_output attn_weightss r&rKTimesFmAttention.forwards $))#2. 88b8$--8 {{=166|DNNqRST ((6 [[/44\BLLQPQR {{=166|DNNqRST (F ;; + +w 6{{//69fjjI\^c>d>d##L '>dkk>^>^&_#$7      % $}}C$2H2H %  % ! "));;;;FFH kk+. ((r%)rr1rr7rrrrrrrrrrrM)rrrrr rintr6r!r"rrr r tuplerKr$rOrPs@r&rrsvR}RR(2%,,25<<226$)||$)!.$)-. $) u||Xell33 4 $)$)r%rc ^\rSrSrSrS\S\4U4SjjrS S\RS\RS\RS \ S \ \ \R\R44 S jjr S rU=r$)TimesFmDecoderLayerzTransformer layer.r1rc>[TU]5 [XS9Ul[ U5Ul[ URURS9Ul g)N)r)r4) r5r6r self_attnr/mlprdr7 rms_norm_epsinput_layernormrs r&r6TimesFmDecoderLayer.__init__sA )&Ff%-f.@.@fFYFYZr%rrrGrrczUnURU5nURUUUS9upXQ-nURXS9nXa4$)N)rrr)rG)rrr)r?rrrGrrascoress r&rKTimesFmDecoderLayer.forwards]!,,]; $')/!/!  !0 B $$r%)rrr)F)rrrrr rrr6r!r"boolrrrKr$rOrPs@r&rrs~[}[[#( %||% %,, %  % x %u||3 4 %%r%rc0\rSrSr\rSrS/rSrSr Sr Sr g) TimesFmPreTrainedModelitimesfmr past_valuesTc[U[R5(a9URRR SUR RS9 g[U[R5(apURRR SUR RS9 URb*[RRUR5 gg[U[R5(aS[RRUR5 [RRUR5 g[U[5(a*[RRUR5 g[U[5(a*[RRUR 5 gg)Nr)meanstd) isinstancer9 Embeddingweightdatanormal_r1initializer_ranger:biasinitzeros_r=ones_rdrr)r?modules r& _init_weights$TimesFmPreTrainedModel._init_weights s fbll + + MM   & &A4;;3P3P & Q  * * MM   & &A4;;3P3P & Q{{&v{{+' - - GGMM&-- ( GGNN6;; '  / / GGNN6== )  0 1 1 GGMM&.. )2r%rN) rrrrr config_classbase_model_prefix_no_split_modulesmain_input_name_supports_sdparr$rr%r&rrs% L!./#ON*r%rc^\rSrSrS\4U4SjjrS\RS\RS\\R\\R\R444Sjr \ \ SS\RS \RS \RS \ S \ S\4 S jj55r\SS\\RS\S\R&S\R(S\ S\\R4 Sjj5r\S\RS\RS\\R\R44Sj5r\S\RS\RS\R4Sj5rSrU=r$) TimesFmModeli"r1c >[TU]U5 Xl[SUR-UR UR S9Ul[R"URUR S9Ul [R"[UR5Vs/sHn[X5PM sn5UlURR"(a[%US9UlUR)5 gs snf)NrrUrWrV)num_embeddings embedding_dim)r1)r5r6r1rR patch_lengthr7r8input_ff_layerr9r freq_sizefreq_emb ModuleListrangenum_hidden_layersrlayersuse_positional_embeddingrg position_emb post_initrs r&r6TimesFmModel.__init__$s   26...**00   F4D4DTZTfTfg mmEJ6KcKcEd eEd  3Ed e   ;; / / :& ID   fsC8inputs patched_padsrc 6URX5up4[R"X@RR:[R "SUR URS9U5nXSS2SS4- USS2SS4- n[R"[R"XRR- 5URR:[R "URRUR URS9U5nXSU44$)zInput is of shape [B, N, P].rCr{N) _timesfm_masked_mean_stdr!wherer1 tolerancetensorrlr|abspad_val)r?rrmusigmarJs r&_forward_transformTimesFmModel._forward_transform7s11&G  KK)) ) LLEKK E  q$}--q$}1EE++ IIf{{222 3dkk6K6K K LL,,GMM'.. Y   U ##r%rpast_values_paddingfreqroutput_hidden_statesc URSnURUSURR5nURUSURR5n[R "[R "US- 5URR:[R"SURURS9U5n[R "[R "XpRR- 5URR:[R"SURURS9U5nURXx5upyUSU- -n[R"Xx/SS9n URU 5n [R"USS9Sn URR (aYUR#U RS5n [R$"U /U RS-SS9n UR'X5n X- n UR)U5nX- n U nUR+U URSURURSS 9n/n/nUR,S URR.HAnU"UUU US 9unnU(aUR1U5 U(dM0UR1U5 MC U(aU /U-nOS n[3UUU(aUOS U SU SS 9$) ax past_values_padding (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The padding indicator of the time series. past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Past values of the time series that serves as input to the model. freq (`torch.LongTensor` of shape `(batch_size,)`): Frequency indices for the time series data. rr}rCrr{r~rT)rsequence_lengthrlr|rN)rrrGr)last_hidden_stater attentionsrr)rrr1rr!rrrrrlr|rr rrminrrconcat_timesfm_shift_padded_seqr_prepare_4d_attention_maskrrappendr)r?rr r rrbsizepatched_inputsrstats concat_inputs model_inputpatched_paddingpos_embf_embrrall_attentionsall_hidden_stateslayerrs r&rKTimesFmModel.forwardKs&!!!$$))%T[[5M5MN*//r4;;;S;ST  IIlS( )DKK,A,A A LLN$8$8AVAV W   {{ IIn{{':':: ;dkk>S>S S LLL$6$6|?R?R S  !% 7 7 U(3+=> >"@bI ))-8  ))Lb9!< ;; / /'' (9(9!(<=GllG9{/@/@/C#CKG44_NG  "K d# $ 88*)//2%% '' 9 [[!@4;;#@#@AE$)+-("3 % !FM !%%f-##!((7B !, 0A A  $ ++):~a(   r%rrrlr|rcUR(a [R"U5RO[R"U5RnUb%UR UR SSSS5nX-nU(a\[R"[R"X4X#S9U-SS9nUR SSX5nUb[R"X5nU$UnU$)a Creates 4D attention mask and combines causal and padding masks if needed. Args: attention_mask: Optional tensor of shape (batch_size, seq_length) containing padding mask sequence_length: Length of the sequence dtype: Data type of the mask device: Device of the mask is_causal: Whether to apply causal masking Returns: 4D attention mask of shape (batch_size, 1, seq_length, seq_length) rrr}r{)diagonal) is_floating_pointr!finforiinforrtriuonesminimum)rrrlr|r min_value causal_masks r&r'TimesFmModel._prepare_4d_attention_masks,/4.E.EEKK&**5;;W\K]KaKa   %+001E1Ea1H!QPRSN+7N ** O=UZ]ffK&**1aRK)!&~!K"-r%paddingc S[R4Sjn[R"SU- SS9nU"U5n[R"URS5nXUSS24nXUSS24nSU- n[R"USS9n [R "U S:H[R "SU RU RS9U 5n [R"Xh-SS9n [R"Xh-S-SS9n X- n X- U S-- n [R "U S :[R "S U RU RS9U 5n [R"U 5nX4$) aCalculates mean and standard deviation of `inputs` across axis 1. It excludes values where `padding` is 1. Args: inputs: A PyTorch tensor of shape [b, n, p]. padding: A PyTorch tensor of shape [b, n, p] with values 0 or 1. Returns: A tuple containing the mean and standard deviation. We return the statistics of the first patch with more than three non-padded values. arrc [R"US:R[R5SS9nUS:R[R5R SS9n[R "US:HUR SS- U5$)Nrrr~r)r!argmaxtoint32sumrr)r1indicesrow_sums r&_get_patch_index?TimesFmModel._timesfm_masked_mean_std.._get_patch_indexskllC1H==#=1EGaxmmEKK0444;G;;w!|SYYq\A-=wG Gr%rrr~rNr{r) r!r"r6rurrrrlr|r)rr/r9pad_sum patch_indicesbidxsr1rmasknum_valid_elements masked_summasked_squared_sum masked_mean masked_var masked_stds r&r%TimesFmModel._timesfm_masked_mean_stdsS  H%,, H ))AKQ/(1  V\\!_-M1,-]A-.3w#YYt3"[[ ! # LL"4":":CUC\C\ ]  YYszq1 "YY q'8a@!5 '<{A~M [[   LLJ$4$4Z=N=N O  ZZ + &&r%r>seqclURup#nUS:HnUR[R5R SS9nSXeR SS9)'[R "X1RS9RSSS5RUSU5nXvSS2SS4- U-nURSU5n U $)zShifts rows of seq based on the first 0 in each row of the mask. Args: mask: mask tensor of shape [B, N] seq: seq tensor of shape [B, N, P] Returns: The shifted sequence. rrr~r})r|N) rr4r!r5r3anyrur|rexpandgather) r>rF batch_sizenum_seq feature_dimnew_maskr7 idx_range shifted_idx shifted_seqs r&r&TimesFmModel._timesfm_shift_padded_seq s,/99( [%)QY++ekk*11a18)+!$$%LL<AA!RKRRS]_acno !1dD=#99WD jjK0 r%)r1rrrr)FF)T)rrrrrr6r!r"rr rr LongTensorrrrK staticmethodrrrlr|rrrr$rOrPs@r&rr"s}&$ll$27,,$ u||U5<<#=>> ?$( #(%* U \\U #--U ll U  U # U  U U n  + .++{{+ +  + %,,  ++Z4'4' 4'QVW\WcWcejeqeqWqQr4'4'l 5<<ELLr%rc^\rSrSrSrS\4U4SjjrS\\RS\\ S\ \R\R\R44Sjr S \RS \ \R\R4S\R4S jr S \RS \RS\R4Sjr\\SS\\RS\\\\R\ 4S\\ S\\RS\\ S\S\S\\S\\S\4Sjj55r\S\RS\ S\\R4Sj5rSrU=r$)TimesFmModelForPredictioni*z/TimesFM model for quantile and mean prediction.r1cB>[TU]U5 XlURUlUR Ul[U5Ul[URUR S[UR5--URS9UlUR5 g)Nrr)r5r6r1context_length context_lenhorizon_length horizon_lenrdecoderrRr7len quantilesr8horizon_ff_layerr)r?r1r@s r&r6"TimesFmModelForPrediction.__init__-s   !00!00#F+ !5))--S9I9I5J1JK00!  r%rr rc ///pTn[U5GHUupgURSn[R"XR-UR UR S9n XR:aURU- n [R"[R"XR UR S9U/SS9n[R"[R"XR U R S9U /SS9n O:XR:a+XpR*SnXRUR-*Sn URU5 URU 5 URX&5 GMX [R"USS9[R"USS9[R"U[RS9RSS54$)a+Formats and pads raw inputs to feed into the model. This function both pads each time series to match the context length, and pads the inputs to meet the SPMD shape requirement. Args: inputs: A list of 1d Tensors. Each Tensor is the context time series of a single forecast task. freq: list of frequencies Returns: A tuple of: - the padded input time series to meet the model required context. - the padding indicator. - the number of padded examples for SPMD so that each core has the same number (a multiple of `batch_size`) of examples. rr{r~Nrkr}r) enumeraterr!zerosr[rlr|rYrr*rstackrr5r) r?rr input_ts input_paddinginp_freqits input_lenr/ num_front_pads r& _preprocess%TimesFmModelForPrediction._preprocess@sw(-/Bv&EA Ikk).>.>">bhhWYW`W`aG+++ $ 0 09 < YY MRTR[R[ \^`aghi))UZZ XXV]VdVd%egn$ouvw---)))+,!$4$4t7G7G$G"H"JK OOB    ) OODG $' KKa ( KK 1 - LL 5 = =b! D  r% model_outputrc URU5nURupEnURXEURR[ URR 5S-5nUupxX8SS2SSS4-USS2SSS4-$)z*Postprocess output of stacked transformer.rN)r_rrr1rZr]r^) r?rnr output_tsbn_rr s r&_postprocess_output-TimesFmModelForPrediction._postprocess_outputks )),7 //aNN1)C)CSI^I^E_bcEcd  D$!4551dD$;N8OOOr% predictionstargetsc,/n[URR5HLupEX!SU4- n[R"US- U-XV-5nUR UR 55 MN [R"U5R 5$)N.r)rbr1r^r!rrrrrd)r?rvrwlossesrhqerrorsr,s r&_quantile_loss(TimesFmModelForPrediction._quantile_losszszdkk334DA3622F99a!ev-qz:D MM$))+ &5{{6"''))r%r window_size future_valuesforecast_context_lenreturn_forecast_on_contexttruncate_negativerrc  Uc URn OUn USRn UV s/sHoU *SPM n n [R"[R"U V s/sHn [R"U 5PM sn 55nUb]/n/n[ U 5HCunn UR URX55 UcM+UR UU/S-5 ME Un UbUnUc$[RS5 S/[U 5-nUcURRnU cURRn URX5unnnURU 5nURU 5nURU 5nUnUR Sn/nUR SUR SUR"-:wa8[%SUR SSUR SSUR"35eURR&nUR"U-S- U-n[)U5GH1nUSS2SUR S24nUSS2U *S24nUSS2U *S24nUR+UUUUU S 9nUR-UR.UR0UR245nU(amUS:XagUSS2SS 2SURR42SS24nUR7UR9S5S UR9S 55nUR;U5 USS2S SU2S4nUSS2S SU2SS24nUR;U5 [R<"UU/S S 9nGM4 U(aF[R<"USS 9SS2SUURR4- UR"-2SS24nO+[R<"USS 9SS2SUR"2SS24nUSS2SS2S4n Ub*U SSS2S 4U SSS2S 4-n USSS2S 4USSS2S 4-nUS:a5U(a.[R>"U S5n [R>"US5nSn!Ub:[@RB"U U5n"UREUSS2SS2SS24U5n#U"U#-n![GWR.U(a URHOSU (a URJOSU UU!S9$s sn fs sn f)aa window_size (`int`, *optional*): Window size of trend + residual decomposition. If None then we do not do decomposition. future_values (`torch.Tensor`, *optional*): Optional future time series values to be used for loss computation. forecast_context_len (`int`, *optional*): Optional max context length. return_forecast_on_context (`bool`, *optional*): True to return the forecast on the context when available, i.e. after the first input patch. truncate_negative (`bool`, *optional*): Truncate to only non-negative values if any of the contexts have non-negative values, otherwise do nothing. output_attentions (`bool`, *optional*): Whether to output the attentions. output_hidden_states (`bool`, *optional*): Whether to output the hidden states. past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Past values of the time series that serves as input to the model. freq (`torch.LongTensor` of shape `(batch_size,)`): Frequency indices for the time series data. Example: ```python >>> from transformers import TimesFmModelForPrediction >>> model = TimesFmModelForPrediction.from_pretrained("google/timesfm-2.0-500m-pytorch") >>> forecast_input = [torch.linspace(0, 20, 100).sin(), torch.linspace(0, 20, 200).sin(), torch.linspace(0, 20, 400).sin()] >>> frequency_input = torch.tensor([0, 1, 2], dtype=torch.long) >>> # Generate >>> with torch.no_grad(): >>> outputs = model(past_values=forecast_input, freq=frequency_input, return_dict=True) >>> point_forecast_conv = outputs.mean_predictions >>> quantile_forecast_conv = outputs.full_predictions ``` Nrrz6No frequency provided via `freq`. Default to high (0).rz=Length of paddings must match length of input + horizon_len: z != z + )rr r rrr}r)axis.r)rrrr*r+r,)&rYr|r!rrdrbextend_timesfm_moving_averagerinfor]r1rrrlr4rr[rrZrr\rtrrrrrsizer concatenatemaximumrDmse_lossr|r(rr)$r?rr r~rrrrrr fcontext_lenr|ririnp_min new_inputs new_freqsrhrerfrg final_outrY full_outputsoutput_patch_lennum_decode_patches step_indexcurrent_paddingdecoder_output fprop_outputs new_full_tsnew_ts mean_outputsr,r quantile_losss$ r&rK!TimesFmModelForPrediction.forwards h '++L/LQ&&0;;{l]^${;))EKK(H22(HIJ  "JI"6*2!!$">">r"OP#$$d1gY]3+ F  < KKP Q3V$D  $ $ = =   '#';;#C#C ,0,<,M!\\$$1"3%9 *N!4400##^%9%9:M *jAo,Ass4Ndkk6N6N4NPQ,QR )11+2B2B12Er;K[K[\]K^_ ##K0#1b*;+;*;Q#>?F'2/@0@/@!(CDK    ,))9f*=BGI=4@ & ,,\BPkDKK$<$<H // Q12X0F VMm+D),>>4E~004:N.66TX))   A<(Hs S S r1c.[R"XS- S4SS5n[R"XRUR S9U- n[R "URSSS5URSSS55R5nX@U- /$)zCCalculates the moving average using PyTorch's convolution function.rrconstantr{r}) rDrr!r*rlr|conv1drsqueeze)r1r~ arr_paddedkernel smoothed_arrs r&r1TimesFmModelForPrediction._timesfm_moving_average(s}UU3q! 4j!D KyyL{Zxx 1b 96;;q!R;PQYY[ L011r%)r1rYr\r_r[)NNNNFFNN)rrrrr rr6rr!r"rrrlrtr|rrrrrr(rKrTlistrr$rOrPs@r&rVrV*s9}&) u||,) 4B%)04.2+0"',0/3b ell+b xellC&7 89:b c] b   - b 'sm b %)b  b $D>b 'tnb  $b b H2U\\22U\\HZ22r%rV)rVrr)2r rp dataclassesrtypingrrrrr!torch.nnr9torch.nn.functional functionalrDmodeling_flash_attention_utilsr modeling_outputsr modeling_utilsr r processing_utilsr utilsrrrllama.modeling_llamar(phi4_multimodal.modeling_phi4_multimodalrconfiguration_timesfmr get_loggerrrrr(Moduler/rRrdrgrrrrrV__all__rr%r&rs7 !66 B/F&>>/U0   H %  )O )  )  6 6  6 ,!299!, \ **Z?)ryy?)D%"))%B*_**8D)DDNG2 6G2T Rr%