fThs` PSrSSKrSSKJr SSKJrJrJr SSKrSSKJ r SSK J r SSK J r SS KJr SS KJrJrJr SS KJrJrJr S S KJr \R4"\5r"SS\ R:5r"SS\ R:5rS\S\R@S\!S\\"S\#S\$4 Sjjr%S]S\R@S\\"\$4S\\"S\$4Sjjr&"SS\ R:5r'"SS\ R:5r("SS\ R:5r)\"S S!\55r*"S"S#\ R:5r+"S$S%\ R:5r,"S&S'\*5r-\"S(S)\55r.\"S*S+\55r/\"S,S-\55r0\"S.S/\55r1\"S0S1\55r2\"S2S3\55r3S4\RhRjS5\R@S6\R@4S7jr6S^S8\R@S9\\R@S6\R@4S:jjr7"S;S<\ R:5r8"S=S>\ R:5r9"S?S@\ R:5r:"SASB\ R:5r;\"SCSD\*55r<"SESF\ R:5r=\"SGSH9"SISJ\*55r>"SKSL\ R:5r?\"SMSH9"SNSO\*55r@\"SPSH9"SQSR\ R:55rA\"SSSH9"STSU\*55rB"SVSW\ R:5rC\"SXSH9"SYSZ\*55rD/S[QrEg)_zPyTorch PatchTST model.N) dataclass)OptionalTupleUnion)nn)ACT2CLS)BaseModelOutput)PreTrainedModel)NegativeBinomialOutput NormalOutputStudentTOutput) ModelOutputauto_docstringlogging)PatchTSTConfigc^\rSrSrSrSS\S\S\S\S\S\S \\ 4U4S jjjr S \ RS \S \4Sjr SS\ RS\\ RS\\\ RS\\ RS\\ RS\S\\ R\\ R\\\ R44SjjrSrU=r$)PatchTSTAttention$z=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsdropout is_decoderbias is_causalconfigc>[TU]5 XlX lX0lX-UlXplUR U-UR:wa[SURSUS35eUR S-UlX@l X`l [R"XUS9Ul [R"XUS9Ul[R"XUS9Ul[R"XUS9Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: ).g࿩r)super__init__rrrhead_dimr ValueErrorscalingrrrLineark_projv_projq_projout_proj) selfrrrrrrr __class__s f/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/patchtst/modeling_patchtst.pyr"PatchTSTAttention.__init__'s "" !.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 4@ ii 4@ ii 4@  )TB tensorseq_lenbszcURX2URUR5RSS5R 5$)Nr)viewrr# transpose contiguous)r+r0r1r2s r-_shapePatchTSTAttention._shapeFs5{{3GQQRSUVWbbddr/ hidden_stateskey_value_statespast_key_valueattention_masklayer_head_maskoutput_attentionsreturnc USLnUR5upn URU5UR-n U(a2Ub/USRSURS:Xa USn USn GO U(aEUR UR U5SU5n UR UR U5SU5n OUbyUR UR U5SU5n UR UR U5SU5n [R"USU /SS9n [R"USU /SS9n ODUR UR U5SU5n UR UR U5SU5n UR(aX4nXR-SUR4nUR XU5R"U6n U R"U6n U R"U6n U RS5n[R"XRSS55nUR5XR-X4:wa-[!SXR-X4SUR535eUbvUR5USX4:wa"[!S USX4SUR535eURXRX5U-nURXR-X5n["R$R'USS9nUbUR5UR4:wa*[!S UR4SUR535eURSSSS5URXRX5-nURXR-X5nU(a;URXRX5nURXR-X5nOSn["R$R)UUR(UR*S 9n[R"UU 5nUR5XR-XR4:wa7[!S XR-XR4SUR535eURXRXR5nURSS5nURXUR,5nUR/U5nUUU4$) z#Input shape: Batch x Time x ChannelNrr4rdimz$Attention weights should be of size z , but is z!Attention mask should be of size z/Head mask for a single layer should be of size )ptrainingz `attn_output` should be of size )sizer)r%shaper8r'r(torchcatrrr#r5reshapebmmr6r$r functionalsoftmaxrrFrr*)r+r:r;r<r=r>r?is_cross_attentionr2tgt_len_ query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r-forwardPatchTSTAttention.forwardIs.T9',,.a{{=1DLL@ *q!''*.>.D.DQ.GG(*J)!,L T[[1A%BBLJ;;t{{3C'Db#NL  'T[[%?SIJ;;t{{='A2sKLN1$5z#BJJ 99nQ&7%FANLT[[%?SIJ;;t{{='A2sKL ??)7NNN*B > {{<#>CCZP ''4 #++Z8 //!$yy/C/CAq/IJ    3#7"J J6nn8Lg7_6`a %%'(*   %""$a(BB 7a8R7SS\]k]p]p]r\st(,,S..'SVddL',,S>>-A7TL}},,\r,B  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVdVdfm?wwL',,S>>-A7TL  %1$5$5c>>7$\ !055cNN6JG]L$( !]]**<4<AAr/) rrrr#rrr'rr*r)r%r()FTFN)NNNNF)__name__ __module__ __qualname____firstlineno____doc__intfloatboolrrr"rITensorr8rr[__static_attributes__ __classcell__r,s@r-rr$sZG  +/CCC C  C  CC(CC>eU\\eCece 488<1526"'vB||vB#5<<0vB!u||!45 vB !. vB "%,,/ vB vB u||Xell3XeELL>Q5RR SvBvBr/rcV^\rSrSrSrS\4U4SjjrS\R4Sjr Sr U=r $)PatchTSTBatchNormzH Compute batch normalization over the sequence length (time) dimension. rc~>[TU]5 [R"URUR S9Ulg)Neps)r!r"r BatchNorm1dd_modelnorm_eps batchnormr+rr,s r-r"PatchTSTBatchNorm.__init__s( FOOLr/inputsclURSS5nURU5nURSS5$)z Parameters: inputs (`torch.Tensor` of shape `(batch_size, sequence_length, d_model)`): input for Batch norm calculation Returns: `torch.Tensor` of shape `(batch_size, sequence_length, d_model)` rr4)r6rs)r+rvoutputs r-r[PatchTSTBatchNorm.forwards7!!!Q''1%%r/)rs r^r_r`rarbrr"rIrfr[rgrhris@r-rkrks+M~M &ell & &r/rkrv mask_ratiounmasked_channel_indiceschannel_consistent_masking mask_valuecUS:dUS:a[SUS35eURupVpxURn [USU- -5n U(a*[R "USXyS9n U R SUS5n O[R "XVXyS9n [R"XVXyS9n SU SS2SS2SU 24'[R"U SS9n [R"U SS9n[R"U SUS 9n U RS5R SSSU5n UbSU SS2USS2SS24'URU R5U5nXS 4$) arandom_masking: Mask the input considering the control variables. Args: inputs (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, num_features)`): The input tensor to mask. mask_ratio (`float`): Masking ratio applied to mask the input data during random pretraining. It is the number between 0 and 1. unmasked_channel_indices (list, *optional*): Indices of channels that will not be masked. channel_consistent_masking (bool, *optional*, defaults to `False`): When true, masking will be same across all channels of a timeseries. Otherwise, masking positions will vary across channels. mask_value (int, *optional*, defaults to 0): Define the value of masked patches for pretraining. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as input Tensor and mask tensor of shape [bs x c x n] rrz Mask ratio z has to be between 0 and 1.deviceNrBrC)rDindex.r) r$rHrrcrIrandrepeatonesargsortgather unsqueeze masked_fillre)rvr{r|r}r~ batch_size num_channelssequence_length num_featuresrlen_keepnoisemask ids_shuffle ids_restore inputs_masks r-random_maskingrsA4A~q;zl2MNOO>Dll;Jo ]]F?a*n56H! :q/I Q a0 :_T ::j ODDAyy--2.K-- 4K <<"K 8D >>"  $ $Q1l ;D+23Q(!Q ./$$TYY[*=K V $$r/num_forecast_mask_patchescP[U[5(aU/nUVs/sHnSPM nnURupgp[R"XgXR S9n /n Sn [ U5n [X5HGupUS::dX:a[SUS35e[Xo-U - 5nU RXU/5 U U- n MI [U SS9n X:aU SSXl- -U SS'OX:aU S SX- -U S S'SnU HunnnUU-nSU UU2S S 2U*S 24'UnM [R"U RS5nU Un U RS 5RSSSU 5n UbSU S S 2US S 2S S 24'URU R5U5nUU S 4$s snf) aiForecast masking that masks the last K patches where K is from the num_forecast_mask_patches. If num_forecast_mask_patches is a list, samples in the batch will be randomly masked by numbers defined in the list. Parameters: inputs (`torch.Tensor`): Input of shape `(bs, num_channels, num_patch, patch_length)` num_forecast_mask_patches (`list`): Number of patches to be masked at the end of each batch sample. e.g. 4 or [3, 5]. unmasked_channel_indices (`list`, *optional*): Indices of channels that are not masked. mask_value (`int`, *optional*, defaults to 0): Values in the masked patches will be filled by `mask_value`. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as inputs Tensor and Mask tensor of shape `(bs, num_channels , num_patch)` or `(bs, tsg1, tsg2, num_channels, num_patch)` rrrznum_forecast_mask_patches z6 should be greater than 0 and less than total patches.c US$)Nr4)xs r-"forecast_masking..>s!A$r/)keyr4rBNr) isinstancercrHrIzerosrsumzipr$appendsortedrandpermrrrre)rvrr|r~rQforecast_mask_ratiosrrrrrt_list total_length total_ratio patch_lengthratiotemp_lenbatch1 patch_lenbatch2permrs r-forecast_maskingrs0+S11%>$?!'@A'@!A'@A>Dll;Jo ;;z WD FL*+K"#<S 1  ?,\N:pq z)K78 |H56  TF /F ay|z'@Aq !  "r 1 )BCr 1 F"( 1h("./VF]A z{ *+#) >>$**Q- (D :D >>"  $ $Q1l ;D+23Q(!Q ./$$TYY[*=K V $$OBs F#cV^\rSrSrSrS\4U4SjjrS\R4Sjr Sr U=r $)PatchTSTPatchifyiVz A class to patchify the time series sequence into different patches Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` rc>[TU]5 URUlURUlUR UlURUR::a&[ SURSURS35e[URUR5UR- UR -S-UlURUR URS- --nURU- Ul g)NzSequence length (z+) has to be greater than the patch length ()r) r!r"context_lengthrr patch_strider$max num_patchessequence_start)r+rnew_sequence_lengthr,s r-r"PatchTSTPatchify.__init__^s %44"//"//   4#4#4 4#D$8$8#99deievevdwwxy    4 4d6G6GH4K\K\\aeararruvv"//$2C2CtGWGWZ[G[2\\"225HHr/ past_valuesc4URSnX R:wa[SUSURS35eUSS2URS2SS24nUR SUR UR S9nURSS5R5nU$)z Parameters: past_values (`torch.Tensor` of shape `(batch_size, sequence_length, num_channels)`, *required*): Input for patchification Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` zInput sequence length (z%) doesn't match model configuration (rN) dimensionrGstep) rHrr$runfoldrrr6r7)r+rrrxs r-r[PatchTSTPatchify.forwardos&++B/ 22 2)/)::_`d`t`t_uuwx Q 3 3 5q89$2C2C$J[J[\!!"b)446 r/)rrrrrrzris@r-rrVs+I~I"5<<r/rcV^\rSrSrSrS\4U4SjjrS\R4Sjr Sr U=r $)PatchTSTMaskingial Class to perform random or forecast masking. Parameters: config (`PatchTSTConfig`): model config Returns: x_mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points rc>>[TU]5 URUlURUlURUlUR UlUR UlURUlUR b[UR 5UlggN) r!r"random_mask_ratior} mask_typerr|r~rrts r-r"PatchTSTMasking.__init__s !'!9!9*0*K*K')))/)I)I&(.(G(G% ++  ( ( 4,243P3P,QD ) 5r/ patch_inputcdURS:Xa8[UURURURUR S9up#OVURS:Xa-[ UURURUR S9up#O[SURS35eUR5nX#4$)a Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Patch input Return: masked_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points random)rvr{r|r}r~forecast)rvrr|r~zInvalid mask type .) rrrr|r}r~rrr$re)r+r masked_inputrs r-r[PatchTSTMasking.forwards >>X %!/"11)-)F)F+/+J+J?? " L$^^z )!1"*.*H*H)-)F)F?? " L$1$..1ACD Dyy{!!r/)r}rr~rrr|rzris@r-rrs+  R~ R!"5<<!"!"r/rcd^\rSrSrSrS\4U4SjjrS S\RS\ \ 4Sjjr Sr U=r $) PatchTSTEncoderLayeriz PatchTST encoder layer rc &>[TU]5 URUl[URUR UR S9UlURS:a [R"UR5O[R"5Ul URS:Xa[U5UlOWURS:Xa/[R "URUR"S9UlO[%URS35eUR(aURS:a [R"UR5O[R"5UlURS:Xa[U5UlOWURS:Xa/[R "URUR"S9UlO[%URS35e[R*"[R,"URUR.UR0S9[2UR4"5UR6S:a [R"UR65O[R"5[R,"UR.URUR0S95UlURS:a [R"UR5O[R"5UlURS:Xa[U5UlOWURS:Xa/[R "URUR"S9UlO[%URS35eUR>Ulg)N)rrrrrs layernormrnz$ is not a supported norm layer type.r ) r!r"channel_attentionrrqnum_attention_headsattention_dropout self_attn path_dropoutrDropoutIdentity dropout_path1 norm_typerknorm_sublayer1 LayerNormrrr$ dropout_path2norm_sublayer2 Sequentialr&ffn_dimrr activation_function ff_dropoutff dropout_path3norm_sublayer3pre_normrts r-r"PatchTSTEncoderLayer.__init__s !'!9!9*nn00,, AG@S@SVW@WRZZ(;(;<]_]h]h]j   { *"3F";D     ,"$,,v~~6??"SD  0 011UVW W  ! !DJDWDWZ[D[F,?,?!@acalalanD ;.&7&?#!![0&(ll6>>v&W# F$4$4#55Y!Z[[-- IIfnnfnn6;; G F.. / 1-3->->-BBJJv(( ) IIfnnfnn6;; G  AG@S@SVW@WRZZ(;(;<]_]h]h]j   { *"3F";D     ,"$,,v~~6??"SD  0 011UVW W r/ hidden_stater?cURup4pVURX4-XV5nUR(a6URUR U5US9upxn XR U5-nO4URXS9upxn UR XR U5-5nUR X4XV5nUR(aURSS5R5nURX5-XF5nUR(a6URURU5US9upzn XRU5-nO4URXS9upzn URXRU5-5nUR X5XF5nURSS5R5nURX4-XV5nUR(a2XRURURU555-nO1URXRURU55-5nUR X4XV5nU4n U(aXR(aUW 4OU4- n U $)ao Parameters: hidden_state (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, d_model)`, *required*): Past values of the time series output_attentions (`bool`, *optional*): Whether or not to return the output attention of all layers Return: `torch.Tensor` of shape `(batch_size, num_channels, sequence_length, d_model)` )r:r?r4r)rHr5rrrrrKrr6r7rrrrr) r+rr?rnum_input_channelsrrqrZrWrQchannel_attn_weightsoutputss r-r[PatchTSTEncoderLayer.forwardsiDPCUCU@ $(()H/c ==+/>>"11,?Sd,:, (Kq(*<*<[*IIL,0>>*,:, (Kq ..|>P>PQ\>]/]^L$++JOe   ! !'11!Q7BBDL',,Z-IK]gL}}7;~~"&"5"5l"CWh8F84 1 ,.@.@.MM 8<~~".8F84 1 $22P>PQUQXQXYeQf>g/ghL$++JOe /  ?U?U &:;\h[j jGr/) rrrrrrrrrrr)r^r_r`rarbrr"rIrfrrer[rgrhris@r-rrs9/(~/(bQELLQXd^QQr/rc4\rSrSr\rSrSrSrSr S Sjr Sr g) PatchTSTPreTrainedModeliLmodelrFc*[U[5(aURR(a([R R URSS9 URRS:Xa*[R R URSSS9 gg[U[R5(aJURRR5 URRRS5 g[U[ 5(a^UR"RRR5 UR"RRRS5 g[U[R$[R&45(akURRR SURR(S9 URb%URRR5 ggg) z Initialize weights g{Gz?)stdrr]g?)meanr?N)rPatchTSTPositionalEncodingr use_cls_tokenrinitnormal_ cls_tokenpositional_encoding_type position_encrrdatazero_weightfill_rkrsr&Conv1dinit_std)r+modules r- _init_weights%PatchTSTPreTrainedModel._init_weightsSsc f8 9 9{{(( 0 0d;{{33x? 3 3#3G@  - - KK   " " $ MM   $ $S )  1 2 2    ! ! & & , , .    # # ( ( . .s 3 BII 6 7 7 MM   & &CT[[5I5I & J{{&   &&('8r/c<[U[5(aX!lggr)rPatchTSTEncodergradient_checkpointing)r+rvalues r-_set_gradient_checkpointing3PatchTSTPreTrainedModel._set_gradient_checkpointingis f 0 0,1 ) 1r/rN)F) r^r_r`rar config_classbase_model_prefixmain_input_namesupports_gradient_checkpointingrr rgrr/r-rrLs"!L#O&+#),2r/rcR^\rSrSrS\4U4SjjrS\R4SjrSr U=r $)PatchTSTEmbeddinginrc>[TU]5 URUlURUlUR(a1[R "UR UR5Ulg[R"5Ul[UR5HGnURR[R "UR UR55 MI gr) r!r"rshare_embeddingrr&rrqinput_embedding ModuleListranger)r+rrQr,s r-r"PatchTSTEmbedding.__init__os "(";";%55   #%99V-@-@&..#QD #%==?D 6445$$++BIIf6I6I6>>,Z[6r/rc jURSnX R:wa[SURSUS35eUR(aUR U5nU$[ U5Vs/sH$o@RU"USS2USS2SS245PM& nn[ R"USS9nU$s snf)z Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Patch input for embedding return: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, d_model)` rz&The defined number of input channels (zQ) in the config has to be the same as the number of channels in the batch input (rNrC)rHrr$rrrrIstack)r+rr embeddingsis r-r[PatchTSTEmbedding.forward{s)..q1 !8!8 889P9P8QRTTfSgghj    --k:JUZZlTmnTmq..q1+aAqj2IJTmJnZQ7Jos,+B0)rrr r^r_r`rarr"rIrfr[rgrhris@r-rrns& \~ \5<<r/rc^\rSrSrSrS\S\4U4Sjjr\S\S\S\ R4Sj5r S\ R4S jrS rU=r$) riz Class for positional encoding rrc>[TU]5 URUlURUlUR(aA[R "[ R"SSSUR55Ul US- nURX5Ul URS:a&[R"UR5Ul g[R"5Ul g)Nrr)r!r"rrr ParameterrIrrqr_init_perpositional_dropoutrrr+rrr,s r-r"#PatchTSTPositionalEncoding.__init__s #11"(";";   \\%++aAv~~*NODN 1 K MM&>6<5N5NQR5RBJJv00 1 XZXcXcXe r/r@c$URS:Xa5[R"[R"XR 5SS9nU$URS:XGa#[R "XR 5n[R"SU5RS5n[R"[R"SUR S5[R"S5UR - *-5n[R"X4-5USS2SSS24'[R"X4-5USS2SSS24'X"R5- nX"R5S -- n[R"US S9nU$[!URS 35e) NrT requires_gradsincosrrr4g@ FzN is not a valid positional encoder. Available types are 'random' and 'sincos'.)rrr#rIrandnrqrarangerexpmathlogsincosrrr$)rrrpositiondiv_terms r-r$#PatchTSTPositionalEncoding._init_pesX  * *h 6<< K(P`deL , , 8 ;;{NNCL||A{3==a@Hyya!CQXHY\b\j\jHjFk!klH$)IIh.A$BLADqD !$)IIh.A$BLADqD !'*;*;*==L'+;+;+=+BCL<< EJL 2234BC r/rcxUR(aURXRSS2SS24-5nURURSS2SS24-nUR UR SUR SS5n[R"X14SS9nU$URXR-5nU$)NrrrBr4rC) rr%rrexpandrHrrIrJ)r+rr cls_tokensrs r-r["PatchTSTPositionalEncoding.forwards   11+@Q@QRSRTVWRW@X2XYK):):2A2q5)AAI"))+*;*;A*>@W@WY[]_`J 99j%>AFL 22;ARAR3RSLr/)rrrr%r)r^r_r`rarbrrcr" staticmethodrr#r$rIrfr[rgrhris@r-rrs]  ~  C  cbll& 5<<  r/rc z^\rSrSrSrS\S\4U4SjjrS S\RS\ \ S\ \ S \ 4S jjr S rU=r$) r iz PatchTST Encoder rrc,>[TU]U5 SUl[U5Ul[ X5Ul[R"[UR5Vs/sHn[U5PM sn5Ul UR5 gs snf)NF)r!r"r rembedderrpositional_encoderrrrnum_hidden_layersrlayers post_init)r+rrrr,s r-r"PatchTSTEncoder.__init__sx  &+#*&1 "