fTh+PSrSSKrSSKJr SSKJr SSKJrJrJ r J r SSK r SSK r SSK J r SSKJr SS KJr SS KJrJrJrJrJrJr S S KJr \R6"\5rSqS r"SS\ R@RB5r"S%Sjr#S%Sjr$"SS\ RJ5r&"SS\ RJ5r'"SS\ RJ5r(\"SS\55r)\"SS\55r*\"SS\55r+\"SS\)55r,\"S S!9"S"S#\)\55r-/S$Qr.g)&zPyTorch RWKV model.N) dataclass)Path)ListOptionalTupleUnion)nn)GenerationMixin)PreTrainedModel) ModelOutputauto_docstringis_bitsandbytes_availableis_ninja_availableis_torch_cuda_availablelogging) RwkvConfigcSSKJn [[5R 5R R R S- S- nSVs/sHo2U- PM nn[ b[ RU:Xag[RSUS35 SS S S S S SU3/nU"SU3U[R"5[R:HUS9qU[ lgs snf)Nr)loadkernelsrwkv)z wkv_op.cppz wkv_cuda.cuzwkv_cuda_bf16.cuz2Loading CUDA kernel for RWKV at context length of .z -res-usagez--maxrregcount 60z--use_fast_mathz-O3z -Xptxas -O3z--extra-device-vectorizationz-DTmax=wkv_)namesourcesverboseextra_cuda_cflags) torch.utils.cpp_extensionrr__file__resolveparentrwkv_cuda_kernelmax_seq_lengthloggerinfor get_verbosityDEBUG)context_length load_kernel kernel_folderfcuda_kernel_filesflagss ^/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/rwkv/modeling_rwkv.pyload_wkv_cuda_kernelr0.s=N**,33::AAIMPVVM4ef4eq*4ef#(8(G(G>(Y KKD^DTTUVW  & .!" E#N# $!&&(GMM9  '5#/gs Cc<\rSrSr\SSj5r\SSj5rSrg)RwkvLinearAttentionNNc UR5upxn U[R:a [SUS[RS35eXy-[ U S5-S:wa[SUSU S[ U S5S 35eUR UlURRS :wdNURRS :wd4URRS :wdURRS :wa [S 5e[R"UR5R55*nUR [R:Xa0UR5nUR5nUR5nUR5nUR5nUR5n[R"U[RS 9n U(dUbUcT[R "UU S [R"UR[RS9nUSS2SS2S4==S-ss'OB[R$"UV s/sHoR'S5PM sn SS9R5nUR [R(:Xa[R*n O[R,n U "XX4X5 OHUR [R(:Xa[R.O[R0n U "XX4U 5 UR3XX4U 5 Ub4[R4"US SS9V s/sHoR7S5PM nn U R9UR 5U4$s sn fs sn f)NzCannot process a batch with z+ tokens at the same time, use a maximum of z with this model. rzThe product of batch size (z) and hidden size (z") needs to be a round multiple of rcudazUCalling the CUDA kernel for wkv attention requires all tensors to be on CUDA devices. memory_formatr )dtypedevicer8籡*G)dim)sizer#r$ ValueErrorminr9 input_dtyper:typetorchexpfloat contiguousfloat16 empty_likecontiguous_formatzerosfloat32cat unsqueezebfloat16forward_with_state_bf16forward_with_state forward_bf16forwardsave_for_backwardchunksqueezeto) ctx time_decay time_firstkeyvaluestate return_state batch_sizeseq_len hidden_sizeoutputs forward_funcs r/rRRwkvLinearAttention.forwardOs+.88:( [ %44 4.wi7b#2233DF   #c+r&: :a ?-j\9L[MZ";34A7  ))    " "f ,  %%/zz&(||  F*tu uii 0 0 2 = = ?@@ 99 %#))+J))+CKKME**, nn  "!!#U5L5LM 5,} --::"'"9"9 aAg$& 5"A5a;;q>5"AqITTVyyENN*/GG /BB V K>$A%..u}} '' %BYBYZ   E4K4KL""58O8OP %-- '~~'H:E:W(66]m]v]v          !      OOK ( OOK ( HH[ ! JJ{ #     NFN)__name__ __module__ __qualname____firstlineno__ staticmethodrRrh__static_attributes__rrrqr/r2r2Ns)<1<1|% % rqr2cUR5upgn[R"U5nUc[R"USS2S4[RS9n [R"USS2S4[RS9n [R"USS2S4[RS9S- n OUupn [R"U5*n[ U5GH n USS2U 4R 5n USS2U 4n[R"XU-5n[R"X- 5n[R"X-U- 5nUU -UU--nUU -U-nUU- RUR5USS2U 4'[R"X-U 5n[R"X-U- 5n[R"U U- 5nUU -UU--n UU -U-n Un GM U(dUbXU /nX4$)Nr)r9r<) r>rC zeros_likerKrDrangerEmaximumrVr9)rXrYrZr[r\r]_ seq_lengthra num_state den_state max_state current_index current_key current_valuemax_for_outpute1e2 numerator denominator max_for_states r/rwkv_linear_attention_cpursxxzA1   c "F }$$SAYemmD $$SAYemmD $$SAYemmDtK */' i ))J''Jz* !]*+113 a./ y 2JK YYy1 2 YY{/.@ ANR-%77 9nr) $- $;#?#? #Mq-  i& YY{]2 3NR-%77 NR' ! %+(u(y1 =rqc [SXX#455nURS5S:Hn[bU(dU(a [XX#XES9$[R XX#XE5$)Nc3R# UHoRRS:gv M g7f)r6N)r:rB).0ts r/ (rwkv_linear_attention..sX3Wa((--6)3Ws%'rr\r])anyr>r#rr2apply)rXrYrZr[r\r]no_cuda one_tokens r/rwkv_linear_attentionrsZXJC3WXXG q I7i(SXtt"((Uaarqc@^\rSrSrSU4SjjrSSjrSSjrSrU=r$) RwkvSelfAttentionc>[TU]5 Xl[SL=(a [RUR :Hn[ 5(a,[5(aU(d[UR 5 X l URnURb UROUnXPl[R "["R$"U55Ul[R "["R$"U55Ul[R "["R$"SSU55Ul[R "["R$"SSU55Ul[R "["R$"SSU55Ul[R0"S5Ul[R4"XESS9Ul[R4"XESS9Ul[R4"XESS9Ul[R4"XTSS9Ulg![a [RS5 GNf=f)Nz9Could not load the custom CUDA kernel for RWKV attention.rrrrFbias)super__init__configr#r$r)rrr0 Exceptionr%r&layer_idr`attention_hidden_sizer ParameterrCemptyrXrY time_mix_keytime_mix_valuetime_mix_receptance ZeroPad2d time_shiftLinearrZr[ receptancera)selfrr kernel_loadedr`r __class__s r/rRwkvSelfAttention.__init__s  (4q9I9X9X\b\q\q9q   $;$=$=m Y$V%:%:;! (( ,2,H,H,TF ( (Ze &;",,u{{3H'IJ,,u{{3H'IJLLQ;)GH ll5;;q![+IJ#%<< Aq+0N#O ,,}599[eLYY{N ))KUSii 5O ) Y WX Ys(H%%IIcpURS5S:XaUbUSSS2SS2UR4nO4URU5nUb USSS2SS2UR4USS2S4'XR-USUR- --nXR-USUR- --nXR -USUR - --nUR U5nURU5n[R"URU55nUb USS2S4USSS2SS2UR4'XdXR4$Nrrr) r>rrrrrrZr[rCsigmoidr)rhiddenr\shiftedrZr[rs r/extract_key_value#RwkvSelfAttention.extract_key_values2 ;;q>Q 5#4Ahq!T]]23Goof-G %aAt}})< =1 (((7a$:K:K6K+LL,,,w!d>Q>Q:Q/RR666AH`H`D`9aa hhsm 5!]]4??:#>?  ,21b5ME!HQ4==( ),,rqc ^TRXS9upEpbUb[U4SjUSS55OSn[TRTRUUUUS9upUbTUSUSSS2SS2TR 4'USUSSS2SS2TR 4'USUSSS2SS2TR 4'TR XH-5U4$) Nr\c3N># UHoSS2SS2TR4v M g7frtr)rrbrs r/r,RwkvSelfAttention.forward..#sFIqaDMM12Is"%r;rrrr )rtuplerrXrYrra) rrr\ use_cacherrZr[ layer_staters ` r/rRRwkvSelfAttention.forward!s(,(>(>v(>(S% JOJ[eFE!"IFFae 1 OO OO  "    ",7NE!HQ4==( ),7NE!HQ4==( ),7NE!HQ4==( ){{:,-u44rq) rrrZrrarrXrYrrrrr[rrtrs) rurvrwrxrrrRrz __classcell__rs@r/rrsP<-&55rqrc6^\rSrSrSU4SjjrSSjrSrU=r$)RwkvFeedForwardi5c8>[TU]5 XlX lURnUR b UR OSUR-n[ R"S5Ul[ R"[R"SSU55Ul [ R"[R"SSU55Ul [ R"X4SS9Ul[ R"X3SS9Ul[ R"XCSS9Ulg)NrrrFr)rrrrr`intermediate_sizer rrrrCrrrrrZrr[)rrrr`rrs r/rRwkvFeedForward.__init__6s   (( (.(@(@(LF $ $RSV\VhVhRh ,,}5LLQ;)GH#%<< Aq+0N#O 99[%H))K5IYY0EJ rqcURS5S:XaUbUSSS2SS2UR4nO4URU5nUb USSS2SS2UR4USS2S4'XR-USUR- --nXR-USUR- --n[ R "[ R"URU555nURU5n[ R"URU55nUb USS2S4USSS2SS2UR4'XV-U4$r) r>rrrrrCsquarerelurZr[rr)rrr\rrZrr[s r/rRRwkvFeedForward.forwardGs# ;;q>Q 5#4Ahq!T]]23Goof-G %aAt}})< =1 (((7a$:K:K6K+LL666AH`H`D`9aa ll5::dhhsm45 3]]4??:#>?  ,21b5ME!HQ4==( )!5((rq)rrZrrrrrr[rrtrurvrwrxrrRrzrrs@r/rr5sK"))rqrc2^\rSrSrU4SjrSSjrSrU=r$) RwkvBlocki[c>[TU]5 XlX lUS:Xa.[R "UR URS9Ul[R "UR URS9Ul [R "UR URS9Ul [X5Ul [X5Ulg)Nr)eps)rrrrr LayerNormr`layer_norm_epsilonpre_lnln1ln2r attentionr feed_forward)rrrrs r/rRwkvBlock.__init__\s   q=,,v'9'9v?X?XYDK<< 2 28Q8QR<< 2 28Q8QR*6<+F=rqcURS:XaURU5nURURU5X#S9upRX-nUR UR U5US9upbX-nX4nU(aXu4- nU$US- nU$)Nr)r\rrrt)rrrrrr)rrr\routput_attentionsrroutputss r/rRRwkvBlock.forwardjs ==A [[(F>>$((6*:%>] #"//0@/N &/  | #G w Grq)rrrrrrr)NFFrrs@r/rr[s >rqrc8\rSrSr\rSrS/rSS/rSr Sr Sr Sr g ) RwkvPreTrainedModeli}rrrXrYTc [U[5(GavURnURRnURR nUR nX#S- - nSX#- - n[R"[U5Vs/sHoU- PM snURRURRS9n U SSSS24n [U5V s/sHn SSXS- - SSU-----PM n n [R"XRRURRS9n [R"[U5Vs/sH oS-S -S- PM snURRURRS9S -n [R"5 XRl[R""UR[$R&"S 5-U -5URl[R("X5URl[R("X5S U--UR*l[R("U S U-5UR,lSSS5 g[U[.5(Ga URnURRnURR nSX#- - n[R"[U5Vs/sHoU- PM snURRURRS9n U SSSS24n [R"5 [R("X5URl[R("X5UR,lSSS5 ggs snfs sn fs snf!,(df  g=fs snf!,(df  g=f) zInitialize the weights.rg?r9r:Ngffffff?g?r g?g333333?) isinstancerrrnum_hidden_layersr`rrCtensorr}rr9r:rXrYno_graddata ones_likemathlogpowrrr) rmodulerrr`r ratio_0_to_1ratio_1_to_almost0i time_weighth decay_speedzigzags r/ _init_weights!RwkvPreTrainedModel._init_weightss6 f/ 0 0H & ? ?  --33K$*$@$@ !#1'<=L!$(D!E ,,*/ *<=* L!> !s2N0 N5N:1CN??OA O? O  O#rrN) rurvrwrxr config_classbase_model_prefix_no_split_modules_keep_in_fp32_modulessupports_gradient_checkpointing _is_statefulrrzrrrqr/rr}s1L$ )<8&*#L7]rqrc\rSrSr%SrSr\\R\ S'Sr \\ \R\ S'Sr \\ \RS4\ S'Sr\\ \RS4\ S'S rg) RwkvOutputia^ Class for the RWKV model outputs. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. state (list of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlast_hidden_stater\. hidden_states attentionsrr)rurvrwrx__doc__rrrC FloatTensor__annotations__r\rrrrrzrrrqr/rrsw,6:x 1 129/3E8D**+ ,3=AM8E%"3"3S"89:A:>Ju00#567>rqrc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \R\ S'Sr \\\RS4\ S'Sr\\\RS4\ S 'S rg) RwkvCausalLMOutputia0 Base class for causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). state (list of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlosslogitsr\.rrrr)rurvrwrxrr rrCrr r r\rrrrrzrrrqr/r r s0)-D(5$$ %,*.FHU&& './3E8D**+ ,3=AM8E%"3"3S"89:A:>Ju00#567>rqr c^\rSrSrU4SjrSrSr\SS\\ RS\\ RS\\ RS\\ \ RS \\ S \\ S \\ S \\ S \\\44Sjj5rSrSrSrU=r$) RwkvModelic >[TU]U5 [R"URUR 5Ul[R"[UR5Vs/sH n[XS9PM sn5Ul [R"UR 5Ul SUlSUlUR!5 gs snf)NrF)rrr Embedding vocab_sizer` embeddings ModuleListr}rrblocksrln_outlayers_are_rescaledgradient_checkpointing post_init)rridxrs r/rRwkvModel.__init__s  ,,v'8'8&:L:LMmmPUV\VnVnPo$pPoYv%DPo$pq ll6#5#56 #( &+# %qs(C cUR$rtrrs r/get_input_embeddingsRwkvModel.get_input_embeddingss rqcXlgrtrrnew_embeddingss r/set_input_embeddingsRwkvModel.set_input_embeddingss(rq input_idsattention_mask inputs_embedsr\rroutput_hidden_states return_dictreturnc 6UbUOURRnUbUOURRnUbUO(UR(dURROSnUbUOURR nUb[ RS5 URUR:XaUR5 UbUb [S5eUcUc [S5eUcURU5nU(aUcURS5URRURR4n [S5V s/sHCn [ R""XS::a UR$O[ R&UR(S .6PME nn US ==S -ss'UR*(a/UR(aU(a[ RS 5 SnUn U(aS OSn U(aS OSn [-UR.5HupUR*(a2UR(a!UR1UR2XXV5upnO U"XXVS9upnUR(a?URR4S:a%US-URR4-S:XaU S- n U(aX4-n U(dMU U4-n M UR7U 5n U(aX4-n U(d[9SXX455$[;U UU U S9$s sn f)a' input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0][0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) state (tuple of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). use_cache (`bool`, *optional*): If set to `True`, the last state is returned and can be used to quickly generate the next logits. NFz<`attention_mask` was passed, but it is unused in this model.zDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedsrrrrgꌠ9Y>)FzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...rr)r\rrr;c3.# UH ocMUv M g7frtrr)rxs r/r$RwkvModel.forward..{st$bq$bs )rr\rr)rrr)trainingruse_return_dictr% warning_oncer_rescale_layersr?rr>r`rr}rCrJr9rKr:r enumerater_gradient_checkpointing_func__call__ rescale_everyrrr)rr&r'r(r\rrr)r*shaperrall_self_attentionsall_hidden_statesrblockrs r/rRRwkvModel.forwards>2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IZ^ZgZgT[[=R=Rmr %0%cd d  =#8TU U   OOI6M "''*DKK,C,CT[[EbEbcE q "A a-"5"5U]][h[o[o"  !H H  & &4==##p" % $5b4"6BD#DKK0JC**t}}373T3TNNM)40 j49!)40 j ((KK--11W 9 99Q> - 1 #$58H$H!  &9ZM&I#+1. M2  14D D t];L$btt t++*   es A Lc URUR(+:XagURRS:Ga[R "5 [ UR5GHupUR(aURRRRS[XRR-5-5 URRRRS[XRR-5-5 M[URRRS5(aURRRR R#S[XRR-5-5 URRRR R#S[XRR-5-5 GM[URRRS5(aOUR%URRU5 UR%URRU5 GM!URRRR#S[XRR-5-5 URRRR#S[XRR-5-5 GM SSS5 UR(+Ulg!,(df  N%=f)Nrr;SCB quant_state)rr1rr8rCrr5rrraweightmul_intrr[hasattrr?div_ _bnb_4bit_dequantize_and_rescale)rblock_idr<s r/r4RwkvModel._rescale_layerss?  # #DMM(9 :  ;; $ $q ('0'=OH}}..55::1HP[P[PiPiDi@j;jk**0077<>qCT_T_TmTmHmDn?no!..44;;@@c(VaVaVoVoJoFpApq(>!"(,}}#4 #!s KM Mc[5(d [S5eSSKnURR UR R UR R5nURS[X RR-5-5 URRURS5SS9RUR5n[!USU5 g) z Perform the dequantization and rescaling of the weights of a given layer. After that operation the layer will be quantized again. z/Please install bitsandbytes to use this method.rNr;cpuF) requires_gradrA)r ImportError bitsandbytes functionaldequantize_4bitrArr@rErCrr8r Params4bitrVr:setattr)r target_layerrGbnbdequant_weights quant_weights r/rF*RwkvModel._bnb_4bit_dequantize_and_rescales )**OP P"..889L9L9Q9QS_SfSfSrSrsQ#h++2K2K&K"LLMvv((););E)BRW(X[[\k\r\rs  h 5rq)rrrrr)NNNNNNNN)rurvrwrxrrr$rrrC LongTensorrrboolrrrrRr4rFrzrrs@r/rrs )15595937$(,0/3&*l E,,-l !!1!12l   1 12 l U../0 l D> l $D>l 'tnl d^l  uj !l l \5066rqrz The RWKV Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc@^\rSrSrS/rU4SjrSrSrSSjr\ SS\ \ RS\ \ RS \ \ RS \ \\ RS \ \ RS \ \S \ \S\ \S\ \S\\\44Sjj5rSrU=r$)RwkvForCausalLMiz head.weightc>[TU]U5 [U5Ul[R "UR URSS9UlUR5 g)NFr) rrrrr rr`rheadr)rrrs r/rRwkvForCausalLM.__init__sH  f% IIf00&2C2C%P  rqcUR$rtr]rs r/get_output_embeddings%RwkvForCausalLM.get_output_embeddingss yyrqcXlgrtr`r"s r/set_output_embeddings%RwkvForCausalLM.set_output_embeddingss" rqc jUbUSS2S4RS5nUbUcSU0nOSU0nX&S'XFS'U$)Nrr(r&r\r)rM)rr&r\r(rkwargs model_inputss r/prepare_inputs_for_generation-RwkvForCausalLM.prepare_inputs_for_generationsY  !!R%(2226I  $+];L'3L %W$-[!rqr&r'r(r\labelsrrr)r*r+c xU bU OURRn URUUUUUUU S9n U Sn URU 5n SnUb*UR"U U4SURR 0U D6nU (dU 4U SS-nUbU4U-$U$[ UU U RU RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0][0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) state (tuple of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` use_cache (`bool`, *optional*): If set to `True`, the last state is returned and can be used to quickly generate the next logits. N)r(r\rrr)r*rrr)r r r\rr) rr2rr] loss_functionrr r\rr)rr&r'r(r\rkrrr)r*rg rwkv_outputsrr r ras r/rRRwkvForCausalLM.forwardsJ&1%F $D>F 'tnF d^F  u(( )F F rqr[)r[rrrs)/rr dataclassesrpathlibrtypingrrrrrCtorch.utils.checkpointr generationr modeling_utilsr utilsr rrrrrconfiguration_rwkvr get_loggerrur%r#r0autogradFunctionr2rrModulerrrrrr rr[__all__rrrqr/r~s[  !// )-+   H %5@g %..11g T)XbC5 C5L#)bii#)L D?]/?]?]D ?? ?: ?? ?@o6#o6o6d i )?i  i X Brq