eThs RSSKrSSKJr SSKrSSKJr SSKJr \"5(aSSKJ r J r J r SSK J r Jr "SS \RR 5r"S S \RR 5r"S S \RR 5rSS\ S\\R(S\\R(S\\S\\4 Sjjr"SS\RR 5r"SS\RR 5r"SS\RR 5rg)N)Optional)GenerationConfig)is_torch_available) HybridCachePreTrainedModel StaticCache)is_torch_greater_or_equal"is_torch_greater_or_equal_than_2_3c^\rSrSrSrSS\S\S\4U4SjjjrS\RS\RS \R4S jr SS\ \RS\ \RS \ \ S \ \ S \RR4 S jjr\SS\RRS\S\S\ S\S\S\S\S \4Sjj5rSrU=r$)%TorchExportableModuleForDecoderOnlyLMz A recipe module designed to make a `PreTrainedModel` exportable with `torch.export`, specifically for decoder-only LM with cache. This module ensures that the exported model is compatible with further lowering and execution in `ExecuTorch`. modelmax_batch_size max_cache_lenc>[TU]5 [URS5(aURRSLa [ S5e[URS5(d'[ R"S5 [U5Ul g URRS:Xa[XU5Ul g [ SURRS35e) aT Initializes the exportable module with `HybridCache`. Args: model (`PreTrainedModel`): The pretrained model to wrap. max_batch_size (int): Maximum batch size for the cache. max_cache_len (int): Maximum sequence length for the cache. Raises: ValueError: If the model is configured with a unsupported cache implementation. use_cacheFz5The model must have caching enabled to be performant.cache_implementationzXUsing `StaticCache` for export as `cache_implementation` is not specified in the config.hybridz"Unsupported cache implementation: z". Please use `hybrid` or `static`.N) super__init__hasattrconfigr ValueErrorlogginginfo$TorchExportableModuleWithStaticCacherr$TorchExportableModuleWithHybridCache)selfrrr __class__s \/var/www/auris/envauris/lib/python3.13/site-packages/transformers/integrations/executorch.pyr.TorchExportableModuleForDecoderOnlyLM.__init__#s" u||[11U\\5K5Ku5TTU Uu||%;<< LLs t=eDDJ||00H<A%Yfg  89Z9Z8[\77 input_idscache_positionreturnc8URRX5$)J Forward pass of the module, which is compatible with the ExecuTorch llm runner. Args: input_ids (`torch.Tensor`): Tensor representing current input token id to the module. cache_position (`torch.Tensor`): Tensor representing current input position in the cache. Returns: torch.Tensor: Logits output from the model. )rforward)rr$r%s r!r)-TorchExportableModuleForDecoderOnlyLM.forwardGszz!!)<IELLSTRUQV^c^h^hDi3A3MSXS_S_ab`ckpkukuSv||"" JJ#<)#/6 #  6: #  r#exported_programpromptmax_new_tokens do_sample temperaturetop_ktop_pdevicec UR5n U"USS9RRU5n U R5n Sn [ U R S5HBn U SS2XS-24n[ R"U /[ RUS9nU "X5nU S- n MD [ U5GHnU SS2SS24n[ R"U /[ RUS9nU "X5nU(aUS:aUU- nOUnUS:a.U[ R"UU5SS:n[S 5UU'US :a[ R"US S 9unn[ R"[ R"USS 9SS 9nUU:nUSSS24R5USSS24'SUS'URSUU5n[S 5UU'[ R"USS 9n[ R"USS9nOUR!SS S9nUR#5S:aUR%S5n[ R&"U U/SS 9n U S- n UR)5UR*:XdGM O UR-U SS S9$)a Generate a sequence of tokens using an exported program. Args: exported_program (`torch.export.ExportedProgram`): The exported model being used for generate. tokenizer: The tokenizer to use. prompt (str): The input prompt. max_new_tokens (int): Maximum number of new tokens to generate. do_sample (bool): Whether to use sampling or greedy decoding. temperature (float): The temperature for sampling. top_k (int): The number of highest probability tokens to keep for top-k sampling. top_p (float): The cumulative probability for nucleus sampling. device (str): The device to use. Returns: str: The generated text. pt)return_tensorsrr.Nr0rB).rGNz-inf?T) descendingdim.).r) num_samples)rKkeepdimr)skip_special_tokens)moduler$toclonerangeshaper4r5r6topkfloatsortcumsumsoftmaxscatter multinomialargmaxrKsqueezecatitem eos_token_iddecode)r; tokenizerr<r=r>r?r@rArBexported_moduler$ generated_ids curr_positionicurr_input_idscurr_cache_position_outputslogitsindices_to_remove sorted_logitssorted_indicescumulative_probssorted_indices_to_removeprobs next_token_ids r!generate.TorchExportableModuleForDecoderOnlyLM.generatews<+113fT:DDGGO ")  yq)*A&q!!e)|4N"',, ejjY_"`  DA Q M+~&A*1bc62N"',, ejjY_"` &nJG?${2F$F19(.FE1J11Mm1\(\%05f F,-3;49JJvRV4W1M>',||EMM-UW4X^`'a$0@%/G,8PQTVYWYVYQY8Z8`8`8b,S!"W578,V4)A(H(H^]u(v%05f F,- f"5 % 1 1%Q G !(2t D   "Q& - 5 5b 9 "II}m&D"MM Q M!!#y'='==k'p a 0dKKr#)rr.NNNN)FrH2rHcpu)__name__ __module__ __qualname____firstlineno____doc__rintrr4Tensorr)rdictboolr7ExportedProgram staticmethodstrrUrr__static_attributes__ __classcell__r s@r!r r sk ! """ ""H=<<= =  =&-115)-!%  ELL) !. !     % %  > ! iL,,66iLiL iL  iL  iLiLiLiL iLiLr#r c ^\rSrSrSrS\4U4SjjrS\RS\R4Sjr \ S\RRS \RS \ S \R4S j5rS rU=r$)ra A recipe module designed to make a `PreTrainedModel` exportable with `torch.export`, specifically for decoder-only LM to `StaticCache`. This module ensures that the exported model is compatible with further lowering and execution in `ExecuTorch`. Note: This class is specifically designed to support export process using `torch.export` in a way that ensures the model can be further lowered and run efficiently in `ExecuTorch`. rc>[TU]5 URc [S5eURR(d [S5eURR S:wa [S5eXl[UR RUR RRRUR RRRUR RRRUR RS9Ul[[!URR"55HYnUR%SU3URR"USS 9 UR%S U3URR&USS 9 M[ [)S UR RR*55UlUR,(ar[.R0"[.R2"URRURR[.R4S 95nUR%S USS 9 gg)a Initializes the wrapper module with the pretrained model. Args: model (`PreTrainedModel`): The pretrained model to wrap. The model must have caching enabled and use a 'static' caching implementation. Raises: AssertionError: If the pretrained model does not have caching enabled or if it does not use a 'static' caching implementation in `model.generation_config`. NzkThe model must have a generation config to be exported with static caching. Please set `generation_config`.zvThe model must have caching enabled to be exported with static caching. Please set `generation_config.use_cache=True`.staticzThe model must use a 'static' caching implementation to be exported with static caching. Please set `generation_config.cache_implementation='static'`.rrrrBr0 key_cache_F persistent value_cache_c3,# UH nSU;v M g7f)CausalLMN).0archs r! @TorchExportableModuleWithStaticCache.__init__..s\<[DZ4/<[sr/mask)rrgeneration_configAssertionErrorrrrr r cache_config batch_sizerrBr0 static_cacherRlen key_cacheregister_buffer value_cacheany architectures is_causalr4trilonesr)rrre causal_maskr s r!r-TorchExportableModuleWithStaticCache.__init__s   " " * 2  &&00 A   " " 7 78 C P   '::$$::77DDOO**66CCQQ:://<<CC**""  s4,,6678A  :aS!143D3D3N3Nq3Q^c d  <s!3T5F5F5R5RST5Ubg h9\DJJ>** %%33%%33**K   G r#r$r%c URup4UR(aURUSU24OSnURS5nURnUR UUUUUSS9nUR $)a` Forward pass of the module, which is compatible with the ExecuTorch runtime. Args: input_ids (`torch.Tensor`): Tensor representing current input token id to the module. cache_position (`torch.Tensor`): Tensor representing current input position in the cache. Returns: torch.Tensor: Logits output from the model. This forward adapter serves two primary purposes: 1. **Making the Model `torch.export`-Compatible**: The adapter hides unsupported objects, such as the `Cache`, from the graph inputs and outputs, enabling the model to be exportable using `torch.export` without encountering issues. 2. **Ensuring Compatibility with `ExecuTorch` runtime**: The adapter matches the model's forward signature with that in `executorch/extension/llm/runner`, ensuring that the exported model can be executed in `ExecuTorch` out-of-the-box. NrT)r$attention_mask position_idsr%past_key_valuesr)rSrr unsqueezerrrj) rr$r%rhseqlen attn_maskrroutss r!r),TorchExportableModuleWithStaticCache.forward'sz*OO :>..DIIngvg56d %//2 ++zz$%)+  {{r#r;prompt_token_idsr=r&c 0URSnX2-nUR5H7upVURS5(dMURSn[XG5n O /n[ [XC55Hsn UR 5R USS2XS-24[R"U /[RS9S9n URUSU R55 Mu [R"W SS2SSS24SS 9R5n URU 5 [U5U:aUR 5R [R"U //[RS9[R"[U5/[RS9S9n [R"U SS2SSS24SS 9R5n URU 5 [U5U:aM[R"U/[RS9$) ay Generate a sequence of tokens using an exported program. This util function is designed to test exported models by simulating the generation process. It processes the input prompt tokens sequentially (no parallel prefill). This generate function is not intended to replace the original `generate` method, and the support for leveraging the original `generate` is potentially planned! Args: exported_program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`. prompt_token_ids (`torch.Tensor`): Tensor representing the input prompt token IDs. max_new_tokens (`int`): Maximum number of new tokens to generate. Note that the total generation length is limited by both `max_new_tokens` and the model's cache size. Returns: torch.Tensor: A tensor containing the generated sequence of token IDs, including the original prompt tokens. rGrrNr.r/)r$r%rrJ)rS named_buffers startswithminrRrOr)r4r5r6appendr^r[r) r;rr=prompt_token_lenmax_generation_length buffer_namebufferrresponse_tokens input_posresult current_tokens r!rr-TorchExportableModuleWithStaticCache.generateKs*,11"5 0 A#3#A#A#C K%%k22 & Q (+,A(Q% $D s#8KLI%,,.66*1ia-.G+GH$||YKuzzJ7F  " "#3A#6y#A#F#F#H I M VAr1H%52>CCE }-/"%::%,,.66,,'8 K$||S-A,B%**U7F"LL2q)9rBGGIM  " "= 1 /"%::||_-UZZ@@r#)rrr)rzr{r|r}r~rrr4rr)rr7rrrrrrrs@r!rrs6Ho6Hp""u||"H/A,,66/AJO,,/Ahk/A /A/Ar#rc^\rSrSrSrS S\S\S\4U4SjjjrS\RS\RS \R4S jr S r U=r $) ri~z A recipe module designed to make a `PreTrainedModel` exportable with `torch.export`, specifically for decoder-only LM to `HybridCache`. This module ensures that the exported model is compatible with further lowering and execution in `ExecuTorch`. rrrc>[TU]5 XlURRR(d [ S5e[ URRS5(a$URRRS:wa [ S5e[URRUUURRURRS9Ul [[URR55HYnURSU3URRUSS9 URS U3URR USS9 M[ g ) aZ Initializes the exportable module with `HybridCache`. Args: model (`PreTrainedModel`): The pretrained model to wrap. max_batch_size (int): Maximum batch size for the cache. max_cache_len (int): Maximum sequence length for the cache. Raises: AssertionError: If the model doesn't have the expected configuration for HybridCache. zModel must have caching enabledrrz,Model must use 'hybrid' cache implementationrrFrrN)rrrrrrrrrrBr0cacherRrrrr)rrrrrer s r!r-TorchExportableModuleWithHybridCache.__init__s'"  zz  ** !BC C ))+ABBzz  55A !OP P!::$$)'::$$**""   s4:://01A  :aS!14::3G3G3JW\ ]  <s!3TZZ5K5KA5N[` a2r#r$r%r&c URup4URS5RUS5n[R"X44[R UR S9nURUUUURSUS9nUR$)r(rrGrFT)r$rrrrr%) rSrexpandr4rr6rBrrrj)rr$r%rseq_lenrrris r!r),TorchExportableModuleWithHybridCache.forwards(oo &//299*bI Z$9T]TdTde**)% JJ)  ~~r#)rrrt) rzr{r|r}r~rrrr4rr)rrrs@r!rr~sn ! *b*b*b *b*bX"<<" "  ""r#rrr8r9r+r,c \[(d [S5eSSKnUR"5 UbUOUR"S//UR S9nUbUOUR"S/UR S9n[ S5(a-URR[U5X40UUbUOSS9nOeUb[R"S 5 Ub[R"S 5 URRR[U5U4S U0S SS 9nUsSSS5 $!,(df  g=f)a Convert a `PreTrainedModel` into an exportable module and export it using `torch.export`, ensuring the exported model is compatible with `ExecuTorch`. Args: model (`PreTrainedModel`): The pretrained model to be exported. example_input_ids (`Optional[torch.Tensor]`): Example input token id used by `torch.export`. example_cache_position (`Optional[torch.Tensor]`): Example current cache position used by `torch.export`. dynamic_shapes(`Optional[dict]`): Dynamic shapes used by `torch.export`. strict(`Optional[bool]`): Flag to instruct `torch.export` to use `torchdynamo`. Returns: Exported program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`. ztorch >= 2.3 is required.rNr.r/z2.6.0Tr1zWDynamic shapes spec will be ignored by convert_and_export_with_cache for torch < 2.6.0.zSThe strict flag will be ignored by convert_and_export_with_cache for torch < 2.6.0.r%F)r2r3 pre_dispatchr,) r ImportErrortorch.export._traceno_gradr5r6r r7rrwarning_trace_export)rr8r9r+r,r4r;s r!convert_and_export_with_cachers5* . -566 "3!> ELLSTRUQV^c^h^hDi '=&H "ell\][^fkfpfpNq  %W - -$||224U;'@-!'!3v 3  )m! uv %||22::4U;')(*@A" ;   G s C'D D+c2^\rSrSrSrU4SjrSrSrU=r$) Seq2SeqLMEncoderExportableModuleiz A wrapper module designed to make a Seq2Seq LM encoder exportable with `torch.export`. This module ensures that the exported encoder model is compatible with ExecuTorch. c.>[TU]5 Xlg)N)rrencoder)r encoder_modelr s r!r)Seq2SeqLMEncoderExportableModule.__init__s $ r#c4URUS9R$)N)r$)rlast_hidden_state)rr$s r!r)(Seq2SeqLMEncoderExportableModule.forward s||i|0BBBr#)r rzr{r|r}r~rr)rrrs@r!rrs %CCr#rc2^\rSrSrSrU4SjrSrSrU=r$)/Seq2SeqLMDecoderExportableModuleWithStaticCachei$z A wrapper module designed to make a Seq2Seq LM decoder exportable with `torch.export`, specifically for use with static caching. This module ensures the exported decoder is compatible with ExecuTorch. c>[TU]5 UR5UlURUlUR Ul[ UR UUS[RS9Ul [[URR55HYnURSU3URRUSS9 URSU3URRUSS9 M[ g)NryrrFrr)rr get_decoderdecoderlm_headrr r4float32rrRrrrr)rrmax_static_cache_lengthrrer s r!r8Seq2SeqLMDecoderExportableModuleWithStaticCache.__init__+s ((* }} ll (;;%1--  s4,,6678A  :aS!143D3D3N3Nq3Q^c d  <s!3T5F5F5R5RST5Ubg h9r#chURUUURSUS9nURUS5nU$)NT)r$encoder_hidden_statesrrr%r)rrr)rdecoder_input_idsrr%ri lm_logitss r!r)7Seq2SeqLMDecoderExportableModuleWithStaticCache.forwardAsD,,'"7 --)  LL, r#)rrrrrrs@r!rr$s i,  r#rcJ^\rSrSrSU4SjjrSrSrS SjrSrSr U=r $) Seq2SeqLMExportableModuleiQc>[TU]5 XlUR5UlUR UlX0l[SUUUUS.S9UlSUl SUl g)NT)rr)r max_lengthrr) rr full_model get_encoderrrmax_hidden_seq_lengthrrexported_encoderexported_decoder)rrrrrmax_cache_lengthr s r!r"Seq2SeqLMExportableModule.__init__Rsi ((* ll %:"!1'!5(!1 " !% $r#cb[UR5RS5R5n[R R SURS9n[R"5 [R R X!4SSU00SS9nSSS5 U$!,(df  W$=f)Nryencoder_seq_lengthmaxr$r.Tr+r,) rrrPevalr4r7Dimrr)rencoder_input_idswrapped_encoder seq_len_dimrs r!_export_encoder)Seq2SeqLMExportableModule._export_encodergs:4<<HKKERWWYll&&';A[A[&\ ]]_$||22!5{UVXcTdFenr 3     _  s 0%B B.c [URURRRURRR S9R S5R5n[RRSURS9n[R"5 [RRUXU4SSU0SS.SS9nSSS5 U$!,(df  W$=f) N)rrrryencoder_hidden_seq_lengthrr.)rrr%Tr) rrrrrrrPrr4r7rrr)rrrr%wrapped_decoderencoder_seq_len_dimrs r!_export_decoder)Seq2SeqLMExportableModule._export_decoderus ;oo(,(>(>(K(K(Y(Y11>>II  RY TV $ll../JPTPjPj.k]]_$||22">J)-./1D-E&*   3   _ s ,)C C.cUbUO"[R"S[RS9nUbUO$[R"S//[RS9nUbUO#[R"S/[RS9nUbUOW[R"UR R RSURR4[RS9nURU5Ul URXhU5UlU$)N)r. r/rr )r4rr6r5zerosrrrrd_modelrrrrr) rrrrr%example_encoder_input_idsexample_decoder_input_idsr9example_encoder_hidden_statess r!r7 Seq2SeqLMExportableModule.exports!2!> EJJw^c^h^hDi ""3!> ELLSTRUQV^c^h^hDi "4B3MSXS_S_ab`ckpkukuSv%0 "''44??T[[EXEXYafanan &!% 4 45N O $ 4 4 %F\!   r#c [R"5 URR5"U5n[R"S//[R S9nS/n[ US- 5HnURR5"XC[R"U/[R S95n[R"USS2SSS24SS9R5nURU5 [R"U//[R S9nXRR:XdM O UsSSS5 $!,(df  g=f)Nrr/r.rGrJ) r4rrrOr5r6rRrr[r^rrr_) rrr=encoder_outputrrcrerj next_tokens r!rr"Seq2SeqLMExportableModule.generates ]]_!2299;A-...557%u||QCuzz7Z #\\&B*:CHHJ $$Z0%*LL:,uzz$R!!9!99/"!5__sDE1E E)rrrrrrr)r.rurirv) rzr{r|r}rrrr7rrrrrs@r!rrQs$os%*   :.!!r#rrv)rtypingrr4+transformers.generation.configuration_utilsrutils.import_utilsr transformersrrr transformers.pytorch_utilsr r nnModuler rrrrrrrrrrr#r!rs H3FFhELEHHOOELPWA588??WAtU588??Ut1559%)! = =  -= %U\\2= TN = TN = @ Cuxx C*ehhoo*Zs!s!r#