eThw|SSKrSSKrSSKrSSKrSSKrSSKrSSKJrJr SSK J r J r SSK J r SSKJr SSKrSSKJrJr SSKJr \R,"5S :waSSKr\"5(aSS KJr SS KJr SS KJr \"5(aSSKrSS KJ r J!r!J"r"J#r#J$r$ \%"\RL\RN-5r(\%"\RL\RR-\RN-S-5r*SS0SS0SS0SS0SS0SS0S.r+Sr,SSR[\+R]55S3r//SQr0"SS5r1\ "SS55r2S \4S!jr3"S"S#\5r4g)$N)ArgumentParser Namespace) dataclassfield)Thread)Optional)is_rich_availableis_torch_available)BaseTransformersCLICommandWindows)Console)Live)Markdown)AutoModelForCausalLM AutoTokenizerBitsAndBytesConfigGenerationConfigTextIteratorStreamerz .!\"#$%&'()*+,\-/:<=>?@[]^_`{|}~textz5There is a Llama in my lawn, how can I get rid of it?zyWrite a Python function that integrates any Python function f(x) numerically over an arbitrary interval [x_start, x_end].z4How many helicopters can a human eat in one sitting?z4Count to 10 but skip every number ending with an 'e'zWhy aren't birds real?z2Why is it important to eat socks after meditating?)llamacode helicopternumbersbirdssocksaa **TRANSFORMERS CHAT INTERFACE** Chat interface to try out a model. Besides chatting with the model, here are some basic commands: - **!help**: shows all available commands - **!status**: shows the current status of the model and generation settings - **!clear**: clears the current conversation and starts a new one - **!exit**: closes the interface am **TRANSFORMERS CHAT INTERFACE HELP** Full command list: - **!help**: shows this help message - **!clear**: clears the current conversation and starts a new one - **!status**: shows the current status of the model and generation settings - **!example {NAME}**: loads example named `{NAME}` from the config and uses it as the user input. Available example names: `z`, `a%` - **!set {ARG_1}={VALUE_1} {ARG_2}={VALUE_2}** ...: changes the system prompt or generation settings (multiple settings are separated by a space). Accepts the same flags and format as the `generate_flags` CLI argument. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options - **!save {SAVE_NAME} (optional)**: saves the current chat and settings to file by default to `./chat_history/{MODEL_NAME}/chat_{DATETIME}.yaml` or `{SAVE_NAME}` if provided - **!exit**: closes the interface ) )max_new_tokensr) do_sampleTr) num_beamsr r ) temperature?r!)top_k2r#)top_pr"r%)repetition_penaltyr"r&) eos_tokensN eos_token_id) eos_token_idsNr(c\rSrSrSS\\S\\4SjjrS\S\4SjrS\4S jr S r S \4S jr S \S \4Sjr SS\ 4SjjrS\S\S\4SjrSrg) RichInterfacewN model_name user_namecf[5UlUcSUlOXlUcSUlgX lg)N assistantuser)r_consoler-r.)selfr-r.s R/var/www/auris/envauris/lib/python3.13/site-packages/transformers/commands/chat.py__init__RichInterface.__init__xs/   )DO(O  #DN&N output_streamreturncrSnURRSURS35 [URSS9n[ U5HupEU(aUS:XaMX%- n/nUR 5HNnUR U5 URS5(aUR S5 M=UR S 5 MP [SRU5R5S S 9nURU5 M S S S 5 URR5 U$!,(df  N*=f) zRStream output from a role, and return the generated text after it's done steaming.z [bold blue]:)consolerefresh_per_secondrz``` z z github-dark) code_themeN) r2printr-r enumerate splitlinesappend startswithrjoinstripupdate) r3r8rliveioutputslineslinemarkdowns r4 stream_outputRichInterface.stream_outputs  l4??*;2>? $--A >$' 6 !q& OO-DLL&u-- T* V,.$BGGEN$8$8$:}U H%57?:  =? >s CD(( D6cURRSURS35nURR5 U$)z!Gets user input from the console. [bold red]: )r2inputr.rA)r3rSs r4rSRichInterface.inputs9 ##k$..1A$FG  r7c8URR5 g)zClears the console.N)r2clear)r3s r4rVRichInterface.clears r7rcURRSURSU35 URR5 g)z%Prints a user message to the console.rRz>:[/ bold red] N)r2rAr.)r3rs r4print_user_message RichInterface.print_user_messages7 k$..)99I$PQ r7colorczURRSUSU35 URR5 g)z,Prints text in a given color to the console.z[bold ]Nr2rA)r3rr[s r4 print_colorRichInterface.print_colors1 fUG1TF34 r7minimalcURR[U(a[O[55 URR5 g)z'Prints the help message to the console.N)r2rArHELP_STRING_MINIMAL HELP_STRING)r3ras r4 print_helpRichInterface.print_helps. HG%8UV r7generation_config model_kwargscURRSUS35 U(aURRSU35 URRSU35 URR5 g)zFPrints the status of the model and generation settings to the console.z[bold blue]Model: r?z[bold blue]Model kwargs: z [bold blue]Nr^)r3r-rgrhs r4 print_statusRichInterface.print_statussf 0 B?@  MM  ";L> J K k*;)<=> r7)r2r-r.)NN)F)__name__ __module__ __qualname____firstlineno__rstrr5rrOrSrVrYr_boolrerdictrj__static_attributes__r7r4r+r+ws '8C= 'HSM '%+?%C%Ns s C $ s?O_cr7r+c\rSrSr%Sr\"SSS0S9r\\\ S'\"SSS0S9r \\\ S '\"SSS 0S9r \\\ S '\"S SS 0S9r \\ S'\"SSS0S9r \\\ S'\"SSS0S9r\\\ S'\"SSS0S9r\\ S'\"SSS0S9r\\ S'\"SSS0S9r\\ S'\"SSS0S9r\\ S'\"SSS 0S9r\\ S!'\"SSS"0S9r\\ S#'\"SSS$0S9r\\ S%'\"SSS&0S9r\\\ S''\"SSS(0S9r\\\ S)'\"S*SS+0S9r\\ S,'\"S-SS.0S9r\\ S/'\"S0S1/S2QS3.S9r\\\ S4'\"S5SS60S9r\\ S7'\"SSS80S9r\\\ S9'\"S5SS:0S9r \\ S;'\"S5SS<0S9r!\\ S='\"S>S?S@S>/S3.S9r"\\ SA'\"S5SSB0S9r#\\ SC'SDr$g)E ChatArgumentsz Arguments for the chat CLI. See the metadata arg for each argument's description -- the medatata will be printed with `transformers chat --help` Nhelpz_Name of the pre-trained model. The positional argument will take precedence if both are passed.)defaultmetadatamodel_name_or_pathzKUsername to display in chat interface. Defaults to the current user's name.r1zSystem prompt. system_promptz./chat_history/zFolder to save chat history. save_folderz"Path to a yaml file with examples. examples_pathzPath to a local generation config file or to a HuggingFace repo containing a `generation_config.json` file. Other generation settings passed as CLI arguments will be applied on top of this generation config.rgrz%Maximum number of tokens to generate.rTz,Whether to sample outputs during generation.rr z Number of beams for beam search.r r"z%Temperature parameter for generation.r!r$zValue of k for top-k sampling.r#z Value of p for nucleus sampling.r%zRepetition penalty.r&zNEOS tokens to stop the generation. If multiple they should be comma separated.r'zQEOS token IDs to stop the generation. If multiple they should be comma separated.r)mainzLSpecific model version to use (can be a branch name, tag name or commit id).model_revisioncpuzDevice to use for inference.deviceautozOverride the default `torch.dtype` and load the model under this dtype. If `'auto'` is passed, the dtype will be automatically derived from the model's weights.)rbfloat16float16float32)rxchoices torch_dtypeFz2Whether to trust remote code when loading a model.trust_remote_codezWhich attention implementation to use; you can run --attn_implementation=flash_attention_2, in which case you must install this manually by running `pip install flash-attn --no-build-isolation`.attn_implementationzIWhether to use 8 bit precision for the base model - works only with LoRA. load_in_8bitzIWhether to use 4 bit precision for the base model - works only with LoRA. load_in_4bitnf4zQuantization type.fp4bnb_4bit_quant_typez#Whether to use nested quantization.use_bnb_nested_quantrt)%rlrmrnro__doc__rr{rrp__annotations__r1r|r}r~rgrintrrqr r!floatr#r%r&r'r)rrrrrrrrrrsrtr7r4rvrvs). u )   ghD(3-$)IY@Z#[M8C=[%6&JhAijKj#(Im@n#oM8C=o(- 1  (x}  v?f6ghNChDFe5fgKgrV5U,VWE3W8Z/[\E5\ %cVEZ<[ \\ %jk!J $)mn$M8C= hiNC9W0XYFCY!&PA "K#$)] ^t*/ r *#efL$efL$ %UFZhmotgu=vww!&uHm?n!o$or7rvargsc[U5$)z3 Factory function used to chat with a local model. ) ChatCommandrs r4chat_command_factoryr%s t r7c\rSrSr\S\4Sj5rSrS\S\4Sjr \S\ 4Sj5r \S"S\S \ \ S\ 4S jj5r \S"S \ \ S\\4S jj5rS\\ S\4SjrS\S\S\\\44Sjr\S\S\ \ S\ \ S\\\\44Sj5r\S\S\ S4Sj5rS\S\\\44SjrS\ S\S\S\\ \\ \ 44S\S\S\\S\\\\\44SjrS rS!rg )#ri,parserc[4nURSUS9nURS5nURS[SSS9 URS[SS S S 9 UR [ S 9 g) z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments chat)dataclass_typeszPositional argumentsmodel_name_or_path_positionalNzName of the pre-trained model.)typeryrxgenerate_flagsaFlags to pass to `generate`, using a space as a separator between flags. Accepts booleans, numbers, and lists of integers, more advanced parameterization should be set through --generation-config. Example: `transformers chat max_new_tokens=100 do_sample=False eos_token_id=[1,2]`. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options*)rryrxnargs)func)rv add_parseradd_argument_group add_argumentrp set_defaultsr)rr chat_parsergroups r4register_subcommandChatCommand.register_subcommand-s)*'''P ../EF  +#tJj    J      &: ;r7c2URU5nXlgN)_handle_deprecated_argsr)r3rs r4r5ChatCommand.__init__Js++D1 r7rr9c SnUR=(d URUlURc [S5eURbSn[R"S[ 5 [ H?up4n[X5nXd:wdMSn[R"SUSUSUS3[ 5 MA U(a[S 5 [5 U$) zr Handles deprecated arguments and their deprecation cycle. To be removed after we fully migrated to the new args. FzOne of the following must be provided: - The positional argument containing the model repo, e.g. `transformers chat ` - the optional --model_name_or_path argument, containing the model repo (deprecated)TzThe --model_name_or_path argument is deprecated will be removed in v4.54.0. Use the positional argument instead, e.g. `transformers chat `.zThe --a. argument is deprecated will be removed in v4.54.0. There are two alternative solutions to specify this generation option: 1. Pass `--generation-config ` to specify a generation config. 2. Pass `generate` flags through positional arguments, e.g. `transformers chat =`z (Press enter to continue)) rr{ ValueErrorwarningswarn FutureWarning_DEPRECATION_MAPgetattrrArS)r3r has_warningsdeprecated_arg default_valuenew_argvalues r4r#ChatCommand._handle_deprecated_argsNs  .2-O-O-jSWSjSj*  - - 5i   $ $ 0L MMK  7G 2N7D1E%#  ^,-siqq * "  7G  / 0 G r7c[R"5S:Xa[R"5$[R "[R "55R$)z)Returns the username of the current user.r )platformsystemosgetloginpwdgetpwuidgetuidpw_namertr7r4 get_usernameChatCommand.get_usernameys: ??  );;= << ,44 4r7Nfilenamec0n[U5US'XS'URnUcG[R"S5nURSUS3n[ R RXB5n[ R"[ R RU5SS9 [US 5n[R"X6S S 9 SSS5 [ R RU5$!,(df  N-=f) z!Saves the chat history to a file.settings chat_historyNz%Y-%m-%d_%H-%M-%Sz/chat_.jsonT)exist_okwr<)indent)varsr}timestrftimerrpathrFmakedirsdirnameopenjsondumpabspath)rrr output_dictfoldertime_strfs r4 save_chatChatCommand.save_chats "&t* J&*N#!!  }}%89H<<=VH:USHww||F5H BGGOOH-= (C A IIkQ /!wwx((! s )C&& C4r|c Uc/nU$SUS./nU$)zClears the chat history.rrolecontentrt)r|rs r4clear_chat_historyChatCommand.clear_chat_historys*  D &-@AD r7rc "[U5S:Xa0$UVs0sH0nSURS5S-S-URS5S_M2 nnUR5VVs0sH+upEXER5S;aUR5OU_M- nnnUR5VVs0sHupEXES:XaSOU_M nnnS[S [ 4S jnUR5VVs0sHupEXF"U5(dSUS3OU_M nnnS R UR5VVs/sH upEUS U3PM snn5nS U-S-nURSS5nURSS5nURSS5nURSS5nURSS5nURSS5n[R"U5nU$s snfs snnfs snnfs snnfs snnf![Ra [S5ef=f)zUParses the generate flags from the user input into a dictionary of `generate` kwargs.r"rr )truefalseNonenullsr9cDURSSS5R5$)N.r;r )replaceisdigit)rs r4 is_number3ChatCommand.parse_generate_flags..is_numbers99S"a(002 2r7z, z: {}z"null"z"true"rz"false"rz"[[z]"r]:zFailed to convert `generate_flags` into a valid JSON object. `generate_flags` = {generate_flags} Converted JSON string = {generate_flags_string}) lensplititemslowerrprqrFrrloadsJSONDecodeErrorr) r3rflaggenerate_flags_as_dictkvrgenerate_flags_stringprocessed_generate_flagss r4parse_generate_flags ChatCommand.parse_generate_flagssF ~ ! #I bp!paoY]# 3(:":S"@$**S/RSBT"Tao!p OeNjNjNl" NldaAGGI)::qwwy ANl " OeNjNjNl!mNlda!F{V"ANl!m 3 3 3VlUqUqUs!tUsTQ!Yq\\q1Xq"HUs!t !% CYC_C_Ca*bCa41aS1#;Ca*b c!$&; ;c A!6 = =h O 5 = =h O 5 = =i Q 5 = =dC H 5 = =dC H!6 = =c3 G '+zz2G'H $('S"q" "n "u +c"## D  s)7G"2G*G'!G")G( 9G.. H tokenizerc URc[5nURX!RUR5upEUR UR URURURURURUUS. nUR"S0UD6 OSUR;ai[RRUR5n[RR!UR5n[R""Xx5nO [R""UR5nUR%UR&5n UR"S0U D6n X:4$)zZ Returns a GenerationConfig object holding the generation parameters for the CLI command. ) rrr r!r#r%r& pad_token_idr(rrt)rgrparse_eos_tokensr'r)rrr r!r#r%r&rHrrrbasenamefrom_pretrainedrr) r3rrrgr r)deprecated_kwargsrrparsed_generate_flagsrhs r4get_generation_parameterization+ChatCommand.get_generation_parameterizations4  ! ! ) 0 2 *.*?*? ??\`\n\n*o 'L"&"5"5!^^!^^#//&*&=&= , - !   $ $ 9'8 9$000''//$*@*@A77++D,B,BC$4$D$DW$W!$4$D$DTE[E[$\!!% 9 9$:M:M N(//H2GH !..r7r'r)cURc URnO URn/nUb/URURUR S555 Ub:URUR S5Vs/sHn[ U5PM sn5 [ U5S:XaURUR5 X44$s snf)z:Retrieves the pad token ID and all possible EOS token IDs.,r)r r(extendconvert_tokens_to_idsrrrrD)rr'r)r all_eos_token_idstoken_ids r4r ChatCommand.parse_eos_tokenss  ! ! )$11L$11L  !  $ $Y%D%DZEUEUVYEZ%[ \  $  $ $MDWDWX[D\%]D\c(mD\%] ^  !Q &  $ $Y%;%; <.. &^s<C model_argsrcUR(a7[SURURURURS9nU$UR (a [SS9nU$SnU$)NT)rbnb_4bit_compute_dtyperbnb_4bit_use_double_quantbnb_4bit_quant_storage)r)rrrrrr)rquantization_configs r4get_quantization_config#ChatCommand.get_quantization_configst  " ""4!'1'='=$.$B$B*4*I*I'1'='= # #" $ $"4!#  #"#' ""r7c[R"URURURS9nUR S;a UR O[ [UR 5nURU5nURURUSUS.n[R"UR4SUR0UD6n[ USS5cURUR5nXb4$)N)revisionr)rNr)r"rr device_maprr hf_device_map) rr rrrrrtorchrrrtor)r3rrrrrhmodels r4load_model_and_tokenizer$ChatCommand.load_model_and_tokenizer&s!11  . .(("44 +/*:*:n*Ld&&RYZ_aeaqaqRr "::4@++#'#;#;& #6  %44  . . BFBXBX \h  5/4 0 8HHT[[)Er7 user_input interfaceexamplesrgrhrcUS:Xa-URUR5nUR5 GO4US:XaUR5 GOUR S5(al[ UR 55S:aOUR 5n[ U5S:XaUSn OSn URXrU 5n URSU S3S S 9 GOUR S 5(aUS SR5n U R 5n U H n S U ;dM URSU S3SS 9 GO= URU 5n UR"S0U D6n UR"S0U D6 GOUR S5(a[ UR 55S:XaUR 5SnX;aBUR5 /nURXNS5 URSXNSS.5 OwSUS[UR55S3nURUSS 9 OFUS:XaUR!UR"UUS9 O$URSUS3SS 9 UR5 XuU4$)z Handles all user commands except for `!exit`. May update the chat history (e.g. reset it) or the generation config (e.g. set a new flag). z!clearz!helpz!saver NzChat saved in !green)rr[z!setr<rz(Invalid flag format, missing `=` after `z;`. Please use the format `arg_1=value_1 arg_2=value_2 ...`.red!examplerr1rzExample z* not found in list of available examples: rz!status)r-rgrh'z/' is not a valid command. Showing help message.rt)rr|rVrerErrrr_rGrrHrYrDlistkeysrjr)r3r*rr+r,rgrhr split_inputrnew_generate_flagsrparsed_new_generate_flagsnew_model_kwargs example_name example_errors r4handle_non_exit_user_commands)ChatCommand.handle_non_exit_user_commandsAs  !**4+=+=>D OO  7 "  "  " "7 + +J4D4D4F0G!0K$**,K;1$&q>~~d(;H  ! !z'C7 ! S  " "6 * *",AB!5!5!7 !3!9!9!; *d?))FtfMAA$ *+-1,E,EFX,Y)#4#;#;#X>W#X ##7&67  " ": . .3z7G7G7I3Ja3O%++-a0L'!,,X-CF-KL V8Nv8VWX|n,VW[\d\i\i\kWlVmmno%%=%F 9 $  " "=="3) #   ! !:,6e'fns ! t  " 44r7c [5(d [S5e[5(d [S5eURnURc[ nO4[ UR5n[R"U5nSSS5 URcUR5nO URnURU5upV[USSS9nURX5up[URUS9n U R!5 UR#UR$5n U R'SS9 U R)5n U R+S5(a8U S:XagUR-U UU WUU U S 9upn U R+S 5(dM_OU R/S U S .5 UR1U S SS9R3UR45n [6R8"U 5nU UUUS.U En[;UR<US9nUR?5 U RAU5nURC5 U R/SUS .5 GM!,(df  GN=f![Da gf=f)NzHYou need to install rich to use the chat interface. (`pip install rich`)zJYou need to install torch to use the chat interface. (`pip install torch`)T)skip_special_tokens skip_prompt)r-r.)rar/z!exit)r*rr+r,rgrhrr2r1rpt)return_tensorsadd_generation_prompt)inputsattention_maskstreamerrg)targetkwargsr0)#r ImportErrorr rr~DEFAULT_EXAMPLESryaml safe_loadr1rr(rrr+rrVrr|rerSrEr<rDapply_chat_templater&rr% ones_likergeneratestartrOrFKeyboardInterrupt)r3rr,rr1r'rgeneration_streamerrgrhr+rr*rDrEgeneration_kwargsthread model_outputs r4runChatCommand.runs^ ""hi i!##jk kyy    %'Hd(()Q>>!,* 99 $$&D99D88>29RVdhi*.*N*Nt*_'!T-O-O[_` &&t'9'9: T*+ &__. ((--!W,@D@b@b'1!%&/%-.?)5!%AcA=&00<< =KKJ GH"66tDhl6mppLL"'!8$&4 3): % # %! u~~>OP (667JK   [\JKS%*)z%  s+*I 4,I!.IB7I I I*)I*rr)rlrmrnro staticmethodrrr5rvrrprrrr4rrrrrtuplerrrr rrr(r+r<rVrsrtr7r4rr,s\##& ] uEY[hEh?i 6N5N5N5! N5 sDcN*+ N5 , N5N54jN5 tDz+T1 2N5dHr7r)5rrrstringrrargparserr dataclassesrr threadingrtypingrrKtransformers.utilsr r r;r rr rich.consoler rich.liver rich.markdownrr% transformersrrrrrset ascii_letters whitespaceALLOWED_KEY_CHARSdigitsALLOWED_VALUE_CHARSrJrcrFr5rdrr+rvrrrtr7r4rjs   .( D( ?? !$&,,v/@/@@A 6==(6+<+<?@A & RRj UpUp Uppyo,or7