fThJSSKJr SSKJrJrJrJr SSKrSSKJ r SSK J r J r SSK Jr SSKJr SSKJrJr SS KJr SS KJr SS KJrJrJrJr S S KJr S SKJ r J!r!J"r"J#r#J$r$J%r%J&r&J'r' SSK(J)r)J*r* SSK+J,r, \RZ"\.5r/\"SS\55r0\"SS9\"SS\555r1"SS\&5r2"SS\'5r3"SS\$5r4"SS\!5r5"S S!\"5r6\"S"S#\%55r7"S$S%\ Rp5r9\"S&S9"S'S(\#\55r:"S)S*\ Rp5r;\"S+S,\%55r<\"S-S9"S.S/\1\,55r=/S0Qr>g)1) dataclass)ListOptionalTupleUnionN)Cache DynamicCache)GenerationMixin)FlashAttentionKwargs)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)Unpack) ModelOutputauto_docstringcan_return_tuplelogging) AutoModel)KwargsForCausalLMLlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModel LlamaRMSNormLlamaRotaryEmbedding) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinc\rSrSr%SrSr\\R\ S'Sr \R\ S'Sr \\ \ \R\ S'Sr \\ \RS4\ S'Sr\\ \RS4\ S 'Sr\\R\ S 'Sr\R\ S 'Sr\\ \ \R\ S 'Sr\\ \RS4\ S 'Sr\\ \RS4\ S'Sr\\R\ S'Srg)CsmOutputWithPast/a Base class for the model autoregressive outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____firstlineno____doc__r&rtorch FloatTensor__annotations__r'r(rr)r*r+r,r-r.r/r0__static_attributes__r1[/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/csm/modular_csm.pyr$r$/s(*X)-D(5$$ %, $FE  $AEOXeE%*;*;$<=>E=AM8E%"3"3S"89:A:>Ju00#567>6:!2!23:.2%++2OS!8E%8I8I2J,K#LSKO%0A0A30F*G!HOHLhuU->->-C'DEL15M8E--.5r;r$z[ The bare Csm Model outputting raw hidden-states without any specific head on top. ) custom_introcJ\rSrSr\rSrSrS/rS/r Sr Sr Sr Sr SrSrSrSrg) CsmPreTrainedModeljmodelTCsmDecoderLayerr(cTURRn[U[R5(aWUR R RSUS9 URb%URR R5 gg[U[R5(adUR R RSUS9 URb2UR R URR5 gg[U[5(aIURn[US- 5H*nUR R URSUS9 M, g[U[5(a&UR R R!S5 gg)Ng)meanstdrg?)configinitializer_range isinstancennLinearweightdatanormal_biaszero_ Embedding padding_idxCsmCodebooksHead num_codebooksrange CsmRMSNormfill_)selfmodulerErSis r< _init_weights CsmPreTrainedModel._init_weightss=kk++ fbii ( ( MM   & &CS & 9{{&   &&('  - - MM   & &CS & 9!!- ""6#5#56<<>.  0 1 1"00M=1,- ""1%--3C-@.  + + MM   $ $S ),r;r1N)r2r3r4r5r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_cache_class_supports_quantized_cache_supports_static_cache_supports_attention_backendrZr:r1r;r<r?r?jsQL&*#*+#4"5!N! $!"&*r;r?c\rSrSrSrg)rUr1Nr2r3r4r5r:r1r;r<rUrUr;rUc\rSrSrSrg)CsmRotaryEmbeddingr1Nrir1r;r<rlrlrjr;rlc\rSrSrSrg)CsmMLPr1Nrir1r;r<rororjr;roc\rSrSrSrg) CsmAttentionr1Nrir1r;r<rrrrrjr;rrc\rSrSrSrg)rBr1Nrir1r;r<rBrBrjr;rBcP^\rSrSr\rU4Sjr\\SS\ RS\ \ RS\ \ RS\ \ RS\ \S\ \ RS \ \S \ \S \ \S \ \ RS \\S\\\44Sjj55rSrU=r$)CsmDepthDecoderModelc>[TU]U5 [R"URUR -UR 5Ul[R"UR URSS9Ul gNF)rN) super__init__rIrPrS vocab_sizebackbone_hidden_size embed_tokensrJ hidden_sizeinputs_embeds_projectorrWrF __class__s r<r|CsmDepthDecoderModel.__init__s]  LL&*>*>ARAR*RU[UpUpq')yy1L1LfN`N`gl'm$r; input_idsbackbone_last_hidden_stateattention_mask position_idsr( inputs_embeds use_cacheoutput_attentionsoutput_hidden_statescache_positionflash_attn_kwargsreturnc Ub:[RR5(d[R S5 SnUbUOUR R nU bU OUR Rn UbUOUR RnUSLUSL- (a [S5eUR(a/UR(aU(a[R S5 SnU(a Uc [5nU ciUbUR5OSn UbURSOURSn Ub URO URn[R "XU -US9n Uc[R""U S- SS 9nXR$-nUR'UU-5nU SS:HnUb X&SS2S4'O?[RR5(dU(a[R)S 5 UR+U5nUR-X6XU5nUnU R/S5nUR1UU5nU (aS OSnU(aS OSnUR2SUR R4H8nU (aUU4- nU"U4UUUUUU US .U D6nUSnU(dM/UUS4- nM: UR7U5nU (aUU4- n[9UU(aUOSUUS 9$)a* backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids from `cache_position` and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.zX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Frrdevice)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.r1)rrpast_key_valuerrrposition_embeddings)last_hidden_stater(r)r*)r7compiler is_compilinglogger warning_oncerFrrr ValueErrorgradient_checkpointingtrainingr get_seq_lengthshaperarangeclampr}rwarningr_update_causal_mask unsqueeze rotary_emblayersnum_hidden_layersnormr )rWrrrrr(rrrrrrpast_seen_tokensinputs_seq_lengthr codebook_idxsoffsetinput_ids_are_first_codebook causal_maskr)rall_hidden_statesall_self_attns decoder_layer layer_outputss r<forwardCsmDepthDecoderModel.forwards*  #ENN,G,G,I,I   M  L1B1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKL] & &//2 "oom\J#7BD0d![[)H4;;+H+HIM#!m%55!) *)."3#-$7 $ M*!,M  =#3"55'J* -0  -!1 1 &+/8Od+%   r;)rr) NNNNNNNNNN)r2r3r4r5r!r\r|rrr7 LongTensorrr8Tensorr boolrr rrr rr: __classcell__rs@r<rwrws*(Ln '+BF1537+/59$(,0/359p ##p %-U->->$?p !. p u//0 p "% p   1 12p D>p $D>p 'tnp !!1!12p $$89p  u-- .p p r;rwc2^\rSrSrU4SjrSSjrSrU=r$)rRi$c>[TU]5 X l[R"[ R "URS- X55Ulg)Nr)r{r|rSrI Parameterr7emptyrK)rWrrSr}rs r<r|CsmCodebooksHead.__init__%s: *ll5;;t/A/AA/E{#_` r;c Uc3URSnUR[R"U5nOUS- nURUn[ URS5Vs/sH9n[ R RUSS2USS24XFR5PM; nn[R"USS9nU$s snf)Nrrdim) rrKr7rrTrI functionallinearTstack)rWr)r seq_lengthcodebook_weightr codebook_idxs r<rCsmCodebooksHead.forward*s  !&,,Q/J"kk%,,z*BCO*Q.M"kk-8O!&o&;&;A&> ? ?  MM q,/A!BODaDcDc d ?   Mq9   s%AB>)rSrKNr2r3r4r5r|rr:rrs@r<rRrR$sa r;rRa$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) c!d^\rSrSrSrSrSrU4SjrSrSr SS\ RS\ \ S\ \ RS \ \ RS \ \ R4 U4S jjjr\\SS\ RS \ \ RS\ \ R$S \ \ RS\ \\ \\ R4S \ \ RS\ \ RS\ \S\ \S\ \S \ \ RS\\\ R$4S\\S\\\44Sjj55rSrU=r$)CsmDepthDecoderForCausalLMi;Nc>[TU]U5 U?[URUR UR 5Ul[U5Ul gr) r{r|lm_headrRrrSr}codebooks_headrwrArs r<r|#CsmDepthDecoderForCausalLM.__init__GsE  L.v/A/A6CWCWY_YjYjk)&1 r;c[S5eNzNot needed for CsmAttributeErrorrWs r<get_output_embeddings0CsmDepthDecoderForCausalLM.get_output_embeddingsM122r;c[S5errrWnew_embeddingss r<set_output_embeddings0CsmDepthDecoderForCausalLM.set_output_embeddingsPrr;rr(rrrc >[T U]"XX4U40UD6nUSSS:HnU(dURS5 URS5 U$)Nrrrr)r{prepare_inputs_for_generationpop) rWrr(rrrkwargs model_inputsis_first_generation_steprs r<r8CsmDepthDecoderForCausalLM.prepare_inputs_for_generationSscw< ~ Y_ $00@#A!#D#I '   9 : (r;rrlabelsrrrlogits_to_keeprrc  nU bU OURRn U bU OURRn UR"SUUUUUUUU U U S. U D6nUSn[ U [ 5(a!U S:Xa [ SS5nO[ U *S5nOU nURUSS2USS24U bU UOS5nUR5nSnUbBUSSS24R5nUR"SUSURRUS.U D6n[UUURURURS9$) a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. N) rrrrr(rrrrrrr.)r'rr} shift_labels)r&r'r(r)r*r1)rFrrrArHintslicer contiguous loss_functionr}rr(r)r*)rWrrrrr(rrrrrrrroutputsr) slice_indicesr'r&rs r<r"CsmDepthDecoderForCausalLM.forwardisx62C1N-TXT_T_TqTq$8$D $++JjJj  **  'A)%+'/!5)      nc * *" %a %~ot < *M$$ !]A- .Q_Qk}0Mqu ""$  !#qr'?557L%%dt{{7M7M\hlrD&#33!//))   r;)rrANNNN) NNNNNNNNNNNr)r2r3r4r5_tied_weights_keys_tp_plan_pp_planr|rrr7rrr r8rrrrrrrrrrrrrr:rrs@r<rr;sHH2 33 ,0595959 ##"%!!1!12    1 12  !!1!12 ,'+BF1537KO59-1$(,0/35934J ##J %-U->->$?J !. J u//0 J "%tE4E4E/F(F"GH J   1 12J ))*J D>J $D>J 'tnJ !!1!12J c5<</0J *+J  u,, -J J r;rc.^\rSrSrU4SjrSrSrU=r$)CsmBackboneModelEmbeddingsic>[TU]5 [R"URUR -UR 5UlURS[R"UR5UR -SS9 g)Naudio_tokens_offsetsF) persistent) r{r|rIrPrSr}rembed_audio_tokensregister_bufferr7rrs r<r|#CsmBackboneModelEmbeddings.__init__sn "$,,0D0DvGXGX0X[a[m[m"n  "ELL1E1E$FIZIZ$Zgl  r;c^URXR-5nURSS9nU$)Nrr)rrsum)rWr input_embedss r<r"CsmBackboneModelEmbeddings.forwards4..y;T;T/TU #''A'. r;)rrrs@r<rrs r;rcH^\rSrSrU4Sjr\\U4Sj55rSrU=r $)CsmBackboneModelicD>[TU]U5 [U5Ulgr)r{r|rrrs r<r|CsmBackboneModel.__init__s  6v>r;c $>[TU]"S0UD6$)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) r1)r{r)rW super_kwargsrs r<rCsmBackboneModel.forwardsw...r;)r) r2r3r4r5r|rrrr:rrs@r<rrs$? / /r;rz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. c#^\rSrSrSS/rU4SjrSrSrSrSr S r \ U4S j5r U4S jr S$S \\R S \\R S\\R S\\R S\\R 4 SjjrS$S \R$S\\S\\R$S\\R(S\\R$4 U4Sjjjr\\S%S \R$S \\R S\\R S\\R S\\R$S\\\\\R(4S\\R(S\\R$S\\S\\S\\S\\R$S\\\R 4S\\S\\\44Sjj55r \!S\R S\S\S \RDS\R S!\4 S"j5r#S#r$U=r%$)&CsmForConditionalGenerationiz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightc>[TU]U5 URUl[R"UR URSS9Ul[R"URUR 5Ul [RU5Ul [RUR5Ul[ R""UR$5UlUR)5 grz)r{r|r}rIrJrrrPtext_vocab_sizeembed_text_tokensr _from_configbackbone_modelrdepth_decoder_config depth_decoderr from_config codec_config codec_model post_initrs r<r|$CsmForConditionalGeneration.__init__s   ++yy!3!3V5F5FUS !#f.D.DfFXFX!Y.;;FC7DDVE`E`a$001D1DE r;c.URR$rrrrs r<get_input_embeddings0CsmForConditionalGeneration.get_input_embeddingss""///r;c$XRlgrr)rWvalues r<set_input_embeddings0CsmForConditionalGeneration.set_input_embeddingss+0(r;cUR$rrrs r<r1CsmForConditionalGeneration.get_output_embeddingss ||r;cXlgrrrs r<r1CsmForConditionalGeneration.set_output_embeddingss% r;cURR(aOURURRR UR RR5 ggr)rFtie_codebooks_embeddings_tie_or_clone_weightsrrrrrArs r< _tie_weights(CsmForConditionalGeneration._tie_weightssL ;; / /  & &##00CC""((55  0r;c>URSS5(a[T U]"U0UD6up4O[T U]"U0UD6nSn[U5n[ UR 5R 5VVs0sH"upxURU5(dMXvSU_M$ n nn[ URR 5RSS0U E5 U Hn[UR XW-5 M SU;aUW4$U$s snnf)Noutput_loading_infoFdepth_decoder__from_model_config) getr{from_pretrainedlenvarsgeneration_configitems startswithrupdatedelattr) clsargsrrA loading_infoprefix prefix_lenattrrdepth_decoder_attrsrs r<r-+CsmForConditionalGeneration.from_pretraineds  ::+U 3 3"''"94"J6"J E<G+TSnURRR5nURSS5 UR 5HupV[ URX5-U5 M [ TU]"U0UD6 g)Nr*transformers_version)rr0 to_diff_dictrr1setattrr{save_pretrained)rWr6rr8r;r:rrs r<rA+CsmForConditionalGeneration.save_pretrained#sq!"00BBOOQ 6=.446KD D**FM5 A7 00r;r input_valuesinput_values_cutoffsrrcURU5nUGb+[RRUS5nX3S:R 5nXfS:n[ R "UR5URS9R[U5S5nXvRS5:n/n[X#5HupXS:n [U RSS- 5Hpn Xn XS-n U SX24nURR!URS55nUR"R%SS5nUR'US5 Mr M [SU55n[ R("UVs/sH7n[RRUSSSUURS- 45PM9 sn5nURR+U5nUR,R.nUU:HnUR0R3U5nUUUU'[ R4"SSUR,R64UR[ R8S 9UR,R:-nUR0R3U5R=S5nXR,R>:HnURAURC5S5UU'UbbURS5RASSUR,R65nUUUU'US :HRES S 9nS UUSUSSS24'UnXTS.$s snf)a8 Merges the input_ids and input_values to produce a single inputs_embeds tensor: 1 - Infers the codec model on the input_values to retreive codebook token. 2 - Embeds codebook tokens and places them at the correct positions in the inputs_embeds tensor. 3 - If labels are provided, expands them to match codebook dimensions and position the target codebook tokens in the inputs_embeds tensor. Args: input_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`): The input ids to embed. input_values (`torch.Tensor` of shape `(batch_size, channels, audio_sequence_length)`): The audio input values to embed. input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`): The cutoffs of the audio input values relative to its batch index, padded with -1 when no audio. Nrrrrr.c3># UHoRSv M g7f)rN)r).0els r< QCsmForConditionalGeneration._merge_input_ids_with_input_values..\s"K9J288A;9Js)rdtypeiTas_tuple)rr)#r rIrpaddiffr7rmaxrexpandr.rziprTrrencode audio_codes transposeappendrget_audio_codes_maskrFaudio_token_idrronesrSlongcodebook_eos_token_idsqueezeaudio_eos_token_idrepeatrnonzero)rWrrCrDrr audio_lengthsinput_values_maskaudio_tokens_listbatch_input_valuesbatch_input_values_cutoffsrY start_idxend_idx audio_batch codec_outputs codebook_idsmax_audio_framesrJbatched_audio_token_idsaudio_codes_maskr[audio_token_mask audio_embedsaudio_eos_frame_idsaudio_eos_embedsaudio_eos_token_masklabels_expanded depth_decoder_ignore_frames_idxss r<"_merge_input_ids_with_input_values>CsmForConditionalGeneration._merge_input_ids_with_input_values-sQ*..y9  ##%==#4#45I6#R 01JKPPRM)!*;"-GfgHg-h*9??BQFGA : =I8Q?G"4S):K5K"LK$($4$4$;$;KMr(rrrc 2>[T U]"S UUUUUS.UD6nUb|URS:XalURS5cZUR UURS5URS5URS5S9nUR USUSSS.5 U$) N)rr(rrrrrrCrDr)rrCrDr)rrrr1)r{rndimr,rwr3) rWrr(rrrrr merged_inputsrs r<r9CsmForConditionalGeneration.prepare_inputs_for_generation}sw< +)')      Y^^q%8\=M=Mo=^=f CC##ZZ7%+ZZ0F%Gzz(+ DM   "/"@MZbLcrvw r;rrrrrrc LU bU OURRn U bU OURRn Ub.URS:XaUR XXH5nUSnUSnSnUR "SUUUUUU U U U S. UD6nUSn[ U [5(a [U *S5OU nURUSS2USS245nSnSnSnSnUbUSS2SS2S4nUR"SUUURRS.UD6nUSS2SS2SS24S :HRS S 9)nUUS SURRS- 24n[RR!US SS9nUR#SS9nUUSUSS- SS24nUUnUR%UUU U U SUS9nUR&nUU-n[)UUUUUR*UR,UR.Ub UR0OSUb UR*OSUb UR,OSUbUR.S9 $SS9 $)a_ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be infered from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "eustlb/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nrrr) rrrr(rrrrrr)r'rr}rrPrGr.rF)rTrN)rrrrr return_dictr) r&r0r+r'r(r)r*r,r-r.r/r1)rFrrrzrwrrHrrrrr}allrSrIrrQrbrr&r$r(r)r*r')rWrrCrrDrr(rrrrrrrrr{backbone_outputsbackbone_hidden_statesrbackbone_logitsr&r0r+depth_decoder_outputsbackbone_labels train_maskdepth_decoder_input_ids train_idxsbackbone_last_hidden_statesdepth_decoder_labelss r<r#CsmForConditionalGeneration.forwardsl2C1N-TXT_T_TqTq$8$D $++JjJj   Y^^q%8 CC)=M*/:M"8,FI..  )%+'/!5)    "2!!48B>SV8W8W~ot4]k ,,'=aPQ>Q'RS ! $  $Q1WoO ..&4;;KaKaekM "!Q(+t388R8@@J&,Z&8>] @Y@Y\]@]>]9]&^ #&(mm&7&78OQW_`&7&a ##++T+:J*@APZ[\P]`aPacdAd*e '#)*#5 $($6$61+F#"3%9 +%7% !"7!;!;  #55D '1",<<*88'22AVAb!6!=!=hl$0+@*O*O$0)>(K(KI^Ij%:%E%E  qu  r;sequence_length target_lengthrM batch_sizec UbUR5S:XaUnU$[R"U5Rn[R"X4XUR S9nUS:wa[R "USS9nU[R"X$R S9URSS5:-nUSSSS2SS24RUSSS5nUbUR5nURSn USS2SS2SS2SU 24USS2SSSS24RUR 5-n U S:Hn USS2SS2SS2SU 24RX5USS2SS2SS2SU 24'U$) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuerMrr)diagonalrrGr)rr7finforfullrtriurreshaperTclonerto masked_fill) rrrrMrrrr min_dtype mask_length padding_masks r<5_prepare_4d_causal_attention_mask_with_cache_positionQCsmForConditionalGeneration._prepare_4d_causal_attention_mask_with_cache_positionFs}<  %.*<*<*>!*C(K*' E*..I** 0Y\j\q\qK!##jjqA 5<< >S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12r;)rrrr rr}r) NNNNNNNNNNNNr)&r2r3r4r5rr|rrrrr& classmethodr-rArr7rrwrr r8rrrrrrrrrrr$r staticmethodrMrr:rrs@r<r r s @1 01&41-1/37;)- NBELL)NBu||,NB'u||4 NB & NB %,,  NBf,0595959 ##"%!!1!12    1 12  !!1!12 >'+/3157;37KO59-1$(,0/35934f ##f u||,f !. f 'u||4 f u//0 f "%tE4E4E/F(F"GHf   1 12f ))*f D>f $D>f 'tnf !!1!12f c5<</0f *+f u'' (!f f P4 444{{ 4  4  44r;r )r?rrwrr )? dataclassesrtypingrrrrr7torch.nnrI cache_utilsr r generationr modeling_flash_attention_utilsr modeling_outputsr rmodeling_utilsrprocessing_utilsrutilsrrrrautorllama.modeling_llamarrrrrrrrconfiguration_csmr r!generation_csmr" get_loggerr2rr$r?rUrlrorrrBrwModulerRrrrr __all__r1r;r<rs "// .)BO-&KK   @.   H % 76 76 76t **  *D   -  X  >  ' z :z z zryy.s !1?s s l  /z//. X"46HX  Xv  r;