o Zh"@s^dZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z mZddlZddlmZddlmZddlmZddlmZmZmZmZmZmZdd lmZmZmZm Z dd l!m"Z"m#Z#dd l$m%Z%m&Z&m'Z'm(Z(m)Z)dd l*m+Z+dd l,m-Z-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4m5Z5e#rddl!m6Z6erddl7m8Z8e.9e:Z;eGddde(Zde>fddZ?Gdddej@ZAGdddej@ZBGd d!d!eBZCGd"d#d#eBZDeBeDeCd$ZEGd%d&d&ej@ZFe-Gd'd(d(e+ZGGd)d*d*eGZHe-Gd+d,d,eGZIe-d-d.Gd/d0d0eGeZJe-d1d.Gd2d3d3e+eZKgd4ZLdS)5zPyTorch Musicgen model.N) dataclass) TYPE_CHECKINGAnyDictListOptionalTupleUnion)CrossEntropyLoss)ACT2FN)%ClassifierFreeGuidanceLogitsProcessorGenerationConfigGenerationMixinGenerationModeLogitsProcessorListStoppingCriteriaList)_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa!_prepare_4d_causal_attention_mask*_prepare_4d_causal_attention_mask_for_sdpa)!flash_attn_supports_top_left_maskis_flash_attn_available)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions ModelOutputSeq2SeqLMOutput)PreTrainedModel)auto_docstringlogging) AutoConfig AutoModel)MusicgenConfigMusicgenDecoderConfig)_flash_attention_forward) BaseStreamerc@sFeZdZUdZdZeejed<dZ e ej ed<dZ e e ed<dS)MusicgenUnconditionalInputa% Args: encoder_outputs (`Tuple[torch.FloatTensor]` of length 1, with tensor shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the text encoder model. attention_mask (`torch.LongTensor`) of shape `(batch_size, sequence_length)`, *optional*): Encoder attention mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: 1 for tokens that are **not masked**, 0 for tokens that are **masked**. guidance_scale (`float`, *optional*): Guidance scale for classifier free guidance, setting the balance between the conditional logits (predicted from the prompts) and the unconditional logits (predicted without prompts). Nencoder_outputsattention_maskguidance_scale)__name__ __module__ __qualname____doc__r+rtorch FloatTensor__annotations__r,r LongTensorr-floatr7r7]/var/www/auris/lib/python3.10/site-packages/transformers/models/musicgen/modeling_musicgen.pyr*Cs  r* input_ids pad_token_iddecoder_start_token_idcCst|dd}||j}|dddf|dddf<|dur$td||d<|dur0td||d k||S) z1 Shift input ids one token to the right. r%r!.NzSMake sure to set the decoder_start_token_id attribute of the model's configuration..rzIMake sure to set the pad_token_id attribute of the model's configuration.) transposeZ new_zerosshapeclone ValueErrorZ masked_fill_)r9r:r;Zshifted_input_idsr7r7r8shift_tokens_rightVs   rCcspeZdZdZdedeffdd ZdedefddZededefd d Ze dd e j d efddZ Z S)%MusicgenSinusoidalPositionalEmbeddingzDThis module produces sinusoidal positional embeddings of any length. num_positions embedding_dimcs t||_|||dSN)super__init__rF make_weights)selfrErF __class__r7r8rIms z.MusicgenSinusoidalPositionalEmbedding.__init__num_embeddingscCs@|||}t|dr|j|jj|jjd}|jd|dddS)NweightsdtypedeviceF) persistent) get_embeddinghasattrtorOrQrRZregister_buffer)rKrNrFZ emb_weightsr7r7r8rJrs  z2MusicgenSinusoidalPositionalEmbedding.make_weightscCs|d}td|d}ttj|tjd| }tj|tjdd|d}tjt |t |gdd |d}|ddkrUtj|t |dgdd}| tS)z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". r!i'r%rQrdimr<)mathlogr2exparangeZint64r6Z unsqueezecatcossinviewzerosrVZget_default_dtype)rNrFZhalf_dimZembr7r7r8rTzs $& z3MusicgenSinusoidalPositionalEmbedding.get_embeddingrr9past_key_values_lengthcCs`|\}}}t|||j}||jdkr$|||j|j|j d| d S)Nrr<) sizer2r]rVrRrOrJoffsetrFZ index_selectradetach)rKr9rcbsz codebooksseq_lenZ position_idsr7r7r8forwards z-MusicgenSinusoidalPositionalEmbedding.forward)r)r.r/r0r1intrIrJ staticmethodrTr2no_gradTensorrj __classcell__r7r7rLr8rDjs rDcseZdZdZ     ddededed ed ed ed eeffd d Z de j dedefddZ     dde j dee j dee e j dee j dee j dede e j ee j ee e j ffddZZS)MusicgenAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsdropout is_decoderbias is_causalconfigcst||_||_||_|||_||_|j||jkr*td|jd|d|jd|_||_ ||_ t j |||d|_ t j |||d|_t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rv)rHrIrrrsrthead_dimrxrBscalingrurwnnLineark_projv_projq_projout_proj)rKrrrsrtrurvrwrxrLr7r8rIs&    zMusicgenAttention.__init__tensorrirgcCs ||||j|jddS)Nr%r!)rarsrzr? contiguousrKrrirgr7r7r8_shapes zMusicgenAttention._shape hidden_stateskey_value_statespast_key_valuer,layer_head_maskoutput_attentionsreturncCsr|du}|\}} } |||j} |r.|dur.|djd|jdkr.|d} |d} nZ|rE|||d|} |||d|} nC|durt|||d|} |||d|} tj|d| gdd} tj|d| gdd} n|||d|} |||d|} |j r| | f}||j d|j f}|| | |j |} | j |} | j |} | d}t| | dd}|||j | |fkrtd||j | |fd||dur ||d| |fkrtd |d| |fd|| ||j | ||}| ||j | |}tjj|dd}|durL||j fkr1td |j fd|| dddd| ||j | |}| ||j | |}|rc| ||j | |}| ||j | |}nd}tjj||j|jd }t|| }|||j | |j fkrtd ||j | |j fd|| ||j | |j }|dd}| || |j}||}|||fS) #Input shape: Batch x Time x ChannelNrr!r%r<rXz$Attention weights should be of size , but is z!Attention mask should be of size z/Head mask for a single layer should be of size ptraining `attn_output` should be of size )rdrr{r@rr~rr2r^rursrzrareshapeZbmmr?rBr| functionalZsoftmaxrtrrrr)rKrrrr,rris_cross_attentionrgtgt_len_ query_states key_states value_statesZ proj_shapeZsrc_len attn_weightsZattn_weights_reshapedZ attn_probs attn_outputr7r7r8rjs        "    zMusicgenAttention.forward)rqFTFNNNNNF)r.r/r0r1rkr6boolrr&rIr2rnrrrjror7r7rLr8rpsV rpcseZdZdZfddZdejdedefddZ dd ejd e ejd e e ejde ejde ejde de eje eje e ejffddZ Z S)MusicgenFlashAttention2aL Musicgen flash attention module. This module inherits from `MusicgenAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cstj|i|t|_dSrG)rHrIr_flash_attn_uses_top_left_maskrKargskwargsrLr7r8rI<s z MusicgenFlashAttention2.__init__rrirgcCs||||j|jSrG)rarsrzrr7r7r8_reshapeDsz MusicgenFlashAttention2._reshapeNFrrrr,rrrc Cs>|du}|\}} } |||d|} |r8|dur8|djd|jdkr8|ddd} |ddd} nb|rO|||d|} |||d|} nK|dur|||d|} |||d|} tj|ddd| gdd} tj|ddd| gdd} n|||d|} |||d|} |j r| dd| ddf}| jd}|dur||djd7}| j }|tj krt rt }nt|jdr|jj}n|jjj }td|d | |} | |} | |} t| | | || |jr|jnd |j|jd }||| d}||}|sd}|||fS) Nr<rr!r%rX_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .rq)rtrwZuse_top_left_mask)rdrrr@r?r~rr2r^rurQZfloat32Zis_autocast_enabledZget_autocast_gpu_dtyperUrxrweightlogger warning_oncerVr(rrtrwrrr)rKrrrr,rrrrgZq_lenrrrrZ kv_seq_lenZ input_dtypeZ target_dtyperrr7r7r8rjGsl             zMusicgenFlashAttention2.forwardr)r.r/r0r1rIr2rnrkrrrrrjror7r7rLr8r5s0  rcseZdZ     d dejdeejdeeejdeejdeejded eejeejeeejfffd d ZZ S) MusicgenSdpaAttentionNFrrrr,rrrcsF|rtdtj|||||dS|dur7|jgddt|jjk r7tdtj|||||dS|du}| \}} } | |} |rb|durb|dj d|j d krb|d} |d } nZ|ry| ||d |} | ||d |} nC|dur| ||d |} | ||d |} tj|d| gdd} tj|d | gdd} n| ||d |} | ||d |} |jr| | f}| | | |} |jr|dur| d krd nd }tjjj| | | ||jr|jnd |d}| ||j| |jfkr td||j| |jfd| |d d}||| |j}||}|d|fS)raMusicgenModel is using MusicgenSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rrr,rN)r%r!r rXa`torch.nn.functional.scaled_dot_product_attention` does not support having an empty attention mask. Falling back to the manual attention implementation. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.Note that this probably happens because `guidance_scale>1` or because you used `get_unconditional_inputs`. See https://github.com/huggingface/transformers/issues/31189 for more information.rr!r%r<TFrq) attn_maskZ dropout_prwrr)rrrHrjmeanr2ZfinforQminanyrdrr@rr~rr^rurwr|rZscaled_dot_product_attentionrrtrsrzrBr?rrrr)rKrrrr,rrrrgrrrrrrwrrLr7r8rjs         zMusicgenSdpaAttention.forwardr) r.r/r0r2rnrrrrjror7r7rLr8rs* r)eagersdpaflash_attention_2cseZdZdeffdd Z        ddejdeejd eejd eejd eejd eejd eeejdee dee dejfddZ Z S)MusicgenDecoderLayerrxc st|j|_t|j|j|j|jddd|d|_|j |_ t |j |_ |j |_ t|j|_t|j|j|j|jdd|d|_t|j|_tj|j|jdd|_tj|j|jdd|_t|j|_dS)NTF)rrrsrtrurvrwrx)rtrurvrxry)rHrI hidden_sizerrMUSICGEN_ATTENTION_CLASSES_attn_implementationZnum_attention_headsZattention_dropout self_attnrtr Zactivation_function activation_fnactivation_dropoutr| LayerNormself_attn_layer_norm encoder_attnencoder_attn_layer_normr}Zffn_dimfc1fc2final_layer_normrKrxrLr7r8rI1s6  zMusicgenDecoderLayer.__init__NFTrr,encoder_hidden_statesencoder_attention_maskrcross_attn_layer_head_maskrr use_cacherc CsZ|} ||}|dur|ddnd} |j|| |||d\}} } tjj||j|jd}| |}d}d}|durk|} ||}|durH|ddnd}|j||||||d\}}}tjj||j|jd}| |}| |} |} ||}| | |}tjj||j |jd}| |}tjj||j|jd}| |}|f}|r|| |f7}| r|| f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. Nr!)rrr,rrrr)rrr,rrr) rrr|rrtrrrrrrrr)rKrr,rrrrrrrZresidualZself_attn_past_key_valueZself_attn_weightsZpresent_key_valueZcross_attn_present_key_valueZcross_attn_weightsZcross_attn_past_key_valueoutputsr7r7r8rjQsT        zMusicgenDecoderLayer.forward)NNNNNNFT) r.r/r0r'rIr2rnrrrrjror7r7rLr8r0s>#    rc@s0eZdZeZdZdZddgZdZdZ ddZ dS)MusicgenPreTrainedModelmodelTrrpcCs|jj}t|tjtjfr%|jjjd|d|j dur#|j j dSdSt|tj rD|jjjd|d|j durF|jj|j  dSdSdS)Nrq)rstd) rxZinitializer_factor isinstancer|r}ZConv1drdataZnormal_rvZzero_ EmbeddingZ padding_idx)rKmodulerr7r7r8 _init_weightss   z%MusicgenPreTrainedModel._init_weightsN) r.r/r0r' config_classbase_model_prefixsupports_gradient_checkpointingZ_no_split_modules_supports_flash_attn_2_supports_sdparr7r7r7r8rs rcseZdZdZdeffdd ZddZddZe dd e e j d e e j d e e j d e e j de e j de e j de eee j de e j de ede ede ede edeeeffddZZS)MusicgenDecoderzw Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MusicgenDecoderLayer`] rxcstj|_j|_j|_j|_j|_j r#t jnd|_ j dtfddtjD|_tjj|_tfddtjD|_tj|_j|_d|_|dS)N?r%csg|] }tjqSr7)r|rr.0rrxrrr7r8 sz,MusicgenDecoder.__init__..csg|]}tqSr7)rrrxr7r8rF)rHrIrt layerdropZmax_position_embeddingsZmax_target_positionsrZd_model num_codebooksZscale_embeddingrZsqrtZ embed_scale vocab_sizer| ModuleListrange embed_tokensrDembed_positionsZnum_hidden_layerslayersr layer_normrattn_implementationgradient_checkpointing post_initrrLrr8rIs(    zMusicgenDecoder.__init__cC|jSrGrrKr7r7r8get_input_embeddingsz$MusicgenDecoder.get_input_embeddingscC ||_dSrGrrKvaluer7r7r8set_input_embeddings z$MusicgenDecoder.set_input_embeddingsNr9r,rr head_maskcross_attn_head_maskpast_key_values inputs_embedsrroutput_hidden_states return_dictrc s| dur| njj} | dur| njj} | dur| njj} | dur$| njj} |dur4|dur4td|durN|dj|jdj\} }}| |f}n|durh| dd}|ddddddfntd|dury|ddjdnd}|durt fddt |D}j d kr|durd|vr|nd}nj d kr|dur| st ||||}nt||||}|dur|durj d krd|vr|nd}n j d kr|dur| st||j|dd }n t||j|dd }|}|||j}tjj|jjd }jrjr| rtd d} | rdnd}| r&dnd}| r2|dur2dnd}| r9dnd}t||gddgD]+\}}|durn| dtjkrntd|dtjd| ddqDtjD]\}}| r||f7}t !dd}jr|j"krqu|dur||nd}jrȈjrȈ#|j$|||||dur||nd|dur||ndd| | }n||||||dur||nd|dur||nd|| | d }|d}| r||| rdndf7}| r||df7}|dur||df7}qu%|}| r||f7}| r$|nd}| s7t&dd|||||fDSt'|||||dS) input_ids (`torch.LongTensor` of shape `(batch_size * num_codebooks, sequence_length)`): Indices of input sequence tokens in the vocabulary, corresponding to the sequence of audio codes. Indices can be obtained by encoding an audio prompt with an audio encoder model to predict audio codes, such as with the [`EncodecModel`]. See [`EncodecModel.encode`] for details. [What are input IDs?](../glossary#input-ids) The `input_ids` will automatically be converted from shape `(batch_size * num_codebooks, target_sequence_length)` to `(batch_size, num_codebooks, target_sequence_length)` in the forward pass. If you obtain audio codes from an audio encoding model, such as [`EncodecModel`], ensure that the number of frames is equal to 1, and that you reshape the audio codes from `(frames, batch_size, num_codebooks, target_sequence_length)` to `(batch_size * num_codebooks, target_sequence_length)` prior to passing them as `input_ids`. encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer<zEYou have to specify either decoder_input_ids or decoder_inputs_embedsrr!cs&g|]}j|dd|fqSrGr)rcodebookinputrKr7r8r4s&z+MusicgenDecoder.forward..rr)rrz[`use_cache=True` is incompatible with gradient checkpointing`. Setting `use_cache=False`...Fr7rrzThe `z` should be specified for z layers, but it is for rr%)r,rrrrrrrr css|] }|dur|VqdSrGr7)rvr7r7r8 sz*MusicgenDecoder.forward..last_hidden_staterr attentionscross_attentions)(rxrrruse_return_dictrBrrr@rdsumrrrrrrQrrrVrRr|rrtrrrrziplenr enumeraterandomuniformrZ_gradient_checkpointing_funcrjrtupler) rKr9r,rrrrrrrrrrrgrriZ input_shapercZ positionsrZall_hidden_statesZall_self_attnsZall_cross_attentionsZnext_decoder_cacherZ mask_nameidxZ decoder_layerZdropout_probabilityrZ layer_outputsZ next_cacher7rr8rjs5                zMusicgenDecoder.forward NNNNNNNNNNNN)r.r/r0r1r'rIrrrrr2r5rnr3rrr rrjror7r7rLr8rsZ      rcseZdZdeffdd ZddZddZdd Ze dd e e j d e e j d e e j de e j de e j de e j de eee j de e j de ede ede ede edeeeffddZZS) MusicgenModelrxcs"t|t||_|dSrG)rHrIrdecoderrrrLr7r8rIs   zMusicgenModel.__init__cC|jjSrGr rrr7r7r8rz"MusicgenModel.get_input_embeddingscC ||j_dSrGrrr7r7r8r z"MusicgenModel.set_input_embeddingscCrrGr rr7r7r8 get_decoderrzMusicgenModel.get_decoderNr9r,rrrrrrrrrrrc Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} | dur$| n|jj} |j||||||||| | | | d } | s=| St| j| j| j | j | j dS)rN) r9r,rrrrrrrrrrr) rxrrrrr rrrrrr)rKr9r,rrrrrrrrrrdecoder_outputsr7r7r8rjs85zMusicgenModel.forwardr )r.r/r0r'rIrrrrrr2r5rnr3rrr rrjror7r7rLr8r sZ      r zK The MusicGen decoder model with a language modelling head on top. )Z custom_introc seZdZdeffdd ZddZddZdd Zd d Zd d Z ddZ e             d3de e jde e jde e jde e jde e jde e jde eee jde e jde e jde ede ede ede edeeeffdd Z       !  d4d"d#Z d5de jd$ed%e efd&d'Zed(d)Ze       d6d*e e jd+e ed,e ed-e ed.e ed/e d0f d1d2ZZ S)7MusicgenForCausalLMrxcsJtt|_j|_tfddtjD|_| dS)Ncs g|] }tjjjddqS)Fry)r|r}rrrrr7r8r1s z0MusicgenForCausalLM.__init__..) rHrIr rrr|rrlm_headsrrrLrr8rI*s   zMusicgenForCausalLM.__init__cCs |jjjSrGrr rrr7r7r8r7rz(MusicgenForCausalLM.get_input_embeddingscCs||jj_dSrGrrr7r7r8r:sz(MusicgenForCausalLM.set_input_embeddingscCrrGrrr7r7r8get_output_embeddings=rz)MusicgenForCausalLM.get_output_embeddingscCrrGrrKZnew_embeddingsr7r7r8set_output_embeddings@rz)MusicgenForCausalLM.set_output_embeddingscCrrGrr )rKr r7r7r8 set_decoderCrzMusicgenForCausalLM.set_decodercCrrGrrr7r7r8rFrzMusicgenForCausalLM.get_decoderNr9r,rrrrrrlabelsrrrrrc s| dur| n|jj} | dur |dur |dur t| |jj|jj}|j||||||||| | | | d }|dtjfdd|jDdd}d}| dur|dddd| j d df}t }tj g|j d}| | |jjkd } t|jjD]%}|dd|fd |j d }| d |fd }||||7}qv||jj}|jd g|j d dR}| s|f|dd}|dur|f|S|St|||j|j|j|jd S)ar input_ids (`torch.LongTensor` of shape `(batch_size * num_codebooks, sequence_length)`): Indices of input sequence tokens in the vocabulary, corresponding to the sequence of audio codes. Indices can be obtained by encoding an audio prompt with an audio encoder model to predict audio codes, such as with the [`EncodecModel`]. See [`EncodecModel.encode`] for details. [What are input IDs?](../glossary#input-ids) The `input_ids` will automatically be converted from shape `(batch_size * num_codebooks, target_sequence_length)` to `(batch_size, num_codebooks, target_sequence_length)` in the forward pass. If you obtain audio codes from an audio encoding model, such as [`EncodecModel`], ensure that the number of frames is equal to 1, and that you reshape the audio codes from `(frames, batch_size, num_codebooks, target_sequence_length)` to `(batch_size * num_codebooks, target_sequence_length)` prior to passing them as `input_ids`. encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N) r,rrrrrrrrrrrcsg|]}|qSr7r7)rheadrr7r8rrz/MusicgenForCausalLM.forward..r%rXrRr>r<.r!)losslogitsrrrr)rxrrCr: bos_token_idrr2stackrr@r rbrRZ masked_fillrrrrarrrrrr)rKr9r,rrrrrrrrrrrrrZ lm_logitsr#r$Zloss_fctrZcodebook_logitsZcodebook_labelsoutputr7r!r8rjIsT<"" zMusicgenForCausalLM.forwardTc Ks| dur|j||jj|jjd\}} ||| }| dur.| dkr.|d}|dur.|d}|dur<|ddddf}||||||||dS)Nr: max_lengthr%r!r%r<)r9r,rrrrrr)build_delay_pattern_maskgeneration_configr:r)apply_delay_pattern_maskrepeat) rKr9r,rrrrrrdelay_pattern_maskr-rr7r7r8prepare_inputs_for_generations,   z1MusicgenForCausalLM.prepare_inputs_for_generationr:r)cCs>|d|j|jd}|j\}}}|dur|n|jj}tj|||ftj|jdd}|j j dkr4|dn|}|d|dkrN|||d|||dfSt |D]N} |j j dkrn|dd| f|dd| | || f<qR|ddd| f|ddd| | || f<|ddd| df|ddd| d| || f<qRtj tj||ftj d||dd} | ttj||ftj d} |j j dkr| jddd } | |j} | || |}|dddddf} | dkdddf} t| dkrt| }n|}|||d}|d d|f||d}||fS) aDBuild a delayed pattern mask to the input_ids. Each codebook is offset by the previous codebook by one, giving a delayed pattern mask at the start of sequence and end of sequence. Take the example where there are 4 codebooks and a max sequence length of 8, we have the delayed pattern mask of shape `(codebooks, seq_len)`: - [P, -1, -1, -1, -1, P, P, P] - [P, P, -1, -1, -1, -1, P, P] - [P, P, P, -1, -1, -1, -1, P] - [P, P, P, P, -1, -1, -1, -1] where P is the special padding token id and -1 indicates that the token is valid for prediction. If we include a prompt (decoder input ids), the -1 positions indicate where new tokens should be predicted. Otherwise, the mask is set to the value in the prompt: - [P, a, b, -1, -1, P, P, P] - [P, P, c, d, -1, -1, P, P] - [P, P, P, e, f, -1, -1, P] - [P, P, P, P, g, h, -1, -1] where a-h indicate the input prompt (decoder input ids) that are offset by 1. Now, we only override the -1 tokens in our prediction. r<NrPr!r%rW)ZdiagonalrrX.)rrr@r,r)r2oneslongrRrxaudio_channelsrZtriurZtrilrepeat_interleaverVZnonzerorr)rKr9r:r)rgrriZinput_ids_shiftedZchannel_codebooksrZ delay_patternmaskZfirst_codebook_idsZ start_idsZfirst_start_idZ pattern_maskr7r7r8r+s:    (.8  z,MusicgenForCausalLM.build_delay_pattern_maskcCs0|jd}|dd|f}t|dk||}|S)zApply a delay pattern mask to the decoder input ids, only preserving predictions where the mask is set to -1, and otherwise setting to the value detailed in the mask.r<.N)r@r2where)r9Zdecoder_pad_token_maskrir7r7r8r-<s z,MusicgenForCausalLM.apply_delay_pattern_maskinputsr,logits_processorstopping_criteria synced_gpusstreamerr)cKst|dur|j}t|}|jdi|}||||dur%|nt}|dur.|nt}d|v} |dddu} | ||j |\} } }| j d|j } |j || | jd|j|d<|j|d<|dddurv| rv|| |||d<| j d}|d duo|jdu}|d duo|jdu}|j|||| | |d }|j| |j|jd \} }|dur|| ||d <|}|jdur|jdkr|t|jd|_|j||| d|| jd}|j||d}|tjtj fvr |j!d| |j"d|\} }|j#| f|||||d|}nt$d|j%r|j&}n|}|'||d }|||j(k)| |j d}|j%r8||_&|S|S)5 Generates sequences of token ids for models with a language modeling head. Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the model's default generation configuration. You can override any `generation_config` by passing the corresponding parameters to generate(), e.g. `.generate(inputs, num_beams=4, do_sample=True)`. For an overview of generation strategies and code examples, check out the [following guide](./generation_strategies). Parameters: inputs (`torch.Tensor` of varying shape depending on the modality, *optional*): The sequence used as a prompt for the generation or as model inputs to the encoder. If `None` the method initializes it with `bos_token_id` and a batch size of 1. For decoder-only models `inputs` should be in the format `input_ids`. For encoder-decoder models *inputs* can represent any of `input_ids`, `input_values`, `input_features`, or `pixel_values`. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default will be used, which had the following loading priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. stopping_criteria (`StoppingCriteriaList`, *optional*): Custom stopping criteria that complement the default stopping criteria built from arguments and a generation config. If a stopping criteria is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. synced_gpus (`bool`, *optional*, defaults to `False`): Whether to continue running the while loop until max_length (needed to avoid deadlocking with `FullyShardedDataParallel` and DeepSpeed ZeRO Stage 3). streamer (`BaseStreamer`, *optional*): Streamer object that will be used to stream the generated sequences. Generated tokens are passed through `streamer.put(token_ids)` and the streamer is responsible for any further processing. kwargs (`Dict[str, Any]`, *optional*): Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder_*. Return: [`~utils.ModelOutput`] or `torch.LongTensor`: A [`~utils.ModelOutput`] (if `return_dict_in_generate=True` or when `config.return_dict_in_generate=True`) or a `torch.FloatTensor`. If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible [`~utils.ModelOutput`] types are: - [`~generation.GenerateDecoderOnlyOutput`], - [`~generation.GenerateBeamDecoderOnlyOutput`] If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible [`~utils.ModelOutput`] types are: - [`~generation.GenerateEncoderDecoderOutput`], - [`~generation.GenerateBeamEncoderDecoderOutput`] Nr+r,rr"rr-r<r) min_lengthr,has_default_max_lengthhas_default_min_lengthmodel_input_name inputs_tensorinput_ids_lengthr(r/r%r,Zinput_ids_seq_lengthZencoder_input_idsZprefix_allowed_tokens_fnr8rRr,r9)r9 expand_sizer8r9r,r:r;Got incompatible mode for generation, should be one of greedy or sampling. Ensure that beam search is de-activated by setting `num_beams=1` and `num_beam_groups=1`.r7)*r,copydeepcopyupdatevalidate_validate_model_kwargsrrget_prepare_model_inputsr%r@r_prepare_special_tokensrRrr-&_prepare_attention_mask_for_generationr)r=_prepare_generated_lengthr+_decoder_start_token_tensorputcpuget_generation_modeappendr _get_logits_processor_get_stopping_criteriarSAMPLE GREEDY_SEARCH_expand_inputs_for_generationnum_return_sequences_samplerBreturn_dict_in_generate sequencesr-_pad_token_tensorr)rKr7r,r8r9r:r;r model_kwargsrequires_attention_maskkwargs_has_attention_maskr9rA batch_sizerCr?r@r/generation_moder output_idsr7r7r8generateEsK          zMusicgenForCausalLM.generate) NNNNNNNNNNNNN) NNNNNNTNNrGNNNNNN)!r.r/r0r'rIrrrrrrrrr2r5rnr3rrr rrjr0rkr+rlr-rmrrrrhror7r7rLr8r$s        | / K rz_ The composite MusicGen model with a text encoder, audio encoder and Musicgen decoder, c$sFeZdZeZdZdZdZdZdZ    dXde ede e de e de e ffd d Z d d Zd dZddZddZddZddZddZddZe   dYde ede ede ede fdd Ze               dZde ejd!e ejd"e ejd#e ejd$e ejd%e ejd&e eejd'e eeejd(e ejd)e ejd*e ejd+e ed,e ed-e ed.e ede ee!ff d/d0Z"          d[d1d2Z#   dYd3e$d4ed5e%eej&fd6e e$d7e e$d8ej'deeje%eej&fffd9d:Z(d;ej&d4e edZ+ d\d4e efd?d@Z,d*ej&fdAdBZ-dCdDZ.dEdFZ/dGdHZ0   dYdIe ej&d7e e$d5e e%eej&fdejfdJdKZ1 d]d6e e e$e2e$fd7e e$de$fdLdMZ3e4      d^dIe ej&d