o Zh@sddlmZmZmZmZddlZddlmZddlm Z ddl m Z m Z m Z ddlmZddlmZddlmZmZdd lmZdd lmZdd lmZmZmZmZmZdd lm Z dd l!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)ddl*m+Z+m,Z,m-Z-ddl.m/Z/m0Z0m1Z1ddl2m3Z3m4Z4e)5e6Z7GdddeZ8Gdddej9Z:Gdddej9Z;Gddde+ZGd d!d!eZ?e'Gd"d#d#e#Z@Gd$d%d%e@ZAGd&d'd'e0ZBGd(d)d)e3ZCe'd*d+Gd,d-d-e@eZDgd.ZEdS)/)CallableOptionalTupleUnionN)ACT2FN)Cache DynamicCacheEncoderDecoderCache)PretrainedConfig)GenerationMixin)_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPast)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)rope_config_validation)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringcan_return_tuplelogging) GlmAttentionGlmRotaryEmbeddingapply_rotary_pos_emb)LlamaDecoderLayer LlamaModeleager_attention_forward) WhisperModelshift_tokens_rightcsheZdZdZdZdgZddddZ              dfdd ZZS)MoonshineConfiga" This is the configuration class to store the configuration of a [`MoonshineModel`]. It is used to instantiate a Moonshine model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Moonshine [UsefulSensors/moonshine-tiny](https://huggingface.co/UsefulSensors/moonshine-tiny). Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 32768): Vocabulary size of the Moonshine model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`MoonshineModel`]. hidden_size (`int`, *optional*, defaults to 288): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 1152): Dimension of the MLP representations. encoder_num_hidden_layers (`int`, *optional*, defaults to 6): Number of hidden layers in the Transformer encoder. decoder_num_hidden_layers (`int`, *optional*, defaults to 6): Number of hidden layers in the Transformer decoder. encoder_num_attention_heads (`int`, *optional*, defaults to 8): Number of attention heads for each attention layer in the Transformer encoder. decoder_num_attention_heads (`int`, *optional*, defaults to 8): Number of attention heads for each attention layer in the Transformer decoder. encoder_num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `encoder_num_key_value_heads=encoder_num_attention_heads`, the model will use Multi Head Attention (MHA), if `encoder_num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details checkout [this paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `num_attention_heads`. decoder_num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `decoder_num_key_value_heads=decoder_num_attention_heads`, the model will use Multi Head Attention (MHA), if `decoder_num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details checkout [this paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `decoder_num_attention_heads`. pad_head_dim_to_multiple_of (`int`, *optional*): Pad head dimension in encoder and decoder to the next multiple of this value. Necessary for using certain optimized attention implementations. encoder_hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder. decoder_hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. max_position_embeddings (`int`, *optional*, defaults to 512): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. decoder_start_token_id (`int`, *optional*, defaults to 1): Corresponds to the "<|startoftranscript|>" token, which is automatically used when no `decoder_input_ids` are provided to the `generate` function. It is used to guide the model`s generation process depending on the task. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). rope_theta (`float`, *optional*, defaults to 10000.0): The base period of the RoPE embeddings. rope_scaling (`Dict`, *optional*): Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value accordingly. Expected contents: `rope_type` (`str`): The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope', 'llama3'], with 'default' being the original RoPE implementation. `factor` (`float`, *optional*): Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In most scaling types, a `factor` of x will enable the model to handle sequences of length x * original maximum pre-trained length. `original_max_position_embeddings` (`int`, *optional*): Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during pretraining. `attention_factor` (`float`, *optional*): Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, using the `factor` field to infer the suggested value. `beta_fast` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear ramp function. If unspecified, it defaults to 32. `beta_slow` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear ramp function. If unspecified, it defaults to 1. `short_factor` (`List[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to short contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `long_factor` (`List[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to long contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `low_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE `high_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE partial_rotary_factor (`float`, *optional*, defaults to 0.9): Percentage of the query and keys which will have rotary embedding. is_encoder_decoder (`bool`, *optional*, defaults to `True`): Whether the model is used as an encoder/decoder or not. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. bos_token_id (`int`, *optional*, defaults to 1): Denotes beginning of sequences token id. eos_token_id (`int`, *optional*, defaults to 2): Denotes end of sequences token id. Example: ```python >>> from transformers import MoonshineModel, MoonshineConfig >>> # Initializing a Moonshine style configuration >>> configuration = MoonshineConfig().from_pretrained("UsefulSensors/moonshine-tiny") >>> # Initializing a model from the configuration >>> model = MoonshineModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```Z moonshinepast_key_valuesencoder_num_key_value_headsencoder_num_attention_headsencoder_num_hidden_layers)num_key_value_headsnum_attention_headsZnum_hidden_layers Ngelusilu{Gz?T@?Frc s||_||_||_||_||_||_||_|dur|}||_| dur$|} | |_| |_ | |_ | |_ | |_ ||_ ||_||_||_||_||_||_||_||_t|tjd||||d|dS)N) bos_token_id eos_token_idis_encoder_decoderdecoder_start_token_id) vocab_size hidden_sizeintermediate_sizer*decoder_num_hidden_layersr)decoder_num_attention_headsr(decoder_num_key_value_headspad_head_dim_to_multiple_ofencoder_hidden_actdecoder_hidden_actmax_position_embeddingsinitializer_ranger= use_cache rope_theta rope_scalingpartial_rotary_factorr<attention_biasattention_dropoutrsuper__init__)selfr?r@rAr*rBr)rCr(rDrErFrGrHrIr=rJrKrLrMr<rNrOr:r;kwargs __class__r>^/var/www/auris/lib/python3.10/site-packages/transformers/models/moonshine/modular_moonshine.pyrQsF zMoonshineConfig.__init__)r-r.r/r0r0r1r1NNNr2r3r4r5r6Tr7Nr8TFr9r6r) __name__ __module__ __qualname____doc__Z model_typeZkeys_to_ignore_at_inferenceZ attribute_maprQ __classcell__r>r>rTrVr&.sB}r&c2eZdZfddZdejdejfddZZS)MoonshineEncoderMLPcsBt||_t||_t|j|j|_ t|j|j|_ dSN rPrQconfigr activation_fnnnLinearr@rAfc1fc2rRr`Z hidden_actrTr>rVrQs  zMoonshineEncoderMLP.__init__ hidden_statesreturncCs"||}||}||}|Sr^)rdrare)rRrgr>r>rVforwards   zMoonshineEncoderMLP.forwardrWrXrYrQtorchTensorrir[r>r>rTrVr] r]cr\)MoonshineDecoderMLPcsFt||_t||_t|j|jd|_ t|j|j|_ dS)Nrr_rfrTr>rVrQ s  zMoonshineDecoderMLP.__init__rgrhcCs8||}|jddd\}}|||}||}|S)Nr)dim)rdchunkrare)rRrgZgater>r>rVris  zMoonshineDecoderMLP.forwardrjr>r>rTrVrn rmrncseZdZdededededef fdd Z     dd ejd e e ejejfd e ejd e e d e ej de ejde ede eje eje e ejffddZZS)MoonshineAttentionr` layer_idx is_causalr,r+cs~|||dt||||_t|d|j|j|_|jj dur:|jj }||j|d|}||j|_ dSd|_ dS)N)r,r+head_dimr6r) updaterPrQrtgetattrr@r,rur`rEhead_dim_padding)rRr`rsrtr,r+Ztarget_multipleZtarget_head_dimrTr>rVrQs  zMoonshineAttention.__init__Nrgposition_embeddingsattention_maskpast_key_valuecache_positionkey_value_statesrSrhcKs|jdd\}} |||| |jj|jdd} |du} |dur9|j|j } | r6d|j|j <|j }n|j }|dur?|n|} | rT|rT| rT|j |j }|j |j }n7|| |d|jj|jdd}|| |d|jj|jdd}| r|dur||||j d|i\}}| s|\}}t| |||\} }|dur|||d}||||j |\}}t}|jjdkr|jjdkr|d d rtd nt|jj}|jr|dur| dkrdnd }|jd krtjj| d |jf} tjj|d |jf}tjj|d |jf}||| |||f|jsd n|j|j|d|\}}|jd kr/|dd|j f}| || d!}|"|}||fS)Nror6rTr|)sincosr|eagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rr9)Zdropoutscalingrt.)#shapeZq_projviewr`r+ruZ transpose is_updatedgetrscross_attention_cacheself_attention_cacheZ key_cacheZ value_cacheZk_projZv_projrvr r#_attn_implementationlogger warning_oncerrtrxrkrb functionalpadtrainingrOrZreshape contiguousZo_proj)rRrgryrzr{r|r}rSZbszZq_lenZ query_statesZis_cross_attentionrZcurrent_statesZ key_statesZ value_statesrr~Z cache_kwargsZattention_interfacertZ attn_outputZ attn_weightsr>r>rVri0s "           zMoonshineAttention.forward)NNNNN)rWrXrYr&intboolrQrkrlrrr LongTensorrrrir[r>r>rTrVrrsD rrc@s eZdZdS)MoonshineRotaryEmbeddingN)rWrXrYr>r>r>rVrsrcs&eZdZdedeffdd ZZS)MoonshineEncoderLayerr`rscs\t||t||d|j|jd|_t||j|_t j |j dd|_ t j |j dd|_ dS)NFr`rsrtr,r+bias)rPrQrrr)r( self_attnr]rFmlprb LayerNormr@input_layernormpost_attention_layernormrRr`rsrTr>rVrQszMoonshineEncoderLayer.__init__)rWrXrYr&rrQr[r>r>rTrVrsrc seZdZddedeeffdd Z           ddejdeejd eejd eejd eej d eej d ee dee dee deej dee ejejfdee ejejfde ej ee ej ej fffddZZS)MoonshineDecoderLayerNr`rscst|j|_t||d|j|jd|_t||d|j|jd|_t||j |_ t j |jdd|_ t j |jdd|_t j |jdd|_dS)NTrFr)rPrQr@rrrCrDr encoder_attnrnrGrrbrrrfinal_layernormrrTr>rVrQs( zMoonshineDecoderLayer.__init__Frgrzencoder_hidden_statesencoder_attention_mask position_idsencoder_position_idsr{rrJr|ryencoder_position_embeddingsrhc  Ks|}||}|jd|||||| | | d| \}}||}d}|dur<|}||}|j|||||| d\}}||}|}||}||}||}|f}|rW|||f7}|S)N)rgrzrr{rrJr|ry)rgr}rzr{rrJr>)rrrrrr)rRrgrzrrrrr{rrJr|ryrrSZresidualZself_attn_weightsZcross_attn_weightsoutputsr>r>rVrisH        zMoonshineDecoderLayer.forwardr^) NNNNNNFFNNN)rWrXrYr&rrrQrkrlrrrr FloatTensorrir[r>r>rTrVrsP     rc@sLeZdZeZdZdZdZddgZdZ dZ dZ dZ ddZ dejfd d Zd S) MoonshinePreTrainedModelmodel input_valuesTrrcCs|jj}t|tjtjfr%|jjjd|d|j dur#|j j dSdSt|tj tj frD|jj d|j durB|j j dSdSt|tjrc|jjjd|d|jdure|jj|j dSdSdS)Nr9)meanstdg?)r`rI isinstancerbrcConv1dweightdataZnormal_rZzero_ GroupNormrZfill_Z EmbeddingZ padding_idx)rRmodulerr>r>rV _init_weightss"    z&MoonshinePreTrainedModel._init_weights input_lengthscCs@t|ddd}t|ddd}t|ddd}|S)zH Computes the output length of the convolutional layers @r6rr)r)rRrZoutput_conv1_lengthZoutput_conv2_lengthZoutput_conv3_lengthr>r>rV _get_feat_extract_output_lengthssz9MoonshinePreTrainedModel._get_feat_extract_output_lengthsN)rWrXrYr&Z config_classZbase_model_prefixmain_input_nameZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_flash_attn_2Z_supports_sdpaZ_supports_cache_classZ_supports_static_cacherrkrrr>r>r>rVrsrcseZdZdZdZdeffdd ZdejfddZ d ejfd d Z e dde e jd e e jde ede edeedef ddZZS)MoonshineEncoderz Transformer encoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MoonshineEncoderLayer`] Args: config: MoonshineConfig rr`cst|_j}tjd|dddd|_tj|d|ddd |_tjd||ddd |_tj d|d d |_ t d |_ t fd dtjD|_tj|dd|_d|_|dS)Nr6rrF) kernel_sizestriderrrr)rrgh㈵>)Z num_groupsZ num_channelsZepsr`cg|]}t|qSr>)r.0idxrr>rV 8z-MoonshineEncoder.__init__..r)rPrQr`r@rbrconv1conv2conv3r groupnormr rotary_emb ModuleListranger*layersr layer_normgradient_checkpointing post_init)rRr`Z embed_dimrTrrVrQ+s   zMoonshineEncoder.__init__rhcC|jSr^rrRr>r>rVget_input_embeddings?z%MoonshineEncoder.get_input_embeddingsvaluecC ||_dSr^r)rRrr>r>rVset_input_embeddingsB z%MoonshineEncoder.set_input_embeddingsNrzroutput_hidden_statesflash_attn_kwargscKs|dur|n|jj}|dur|n|jj}|durtd|d}tj||}| |}tj | |}tj | |}| ddd}|dur||jd}d}|ddd|fdd|f}|jjd krv|d krs|nd}n|jjd kr|st||j}nt||j}tjd|jd|jd d} ||| } |rd nd} |rd nd} |jD]#} |r| |f7} | |f|| || d|}|d}|r| |df7} q||}|r| |f7} t|| | dS)a Args: input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding indices in `input_values`. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzYou must specify input_values.r6rrro.flash_attention_2r9rdevicer>)rzrrrylast_hidden_staterg attentions)r`rr ValueError unsqueezerbrtanhrrr2rrZpermuterrranyrdtyper rkarangerrrrr)rRrrzrrrrgmask_lendownsample_striderryall_hidden_statesall_self_attnsZ encoder_layer layer_outputsr>r>rVriEsb            zMoonshineEncoder.forward)NNNN)rWrXrYrZrr&rQrbModulerrrrrkrrlrrrrrir[r>r>rTrVr!s0rcseZdZdZdeffdd Z           ddeejdeej deejdee d eej d ee d ee d ee d eejdeej deej de edeeeffddZZS)MoonshineDecoder input_idsr`csBttjjdd|_tfddtjD|_ dS)NFrcrr>)rrrr>rVrrz-MoonshineDecoder.__init__..) rPrQrbrr@normrrrBrrRr`rTrrVrQs  zMoonshineDecoder.__init__Nrzrr' inputs_embedsrJrrr|rrrrhc  Ksv|dur|n|jj}|dur|n|jj}|dur|n|jj}|du|duAr*td|jr9|jr9|r9tdd}|durB| |}|rS|durSt } t }t | |}| duro|dur_| nd}t j|||jd|jd} |durx| d}|||| ||}|}|||}|rdnd}|rdnd}|r| durdnd}| dur| jd }d }| d dd|fd d|f} |jjd kr| d kr| nd} n|jjdkr|st| |j|jd } n t| |j|jd } |jD]5}|r||f7}||f|| | ||||| |d | }|d}|r||df7}| dur||df7}q||}|r-||f7}t||r4|nd|||dS)a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding indices in `encoder_hidden_states`. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Frr6rr>r.rr9r) rzrrrr{rrJr|ryr)rr'rgrcross_attentions)r`rrrJrrrrrZ embed_tokensr r Zget_seq_lengthrkrrrrZ_update_causal_maskrrrrrr rrr)rRrrzrr'rrJrrr|rrrrrZpast_seen_tokensZ causal_maskrgryrrZall_cross_attentionsrrZ decoder_layerrr>r>rVris                 zMoonshineDecoder.forward) NNNNNNNNNNN)rWrXrYrr&rQrrkrrlrrrrrrrrrir[r>r>rTrVrsR       rc@seZdZee            ddeejdeejdeejdeejdee e ejdee e e ejfdee ejd ee ejd ee d ee d ee d eejde fddZdS)MoonshineModelNrrzdecoder_input_idsdecoder_attention_maskencoder_outputsr'decoder_inputs_embedsdecoder_position_idsrJrrr|rhc  Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur,|j||| | d}n"t|tsNt|dt|dkr?|dndt|dkrJ|dndd}|j||||j |||| | | | d } t | j | j | j | j | j|j |j |j dS) a\ input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, MoonshineModel >>> from datasets import load_dataset >>> model = MoonshineModel.from_pretrained("UsefulSensors/moonshine-tiny") >>> feature_extractor = AutoFeatureExtractor.from_pretrained("UsefulSensors/moonshine-tiny") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_values = inputs.input_values >>> decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id >>> last_hidden_state = model(input_values, decoder_input_ids=decoder_input_ids).last_hidden_state >>> list(last_hidden_state.shape) [1, 2, 288] ``` N)rzrrrr6rr) rrzrrr'rrrJrrr|)rr'decoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentions)r`rrrJencoderrrlendecoderrrr'rgrr)rRrrzrrrr'rrrJrrr|Zdecoder_outputsr>r>rVri;sPP zMoonshineModel.forward) NNNNNNNNNNNN)rWrXrYrrrrkrrrrr rrrir>r>r>rVr:sT      rzj The Moonshine Model with a language modeling head. Can be used for automatic speech recognition. )Z custom_introcs"eZdZdgZdeffdd ZddZddZd d Zd d Z d e j fddZ e e             d deejdeejdeejdeejdeeeejdeeeeejfdeeejdeeejdeedeedeedeejdeejd efddZZS)!!MoonshineForConditionalGenerationzproj_out.weightr`cs8t|t||_tj|j|jdd|_| dS)NFr) rPrQrrrbrcr@r?proj_outrrrTr>rVrQs   z*MoonshineForConditionalGeneration.__init__cC |jSr^)r get_encoderrr>r>rVrrz-MoonshineForConditionalGeneration.get_encodercCrr^)r get_decoderrr>r>rVrrz-MoonshineForConditionalGeneration.get_decodercCrr^rrr>r>rVget_output_embeddingsrz7MoonshineForConditionalGeneration.get_output_embeddingscCrr^r)rRZnew_embeddingsr>r>rVset_output_embeddingsrz7MoonshineForConditionalGeneration.set_output_embeddingsrhcCrr^)rrrr>r>rVrrz6MoonshineForConditionalGeneration.get_input_embeddingsNrrzrrrr'rrrJrrr|labelscCs| dur|dur|durt| |jj|jj}|j||||||||| | | | d }||j}d}| dur=|j|| |jjd}t |||j |j |j |j |j|j|jd S)a input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> import torch >>> from transformers import AutoProcessor, MoonshineForConditionalGeneration >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("UsefulSensors/moonshine-tiny") >>> model = MoonshineForConditionalGeneration.from_pretrained("UsefulSensors/moonshine-tiny") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = processor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_values = inputs.input_values >>> generated_ids = model.generate(input_values, max_new_tokens=100) >>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> transcription 'Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.' ```N) rzrrrr'rrrJrrr|)logitsr r?) lossr r'rrrrrr)r%r`Z pad_token_idr=rrrZ loss_functionr?rr'rrrrrr)rRrrzrrrr'rrrJrrr|r rr r r>r>rVrisDY z)MoonshineForConditionalGeneration.forward) NNNNNNNNNNNNN)rWrXrYZ_tied_weights_keysr&rQrrrr rbrrrrrrkrrrrr rrrir[r>r>rTrVrsh      r)r&rrr)FtypingrrrrrkZtorch.nnrbZ activationsrZ cache_utilsrr r Zconfiguration_utilsr Z generationr Zmodeling_attn_mask_utilsr rZmodeling_flash_attention_utilsrZmodeling_layersrZmodeling_outputsrrrrrZmodeling_rope_utilsrZmodeling_utilsrrZprocessing_utilsrutilsrrrZglm.modeling_glmrrr Zllama.modeling_llamar!r"r#Zwhisper.modeling_whisperr$r%Z get_loggerrWrr&rr]rnrrrrrrrrrr__all__r>r>r>rVsR         NtX%