o Zh @sddlZddlZddlmZmZmZddlZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZmZdd lmZmZmZmZmZmZdd lmZdd lmZm Z m!Z!d dl"m#Z#eroddlm$Z$e!%e&Z'Gddde j(Z)Gddde j(Z*Gddde j(Z+Gddde j(Z,Gddde j(Z-Gddde j(Z.Gddde j(Z/Gddde/Z0Gd d!d!e/Z1Gd"d#d#e j(Z2e/e1e0d$Z3Gd%d&d&e j(Z4Gd'd(d(e j(Z5Gd)d*d*e j(Z6Gd+d,d,e j(Z7eGd-d.d.eZ8  dKd/ee9e9fd0e:d1e9d2eej;d3e9d4ejeGd7d8d8e8Z?d9Z@ed:d;Gdd;Gd?d@d@e8ZBeGdAdBdBe8ZCGdCdDdDe j(ZDGdEdFdFe j(ZEedGd;GdHdIdIe8ZFgdJZGdS)LN)OptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)!flash_attn_supports_top_left_maskis_flash_attn_available)BaseModelOutputCausalLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)PreTrainedModel)auto_docstringis_peft_availablelogging)Data2VecAudioConfig)_flash_attention_forwardcs&eZdZdfdd ZddZZS)Data2VecAudioConvLayerrcs|t|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ tj |jdd|_ t |j|_dS)Nrr) kernel_sizestridebiasTZelementwise_affine)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_strideZ conv_biasconv LayerNorm layer_normrfeat_extract_activation activationselfconfiglayer_id __class__c/var/www/auris/lib/python3.10/site-packages/transformers/models/data2vec/modeling_data2vec_audio.pyr )s  zData2VecAudioConvLayer.__init__cCs:||}|dd}||}|dd}||}|S)N)r' transposer)r+r- hidden_statesr2r2r3forward8s     zData2VecAudioConvLayer.forwardr__name__ __module__ __qualname__r r9 __classcell__r2r2r0r3r(src$eZdZfddZddZZS)Data2VecAudioPadLayercs*t|ddkrd|_dSd|_dS)Nrr)rr num_pad_remove)r-num_conv_pos_embeddingsr0r2r3r Ds  zData2VecAudioPadLayer.__init__cCs,|jdkr|ddddd|j f}|SNr)rCr7r2r2r3r9Hs zData2VecAudioPadLayer.forwardr;r2r2r0r3rACs rAcr@) Data2VecAudioPositionalConvLayercs\ttj|j|j|j|jd|jd|_t|j|_ t |j |_ tj |jdd|_dS)NrB)rpaddinggroupsFr)rr rr$ hidden_sizeZconv_pos_kernel_sizeZnum_conv_pos_embedding_groupsr'rArGrr*r+r(r)r-r.r0r2r3r Os   z)Data2VecAudioPositionalConvLayer.__init__cCsD||}||}|dd}||}|dd}||}|SNrrB)r'rGr6r)r+r7r2r2r3r9^s      z(Data2VecAudioPositionalConvLayer.forwardr;r2r2r0r3rFNs rFcr@)$Data2VecAudioPositionalConvEmbeddingcs.ttfddtjD|_dS)Ncg|]}tqSr2)rF.0_r.r2r3 mzAData2VecAudioPositionalConvEmbedding.__init__..)rr r ModuleListrangerDlayersrJr0rQr3r js  z-Data2VecAudioPositionalConvEmbedding.__init__cCs0|dd}|jD]}||}q |dd}|SrK)r6rV)r-r8layerr2r2r3r9ps    z,Data2VecAudioPositionalConvEmbedding.forwardr;r2r2r0r3rLi rLcs0eZdZdZfddZddZddZZS)Data2VecAudioFeatureEncoderz.Construct the features from raw audio waveformcs:ttfddtjD|_d|_d|_dS)Ncsg|]}t|dqS))r/)rrOirQr2r3rR~sz8Data2VecAudioFeatureEncoder.__init__..FT) rr rrTrUZnum_feat_extract_layers conv_layersgradient_checkpointing_requires_gradrJr0rQr3r {s  z$Data2VecAudioFeatureEncoder.__init__cCs|D]}d|_qd|_dSNF) parameters requires_gradr^r-paramr2r2r3_freeze_parameterss  z.Data2VecAudioFeatureEncoder._freeze_parameterscCs\|dddf}|jr|jrd|_|jD]}|jr'|jr'|jr'||j|}q||}q|S)NT)r^trainingrar\r]_gradient_checkpointing_func__call__)r- input_valuesr8Z conv_layerr2r2r3r9s   z#Data2VecAudioFeatureEncoder.forward)r<r=r>__doc__r rdr9r?r2r2r0r3rYxs  rYcr@)Data2VecAudioFeatureProjectioncsJttj|jd|jd|_t|jd|j|_ t |j |_ dS)Nr5Zeps) rr rr(r!layer_norm_epsr)LinearrI projectionDropoutZfeat_proj_dropoutdropoutrJr0r2r3r s z'Data2VecAudioFeatureProjection.__init__cCs&||}||}||}||fSN)r)rnrp)r-r8Znorm_hidden_statesr2r2r3r9s   z&Data2VecAudioFeatureProjection.forwardr;r2r2r0r3rjrXrjcseZdZdZ     ddededed ed ed ed eeffd d Z de j dedefddZ     dde j dee j dee e j dee j dee j dede e j ee j ee e j ffddZZS)Data2VecAudioAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsrp is_decoderr is_causalr.cst||_||_||_|||_||_|j||jkr*td|jd|d|jd|_||_ ||_ t j |||d|_ t j |||d|_t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g)r)rr rtrurphead_dimr. ValueErrorscalingrvrwrrmk_projv_projq_projout_proj)r-rtrurprvrrwr.r0r2r3r s&    zData2VecAudioAttention.__init__tensorseq_lenbszcCs ||||j|jddSrK)viewrurxr6 contiguousr-rrrr2r2r3_shapes zData2VecAudioAttention._shaper8key_value_statespast_key_valueattention_masklayer_head_maskoutput_attentionsreturncCsr|du}|\}} } |||j} |r.|dur.|djd|jdkr.|d} |d} nZ|rE|||d|} |||d|} nC|durt|||d|} |||d|} tj|d| gdd} tj|d| gdd} n|||d|} |||d|} |j r| | f}||j d|j f}|| | |j |} | j |} | j |} | d}t| | dd}|||j | |fkrtd||j | |fd||dur ||d| |fkrtd |d| |fd|| ||j | ||}| ||j | |}tjj|dd}|durL||j fkr1td |j fd|| dddd| ||j | |}| ||j | |}|rc| ||j | |}| ||j | |}nd}tjj||j|jd }t|| }|||j | |j fkrtd ||j | |j fd|| ||j | |j }|dd}| || |j}||}|||fS) #Input shape: Batch x Time x ChannelNrrBrr5dimz$Attention weights should be of size , but is z!Attention mask should be of size z/Head mask for a single layer should be of size )pre `attn_output` should be of size )sizer}rzshaperr{r|torchcatrvrurxrreshapeZbmmr6ryr functionalsoftmaxrprertr~)r-r8rrrrris_cross_attentionrtgt_lenrP query_states key_states value_statesZ proj_shapeZsrc_len attn_weightsZattn_weights_reshapedZ attn_probs attn_outputr2r2r3r9s        "    zData2VecAudioAttention.forward)rsFTFNNNNNF)r<r=r>riintfloatboolrrr rTensorrrr9r?r2r2r0r3rrsV rrcseZdZdZfddZdejdedefddZ dd ejd e ejd e e ejde ejde ejde de eje eje e ejffddZ Z S)Data2VecAudioFlashAttention2aV Data2VecAudio flash attention module. This module inherits from `Data2VecAudioAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cstj|i|t|_dSrq)rr r _flash_attn_uses_top_left_mask)r-argskwargsr0r2r3r Os z%Data2VecAudioFlashAttention2.__init__rrrcCs||||j|jSrq)rrurxrr2r2r3_reshapeWsz%Data2VecAudioFlashAttention2._reshapeNFr8rrrrrrc Cs>|du}|\}} } |||d|} |r8|dur8|djd|jdkr8|ddd} |ddd} nb|rO|||d|} |||d|} nK|dur|||d|} |||d|} tj|ddd| gdd} tj|ddd| gdd} n|||d|} |||d|} |j r| dd| ddf}| jd}|dur||djd7}| j }|tj krt rt }nt|jdr|jj}n|jjj }td|d | |} | |} | |} t| | | || |jr|jnd |j|jd }||| d}||}|sd}|||fS) Nr5rrBrrr4_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .rs)rprwZuse_top_left_mask)rrr}rr6r{r|rrrvdtypefloat32Zis_autocast_enabledZget_autocast_gpu_dtypehasattrr.rweightlogger warning_oncetorrerprwrrr~)r-r8rrrrrrrZq_lenrPrrrZ kv_seq_lenZ input_dtypeZ target_dtyperrr2r2r3r9Zsl             z$Data2VecAudioFlashAttention2.forwardr)r<r=r>rir rrrrrrrr9r?r2r2r0r3rHs0  rcseZdZ     d dejdeejdeeejdeejdeejded eejeejeeejfffd d ZZ S) Data2VecAudioSdpaAttentionNFr8rrrrrrcs|rtdtj|||||dS|du}|\}} } ||} |r=|dur=|djd|jdkr=|d} |d} nZ|rT|||d|} || |d|} nC|dur|||d|} || |d|} t j |d| gdd} t j |d| gdd} n|||d|} || |d|} |j r| | f}|| | |} |j r|dur| dkrd nd }t jjj| | | ||jr|jnd |d }|||j| |jfkrtd ||j| |jfd||dd}||| |j}||}|d|fS)raData2VecAudioModel is using Data2VecAudioSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` . Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rrrrNrrBrr5rTFrs)Z attn_maskZ dropout_prwrr)rrrr9rr}rrr{r|rrrvrwrrZscaled_dot_product_attentionrerprurxryr6rrtr~)r-r8rrrrrrrrrPrrrrwrr0r2r3r9sh        z"Data2VecAudioSdpaAttention.forwardr) r<r=r>rrrrrr9r?r2r2r0r3rs* rcr@)Data2VecAudioFeedForwardcsptt|j|_t|j|j|_ t |j t r"t |j |_n|j |_t|j|j|_t|j|_dSrq)rr rroZactivation_dropoutintermediate_dropoutrmrIZintermediate_sizeintermediate_dense isinstanceZ hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrJr0r2r3r ,s  z!Data2VecAudioFeedForward.__init__cCs6||}||}||}||}||}|Srq)rrrrrr7r2r2r3r99s     z Data2VecAudioFeedForward.forwardr;r2r2r0r3r+  r)eagerZsdpaflash_attention_2cs&eZdZfddZdddZZS)Data2VecAudioEncoderLayercsltt|j|j|j|jdd|_t |j |_ tj |j|j d|_t||_tj |j|j d|_dS)NF)rtrurprvrk)rr DATA2VEC_AUDIO_ATTENTION_CLASSES_attn_implementationrIZnum_attention_headsZattention_dropout attentionrrorrpr(rlr)r feed_forwardfinal_layer_normrJr0r2r3r Ks  z"Data2VecAudioEncoderLayer.__init__NFcCsf|}|j|||d\}}}||}||}||}|||}||}|f}|r1||f7}|S)Nrr)rrpr)rr)r-r8rrZ attn_residualrrPoutputsr2r2r3r9Ys    z!Data2VecAudioEncoderLayer.forwardr_r;r2r2r0r3rJs rc sLeZdZfddZ    d dejdeejded ed ef d d Z Z S)Data2VecAudioEncodercsrt|_t|_tjjjd|_ t j |_ t fddtjD|_d|_jdk|_dS)NrkcrMr2)rrNrQr2r3rRtrSz1Data2VecAudioEncoder.__init__..Fr)rr r.rLpos_conv_embedrr(rIrlr)rorrprTrUnum_hidden_layersrVr]r_use_flash_attention_2rJr0rQr3r ns   zData2VecAudioEncoder.__init__NFTr8rroutput_hidden_states return_dictcCs|rdnd}|r dnd}|dur_|ddd|jd}d||<|jr2|dur/d|vr/|nd}n-d|ddddddfj|jd}|t|jj}| |jdd|jd|jd}| |} || }| |}| |}t pxt|} |jD]G} |r||f}tg} |jr| |jjkrdnd } | r| r|jr|jr|| j|||}n| |||d }|d}| rd }|r||df}q||r||f}|std d |||fDSt|||dS)Nr2r5rrBr?rTFrNNcss|] }|dur|VqdSrqr2)rOvr2r2r3 sz/Data2VecAudioEncoder.forward..)last_hidden_stater8 attentions) unsqueezerepeatrrrrrZfinfominexpandrr)rpr r rVrandrer. layerdropr]rfrgtupler )r-r8rrrrZall_hidden_statesZall_self_attentionsZexpand_attention_maskZposition_embeddingsZ synced_gpusrWZdropout_probabilityZskip_the_layerZ layer_outputsr2r2r3r9xs`   &        zData2VecAudioEncoder.forward)NFFT) r<r=r>r rrrrrr9r?r2r2r0r3rms"  rcr@)Data2VecAudioAdapterLayercs0ttj|jd|j|j|jdd|_dS)NrBr)rrG)rr rr$output_hidden_sizeZadapter_kernel_sizeadapter_strider'rJr0r2r3r s  z"Data2VecAudioAdapterLayer.__init__cCs||}tjj|dd}|S)Nrr)r'rrZglur7r2r2r3r9s z!Data2VecAudioAdapterLayer.forwardr;r2r2r0r3rs  rcr@)Data2VecAudioAdaptercsptjjkrtjj|_tj|_nd|_|_t fddt j D|_ j |_ dS)Nc3s|]}tVqdSrq)rrNrQr2r3rsz0Data2VecAudioAdapter.__init__..)rr rrIrrmprojr(proj_layer_normrTrUnum_adapter_layersrVrrJr0rQr3r s     zData2VecAudioAdapter.__init__cCsr|jdur|jdur||}||}|dd}|jD]}tj}|jr,||jkr0||}q|dd}|SrK)rrr6rVnprandomrer)r-r8rWZlayerdrop_probr2r2r3r9s      zData2VecAudioAdapter.forwardr;r2r2r0r3rrrc@sheZdZeZdZdZdZdZdZ ddZ dde e j efdeefd d Z dd ed e j fd dZdS)Data2VecAudioPreTrainedModeldata2vec_audiorhTcCsZt|tr(td|jj}tjj|jj | |dtjj|jj | |ddSt|t r8tj |j j ddSt|tjrX|j jjd|jjd|j durV|j jdSdSt|tjtjfr||j durl|j j|j durz|j jddSdSt|tjrtj|j |j durt|j|j|jd}tjj|j | |ddSdSdS)zInitialize the weightsr)abrrs)meanstdNr)rrjmathsqrtrnZ in_featuresrinituniform_rrrFZ constant_r'rmdataZnormal_r.Zinitializer_rangeZzero_r(Z GroupNormZfill_r$Zkaiming_normal_rHZ in_channelsr)r-modulekr2r2r3 _init_weightss0         z*Data2VecAudioPreTrainedModel._init_weightsN input_lengths add_adaptercCsn|dur|jjn|}dd}t|jj|jjD] \}}||||}q|r5t|jjD] }||d|jj}q*|S)zH Computes the output length of the convolutional layers NcSstj|||dddS)Nfloor)Z rounding_moder)rdiv input_lengthrrr2r2r3_conv_out_lengthszWData2VecAudioPreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengthr)r.rzipr%r&rUrr)r-rrrrrrPr2r2r3 _get_feat_extract_output_lengthssz=Data2VecAudioPreTrainedModel._get_feat_extract_output_lengthsfeature_vector_lengthrcCs|jdddddf}|j||d}|tj}|jd}tj||f|j|jd}d|tj |jd|jd|df<| dgd dg }|S)Nr5rrr)rdevicer)r) Zcumsumrrrlongrzerosrrarangeflipr)r-rrrZnon_padded_lengthsZoutput_lengths batch_sizer2r2r3"_get_feature_vector_attention_mask,s  "z?Data2VecAudioPreTrainedModel._get_feature_vector_attention_maskrq)r<r=r>rZ config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingZ_supports_flash_attn_2Z_supports_sdparrr LongTensorrrrrrr2r2r2r3rs(  rr mask_prob mask_lengthr min_masksrcs|\}dkr tdkrtdddtjdfdd}|dur:|d n fd d t|D}tj |ft d }g} |} | d krZ|S|D];} || } tjj t | d| dd} t | d kr}d}n| d }t| tj| | tjd |g} | | q\t| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkrd| | dk<t|| dd |S)af Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://arxiv.org/abs/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr}|d|kr*t|dd}|S)z;Given input length, compute how many spans should be maskedrr)rmax)rnum_masked_spanepsilonrrrsequence_lengthr2r3compute_num_masked_spangs  z6_compute_mask_indices..compute_num_masked_spanNr5csg|]}qSr2r2rN)rr2r3rRzsz)_compute_mask_indices..rrF)replace)ryrrritemdetachsumtolistrUrrchoicerlenZ concatenateonesZint32appendarrayZ broadcast_torr Zput_along_axis)rrrrrrrrZ spec_aug_maskZspec_aug_mask_idxsZmax_num_masked_spanrr Zspec_aug_mask_idxZdummy_mask_idxoffsetsr2r r3_compute_mask_indicesAs\    rcseZdZdeffdd ZddZ  ddejdeejd eej fd d Z e     dd eej d eej deejd ee dee dee deeeffddZZS)Data2VecAudioModelr.cs|t|||_t||_t||_|jdks|jdkr)t t |j |_t||_|jr5t|nd|_|dS)Nrs)rr r.rYfeature_extractorrjfeature_projectionmask_time_probmask_feature_probr ParameterrrrIrmasked_spec_embedrencoderrradapter post_initrJr0r2r3r s     zData2VecAudioModel.__init__cCs|jdS Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)rrdr-r2r2r3freeze_feature_encodersz)Data2VecAudioModel.freeze_feature_encoderNr8mask_time_indicesrcCst|jdds |S|\}}}|dur|j|j||<n-|jjdkrK|jrKt||f|jj|jj ||jj d}t j ||j t jd}|j|j||<|jjdkr|jrt||f|jj|jj|jjd}t j ||j t jd}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://arxiv.org/abs/1904.08779). Zapply_spec_augmentTNr)rrrr)rr)rrrr5)getattrr.rr"rrrrerZmask_time_lengthZmask_time_min_masksrrrrr Zmask_feature_lengthZmask_feature_min_masksr)r-r8r*rrrrIZmask_feature_indicesr2r2r3_mask_hidden_statess4 z&Data2VecAudioModel._mask_hidden_statesrhrrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}||}|dd}|dur8|j|jd|dd}||\}}|j |||d}|j |||||d} | d}|j dur_| |}|sk||f| ddSt ||| j | jd S) a/ mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. NrrBFr)r*rrrrrr)rextract_featuresr8r)r.rruse_return_dictrr6rrrr,r#r$Data2VecAudioBaseModelOutputr8r) r-rhrr*rrrr.r8Zencoder_outputsr2r2r3r9s@     zData2VecAudioModel.forwardrNNNNN)r<r=r>rr r)rZ FloatTensorrrr,rrrrrr0r9r?r2r2r0r3rsB  . rrBzu Data2VecAudio Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). )Z custom_introcseZdZfddZddZddZe     ddeej d eej d ee d ee d ee d eej de e e ffddZZS)Data2VecAudioForCTCcsxt|t||_t|j|_|jdur t d|j dt |dr+|j r+|j n|j}t||j|_|dS)a7 target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`Data2VecAudioForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `Data2VecAudioForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.r)rr rrrroZ final_dropoutrp vocab_sizeryr1rrrrIrmlm_headr%)r-r.rr0r2r3r Gs     zData2VecAudioForCTC.__init__cCtdt|dSr'The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.Nwarningswarn FutureWarningr)r(r2r2r3freeze_feature_extractorb  z,Data2VecAudioForCTC.freeze_feature_extractorcC|jjdSr&rrrdr(r2r2r3r)nz*Data2VecAudioForCTC.freeze_feature_encoderNrhrrrrlabelsrc Cs||dur|n|jj}|dur||jjkrtd|jj|j|||||d}|d}||}||} d} |dur|durC|ntj |tj d}| | d tj } |dk} | d} || }tjj| dtjddd}tjjjd d tjj||| | |jj|jj|jjd } Wdn1swY|s| f|td}| dur| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: r-rrr5)rrrF)enabled)blankZ reductionZ zero_infinitylosslogitsr8r)r.r/r r3ryrrpr4rZ ones_likerrrrZ masked_selectrrZ log_softmaxrr6backendsZcudnnflagsZctc_lossZ pad_token_idZctc_loss_reductionZctc_zero_infinity_HIDDEN_STATES_START_POSITIONrr8r)r-rhrrrrrArr8rFrErZ labels_maskZtarget_lengthsZflattened_targetsZ log_probsoutputr2r2r3r9usN      zData2VecAudioForCTC.forwardr1)r<r=r>r r<r)rrrrrrrrr9r?r2r2r0r3r2As2   r2z Data2VecAudio Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. cseZdZfddZddZddZddZe dd ee j d ee j d ee d ee dee dee j de e effddZZS)&Data2VecAudioForSequenceClassificationcst|t|dr|jrtdt||_|jd}|jr*t t |||_ t |j|j|_t |j|j|_|dS)NrzdSequence classification does not support the use of Data2VecAudio adapters (config.add_adapter=True)r)rr rrryrrruse_weighted_layer_sumrr!rr layer_weightsrmrIZclassifier_proj_size projector num_labels classifierr%r-r. num_layersr0r2r3r s    z/Data2VecAudioForSequenceClassification.__init__cCr5)z Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. r7Nr8r(r2r2r3r<r=z?Data2VecAudioForSequenceClassification.freeze_feature_extractorcCr>r&r?r(r2r2r3r)r@z=Data2VecAudioForSequenceClassification.freeze_feature_encodercC|jD]}d|_qdSz Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated. FNrr`rarbr2r2r3freeze_base_modelz8Data2VecAudioForSequenceClassification.freeze_base_modelNrhrrrrrArcCsz|dur|n|jj}|jjrdn|}|j|||||d}|jjrB|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |}|durV|jdd} n+||jd|} | ddd|jd} d || <|j dd| j dd dd} || } d}|durt}|| d|jj| d}|s| f|td}|dur|f|S|St|| |j|jd S)  input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`Data2VecAudioProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTr-rrr5rrBrsrD)r.r/rLrrIrstackrrrrMrrrNrrrrrrPrrOrr8r)r-rhrrrrrArr8 norm_weightsZ pooled_outputZ padding_maskZexpand_padding_maskrFrEloss_fctrJr2r2r3r9sH    z.Data2VecAudioForSequenceClassification.forwardr1)r<r=r>r r<r)rVrrrrrrrrr9r?r2r2r0r3rKs4   rKcseZdZfddZddZddZddZe dd ee j d ee j d ee j d ee dee dee de e effddZZS)(Data2VecAudioForAudioFrameClassificationcszt|t|dr|jrtdt||_|jd}|jr*t t |||_ t |j|j|_|j|_|dS)NrzgAudio frame classification does not support the use of Data2VecAudio adapters (config.add_adapter=True)r)rr rrryrrrrLrr!rrrMrmrIrOrP init_weightsrQr0r2r3r 7s    z1Data2VecAudioForAudioFrameClassification.__init__cCr5r6r8r(r2r2r3r<Gr=zAData2VecAudioForAudioFrameClassification.freeze_feature_extractorcCr>r&r?r(r2r2r3r)Sr@z?Data2VecAudioForAudioFrameClassification.freeze_feature_encodercCrSrTrUrbr2r2r3rVZrWz:Data2VecAudioForAudioFrameClassification.freeze_base_modelNrhrrArrrrcCs|dur|n|jj}|jjrdn|}|j|||||d}|jjrB|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |} d} |durht} | | d|jtj| d|jdd} |su| f|td} | St| | |j|jd S) rXNTr-rrr5r)ZaxisrD)r.r/rLrrIrrYrrrrMrrrPrrOZargmaxrr8r)r-rhrrArrrrr8rZrFrEr[rJr2r2r3r9bs: (z0Data2VecAudioForAudioFrameClassification.forwardr1)r<r=r>r r<r)rVrrrrrrrrr9r?r2r2r0r3r\5s4   r\cs&eZdZdfdd ZddZZS) AMSoftmaxLoss>@皙?csFtt|||_||_||_tjt ||dd|_ t |_ dS)NT)ra) rr^r scalemarginrOrr!rZrandnrrrE)r-Z input_dimrOrarbr0r2r3r s zAMSoftmaxLoss.__init__c Csx|}tjj|jdd}tjj|dd}t||}||j}tj||j }|j t | ||}| ||}|S)Nrrr)flattenrr normalizerrmmrbZone_hotrOrawhererrE) r-r8rArZ cos_thetapsiZonehotrFrEr2r2r3r9s   zAMSoftmaxLoss.forward)r_r`r;r2r2r0r3r^sr^cs4eZdZdfdd ZdejdejfddZZS) TDNNLayerrcsvt|dkr|j|dn|j||_|j||_|j||_|j||_t |j|j|j|_ t |_ dS)Nrr)rr tdnn_dimr"r# tdnn_kernelrZ tdnn_dilationdilationrrmkernelZReLUr+r,r0r2r3r s "   zTDNNLayer.__init__r8rcCstr ddlm}trt|j|rtd|dd}|jj |j |j |j dd}t jj|||jj|jd}|dd}||}|S)Nr) LoraLayerzDetected LoRA on TDNNLayer. LoRA weights won't be applied due to optimization. You should exclude TDNNLayer from LoRA's target modules.rrB)rk)rZpeft.tuners.lorarmrrlr9r:r6rrr#rr"rrZconv1drrkr+)r-r8rmrr2r2r3r9s      zTDNNLayer.forwardr:)r<r=r>r rrr9r?r2r2r0r3rhs rhzq Data2VecAudio Model with an XVector feature extraction head on top for tasks like Speaker Verification. cseZdZfddZddZddZddZd eej e ffd d Z e dd e ejde ejde ede ede ede ejdeeeffddZZS)Data2VecAudioForXVectorcstt|_jd}jrtt |||_ t j j d|_fddttj D}t||_t j ddj|_t jj|_tjj|_|dS)Nrrcsg|]}t|qSr2)rhrZrQr2r3rRsz4Data2VecAudioForXVector.__init__..r5rB)rr rrrrLrr!rrrMrmrIrirNrUrrTtdnnZxvector_output_dimrrPr^rO objectiver])r-r.rRZ tdnn_layersr0rQr3r s     z Data2VecAudioForXVector.__init__cCr5r6r8r(r2r2r3r<r=z0Data2VecAudioForXVector.freeze_feature_extractorcCr>r&r?r(r2r2r3r)r@z.Data2VecAudioForXVector.freeze_feature_encodercCrSrTrUrbr2r2r3rVrWz)Data2VecAudioForXVector.freeze_base_modelrcCs&dd}|jjD]}|||d}q|S)z? Computes the output length of the TDNN layers cSs|||dS)Nrr2rr2r2r3rszJData2VecAudioForXVector._get_tdnn_output_lengths.._conv_out_lengthr)r.rj)r-rrrr2r2r3_get_tdnn_output_lengths s z0Data2VecAudioForXVector._get_tdnn_output_lengthsNrhrrrrrArcCs|dur|n|jj}|jjrdn|}|j|||||d}|jjrB|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |}|jD]} | |}qN|durf|jdd} |jdd} nC||j dd} || }g} g} t|D]"\}}| ||d|fjdd| ||d|fjddq|t| } t| } tj| | gdd}||}||}d}|dur|||}|s||f|td}|dur|f|S|St||||j|jdS) rXNTr-rrr5r)rErFZ embeddingsr8r)r.r/rLrrIrrYrrrrMrrrNrorrrrq enumeraterrrrPrprr8r)r-rhrrrrrArr8rZZ tdnn_layerZ mean_featuresZ std_featuresZfeat_extract_output_lengthsZtdnn_output_lengthsr[lengthZstatistic_poolingZoutput_embeddingsrFrErJr2r2r3r9s\           zData2VecAudioForXVector.forwardr1)r<r=r>r r<r)rVrrrrrqrrrrrrr9r?r2r2r0r3rns6   rn)r\r2rKrnrrrE)Hrr9typingrrrnumpyrrrZtorch.nnrZ activationsrZintegrations.deepspeedr Zintegrations.fsdpr Zmodeling_flash_attention_utilsr r Zmodeling_outputsr rrrrrZmodeling_utilsrutilsrrrZconfiguration_data2vec_audiorrZ get_loggerr<rModulerrArFrLrYrjrrrrrrrrrrrrrrZndarrayrr0rrIr2rKr\r^rhrn__all__r2r2r2r3s           #zi#UQ  wwrh