fThBSrSSKrSSKJrJrJrJr SSKrSSK r SSK r SSK J r SSK J r JrJr SSKJr SSKJr SS KJr SS KJr SS KJrJr SS KJrJrJrJrJ r SS K!J"r" SSK#J$r$J%r% SSK&J'r'J(r( \%RR"\*5r+Sr,S\ RZS\.S\.4Sjr/S{S\ RZS\.S\\ RZ4Sjjr0S|S\\.\.4S\1S\.S\\ RdS\.S\Rf4 Sjjr4"SS \ Rj5r6"S!S"\ Rj5r7"S#S$\ Rj5r8"S%S&\ Rj5r9"S'S(\ Rj5r:"S)S*\ Rj5r;"S+S,\ RRj5r<"S-S.\ Rj5r="S/S0\ Rj5r>"S1S2\ Rj5r?"S3S4\ Rj5r@"S5S6\ Rj5rA"S7S8\ Rj5rB"S9S:\ Rj5rC"S;S<\ Rj5rD"S=S>\ Rj5rE"S?S@\ Rj5rF"SASB\ Rj5rG"SCSD\ Rj5rH"SESF\ Rj5rI"SGSH\ Rj5rJ\$"SISJ\"55rK"SKSL\K5rL"SMSN\K5rM"SOSP\K5rN"SQSR\K5rO"SSST\K5rP"SUSV\K5rQ"SWSX\K5rR"SYSZ\K5rS"S[S\\ Rj5rT"S]S^\ Rj5rU\$"S_S`9"SaSb\K55rV\$"ScS`9"SdSe\K\55rWS}Sf\KS\ RSg\\ RS\\ RdSh\1Si\1Sj\1Sk\\ RjSl\YSm\YS\\ R\\ R\ R444SnjjrZ\$"SoS`9"SpSq\K55r[\$"SrS`9"SsSt\K55r\"SuSv\ Rj5r]\$"SwS`9"SxSy\"55r^/SzQr_g)~zPyTorch SpeechT5 model.N)ListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossL1Loss)ACT2FN)GenerationMixin)is_deepspeed_zero3_enabled)is_fsdp_managed_module)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutputSeq2SeqSpectrogramOutput)PreTrainedModel)auto_docstringlogging)SpeechT5ConfigSpeechT5HifiGanConfig input_ids pad_token_iddecoder_start_token_idcURUR5nUSS2SS24R5USS2SS24'X#SS2S4'Uc [S5eUR US:HU5 U$)z) Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)rrrshifted_input_idss f/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/speecht5/modeling_speecht5.pyshift_tokens_rightr)4sz"++IOO<(CRC0668ae4adLMM""#4#AB%%&:f&DcJ  //r*r# mask_prob mask_length min_masksreturnc ^^^^^UunmTS:a [S5eTT:a[STSTS35e[RRS5R 5mUUUUU4SjnUb-UR 5R S5R5O[U5Vs/sHnTPM snn[R"UT4[S 9n /n U"T5n U S :XaU $UHn U"U 5n [RR[R"U TS- - 5U S S 9n[U5S :XaTS- nOUS n[R"U[R"X- [R S 9U-/5nU R#U5 M [R$"U 5n [R&"U SS2SS2S4X[T45n U R)X[T-5n [R"T5SSSS24n[R&"UX[T45R)X[T-5nU U-n U R+5TS- :a TS- XTS- :'[R,"XSS5 U $s snf) a* Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://arxiv.org/abs/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `c>[TU-T- T-5n[UT5nUT-T:aTT-nUTS- - U:a[UTS- - S5nU$)z;Given input length, compute how many spans should be maskedrr)intmax) input_lengthnum_masked_spanepsilonr4r3r5sequence_lengths r(compute_num_masked_span6_compute_mask_indices..compute_num_masked_spansoi,6DwNOoy9 [ (? :-r?s `` ` @@r(_compute_mask_indicesrcZs0#(JQABB_$]^i]j&&7q :  iinnQ$$&G  $  % ##B'..0',Z'8 9'8!o'8 9HHj/:$GM1/Ba% 1,?II,, IIlkAo6 7RW-   !Q &-q0N.q1NNN (;(MUWU]U] ^ao o p  !!"34/&2"451a:&+(V,33JVa@abii $T4]3Goog 'UV^^+5G,g5/A"55GVYZGZ!0CCDmB? w:s( I0c2^\rSrSrSU4SjjrSrSrU=r$)SpeechT5NoLayerNormConvLayercb>[TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [URUl g)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__s r(rm%SpeechT5NoLayerNormConvLayer.__init__s [TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [ R"URSS9Ul [URUlg)NrrrhT)elementwise_affine)rlrmrnrorprrqrrrsrtru LayerNorm layer_normr rvrwrxs r(rm#SpeechT5LayerNormConvLayer.__init__s [TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [URUl [ R"URURSS9Ulg)NrrrhT) num_groups num_channelsaffine)rlrmrnrorprrqrrrsrtrur rvrw GroupNormrrxs r(rm#SpeechT5GroupNormConvLayer.__init__s [TU]5 SUlX lX0lUR XR-X#5 gN)rlrmoffsetrr make_weights)ryrrrr|s r(rm.SpeechT5SinusoidalPositionalEmbedding.__init__ s8  *& -++5}Rr*num_embeddingscURXU5n[US5(a8URURRURR S9nUR SUSS9 g)NweightsrCdeviceF persistent) get_embeddinghasattrtorrCrregister_buffer)ryrrr emb_weightss r(r2SpeechT5SinusoidalPositionalEmbedding.make_weights's\(( T 4 # #%..t||/A/A$,,J]J].^K Y Fr*cUS-n[R"S5US- - n[R"[R"U[R S9R 5U*-5n[R"U[R S9R 5RS5URS5-n[R"[R"U5[R"U5/SS9RUS5nUS-S:Xa,[R"U[R"US5/SS9nUbSXBSS24'UR[R"55$) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". ri'rrBrdimr!N)mathlogtorchexprPint64float unsqueezecatsincosviewrMrget_default_dtype)rrrhalf_dimembs r(r3SpeechT5SinusoidalPositionalEmbedding.get_embedding/s !A%hhuoA.ii XU[[AGGISDPQll>=CCEOOPQRUXUbUbcdUeeii338a@EEnVXY 1  !))S%++na"@AqIC  ""#CQ vve--/00r*rpast_key_values_lengthcUR5up4URXRU5RUR5nURS-U-nX`R RS5:a3UR X`R-URUR5 UR RSURS55RX4S5R5$)Nrrr!) size"create_position_ids_from_input_idsrrrrrrr index_selectrrI)ryrrbszseq_len position_idsmax_poss r(r-SpeechT5SinusoidalPositionalEmbedding.forwardAs ~~' >>yJZJZ\rsvv    ""Q&0 \\&&q) )   g 3T5G5GIYIY Z||((L,=,=b,ABGGVXY``bbr*cURU5R5n[R"USS9R U5U-U-nUR 5U-$)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)ner:rcumsumtype_aslong)ryrrrmaskincremental_indicess r(rHSpeechT5SinusoidalPositionalEmbedding.create_position_ids_from_input_idsPsW||K(,,.$||Da8@@FI__cgg"'')K77r*)rrrrr)rrrr__doc__r:rrmr staticmethodrrno_gradTensorrrrrrs@r(rrsNScS#SHUXMSSG3GsGQYZ]Q^G1c1#1HUXM11" ]]_ c cs c cbc88478QYZ]Q^88r*rc.^\rSrSrU4SjrSrSrU=r$)SpeechT5PositionalConvEmbeddingibc>[TU]5 [R"URURUR UR S-UR S9Ul[RRn[[RRS5(a$[RRRn[5(Ga%SSK nURRURR SS9 U"URSSS9UlSSS5 [URS5(aUURRR R"nURRR R$nO,URR&nURR(nURR+X5 URR+X5 OU"URSSS9Ul[-UR 5Ul[0UR2Ulg!,(df  GN,=f) Nr)ripaddinggroups weight_normr) modifier_rankweight)namerparametrizations)rlrmrrq hidden_sizenum_conv_pos_embeddingsnum_conv_pos_embedding_groupsruutilsrrrr deepspeedzeroGatheredParametersr original0 original1weight_gweight_vregister_external_parameterSpeechT5SamePadLayerrr rvrw)ryrzrrrrr|s r(rm(SpeechT5PositionalConvEmbedding.__init__cs II      6622a777   hh** 288,,m < <((33??K % ' ' 224993C3CST2U' aH Vtyy"4559955<<FF9955<<FF99--99-- NN 6 6t F NN 6 6t F#DIIH!DDI+F,J,JK  !?!?@VUs I IcURSS5nURU5nURU5nURU5nURSS5nU$Nrr)rrurrwrs r(r'SpeechT5PositionalConvEmbedding.forwardsV%//15  -0  ]3  6 %//15 r*)rwrurrrs@r(rrbsABr*rc6^\rSrSrSrSU4SjjrSrSrU=r$) SpeechT5ScaledPositionalEncodingiuK Scaled positional encoding, see §3.2 in https://arxiv.org/abs/1809.08895 c >[R"X25n[R"SU5RS5n[R"[R"SUS[R S9R 5[R"S5U- *-5n[R"UR 5U-5USS2SSS24'[R"UR 5U-5USS2SSS24'URS5n[TU]15 URSUSS9 [R"US 9UlX l[RR%[R&"S 55Ulg) NrrrrBg@peFrp?)rrMrPrrrrrrrrrlrmrrDropoutdropoutr Parametertensoralpha)ryrrmax_lenrpositiondiv_termr|s r(rm)SpeechT5ScaledPositionalEncoding.__init__s" [[ &<<7+55a899ell1c1EKKHNNPUYU]U]^eUfilUlSmmoii 08 ;<1add7 ii 08 ;<1add7 \\!_  T2%8zzG, XX'' S(9: r*cXRURSS2SURS524--nURU5nU$)Nr)r rrr)ryrs r(r(SpeechT5ScaledPositionalEncoding.forwards@JJMchhqkM)9!:::ll3 r*)r rr)i) rrrrrrmrrrrs@r(rrs ;r*rc2^\rSrSrSU4SjjrSrSrU=r$)"SpeechT5RelativePositionalEncodingic>[TU]5 XlX l[R R SU-U5Ulgr)rlrmr max_lengthrr Embeddingpe_k)ryrrr|s r(rm+SpeechT5RelativePositionalEncoding.__init__s4 $HH&&q:~s; r*ctURSn[R"SU5RUR[R S9nUSS2S4USSS24- nUR *X3UR *:'UR S- X3UR :'X0R -nURU5$)NrrrrC)r#rrPrrrrr)ryrrpos_seqs r(r*SpeechT5RelativePositionalEncoding.forwards%%a(,,q'*--]5I5IQVQ[Q[-\!T'"WT1W%55/3.>4??**+.2oo.A4??*+OO+yy!!r*)rrr)irrs@r(rrs< " "r*rc.^\rSrSrU4SjrSrSrU=r$)ricR>[TU]5 US-S:XaSUlgSUlg)Nrrr)rlrmnum_pad_remove)ryrr|s r(rmSpeechT5SamePadLayer.__init__s) #:Q#>!#Car*cXURS:aUSS2SS2SUR*24nU$Nrrrs r(rSpeechT5SamePadLayer.forwards6    ")!Q0F43F3F2F0F*FGMr*r"rrs@r(rrsKr*rc8^\rSrSrSrU4SjrSrSrSrU=r $)SpeechT5FeatureEncoderiz.Construct the features from raw audio waveformc >[TU]5 URS:Xa@[USS9/[ UR S- 5Vs/sHn[ XS-S9PM sn-nOVURS:Xa-[ UR 5Vs/sH n[XS9PM nnO[SURS35e[R"U5Ul SUl S Ul gs snfs snf) Ngroupr)r{rlayerz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT)rlrmfeat_extract_normrrLnum_feat_extract_layersrerr%r ModuleList conv_layersgradient_checkpointing_requires_grad)ryrzir,r|s r(rmSpeechT5FeatureEncoder.__init__s   # #w .5fqIJNSTZTrTruvTvNwNNw,V!eDNwNK % % 0HMfNlNlHmHm1*6>Hm K01I1I0JJst ==5&+#"Ns C C%cNUR5H nSUlM SUlgNF) parameters requires_gradr.)ryparams r(_freeze_parameters)SpeechT5FeatureEncoder._freeze_parameterss#__&E"'E '#r*cBUSS2S4nUR(aUR(aSUlURH\nUR(a@UR(a/UR(aUR UR U5nMTU"U5nM^ U$NT)r.trainingr4r,r-_gradient_checkpointing_func__call__)ryr+r conv_layers r(rSpeechT5FeatureEncoder.forwards$QW-    4==*.M '**J""t'B'Bt}} $ A A''!! !+= 9 +r*)r.r,r-) rrrrrrmr6rrrrs@r(r%r%s8#&$ r*r%c.^\rSrSrU4SjrSrSrU=r$)SpeechT5FeatureProjectionic4>[TU]5 [R"URSUR S9Ul[R"URSUR5Ul [R"UR5Ul g)Nr!eps) rlrmrrrnlayer_norm_epsrLinearr projectionrfeat_proj_dropoutrryrzr|s r(rm"SpeechT5FeatureProjection.__init__sf ,,vr':@U@UV))FOOB$79K9KLzz&":":; r*cnURU5nURU5nURU5nX4$r)rrFr)ryrnorm_hidden_statess r(r!SpeechT5FeatureProjection.forwards7!__];(:;  ]3 00r*)rrrFrrs@r(r@r@s< 11r*r@cL^\rSrSrU4SjrSrSS\RS\\RS\\R4Sjjr S\ S\R4S jr S \\R\ 44S jrSS \RS\\RS\\R4S jjrSrU=r$)SpeechT5SpeechEncoderPrenetic>[TU]5 Xl[U5Ul[ U5UlURS:dURS:aG[R"[R"UR5R55Ul[!U5Ul[%UR&UR(-S-URUR(5Ulg)Nr0r)rlrmrzr%feature_encoderr@feature_projectionmask_time_probmask_feature_probrrrrruniform_masked_spec_embedrpos_conv_embedrmax_speech_positionsrpos_sinusoidal_embedrHs r(rm$SpeechT5SpeechEncoderPrenet.__init__s  5f=";F"C  3 &&*B*BS*H%'\\%,,v?Q?Q2R2[2[2]%^D "=fE$I  ' '&*=*= = A      % !r*c8URR5 gr)rPr6rys r(freeze_feature_encoder2SpeechT5SpeechEncoderPrenet.freeze_feature_encoders //1r*r+r-mask_time_indicescURU5nURSS5nUbURURSU5nUR U5upTUR XSUS9nUR U5nXV-nUb URS5R5nO;[R"URSS[RURS9nURU5nXX-nXR4$)Nrr)r^r-r) rPr"_get_feature_vector_attention_maskr#rQ_mask_hidden_statesrVrrrrMrrX) ryr+r-r^extract_featuresrpositional_conv_embedding padding_mask positional_sinusoidal_embeddingss r(r#SpeechT5SpeechEncoderPrenet.forwards  // =+55a;  %!DD &&q)N +/*A*ABR*S' 00 ~1 %)$7$7 $F!%A  %),,Q/446L ;;}':':2A'>ejjYfYmYmnL+/+D+D\+R(%H ,,r*feature_vector_lengthcURSS9SS2S4nURU5R[R5nUR Sn[R "XQ4URURS9nSU[R"UR SURS9US- 4'URS/5RS5RS/5R5nU$)Nr!rrrrr) r _get_feat_extract_output_lengthsrrrr#rMrCrrPfliprN)ryrgr-non_padded_lengthsoutput_lengthsrZs r(r`>SpeechT5SpeechEncoderPrenet._get_feature_vector_attention_mask=s,22r2:1b5A>>?QRUUV[V`V`a#))!,   /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr*r\cSn[URRURR5Hup4U"XU5nM U$)z8 Computes the output length of the convolutional layers c8[R"X- USS9S-$)Nfloor) rounding_moder)rdiv)r<rirjs r(_conv_out_lengthVSpeechT5SpeechEncoderPrenet._get_feat_extract_output_lengths.._conv_out_lengthRs 99\7wWZ[[ [r*)ziprzrrrs)ryr\rtrirjs r(rj'> @W@W#X K,]PM$Yr*rc[URSS5(dU$UR5upEnUb(URR UR 5X'OURR S:aUR(a[XE4URR URRUURRS9n[R"X!R[RS9nURR UR 5X'URRS:aUR(a[XF4URRURR URR"S9n[R"XqR[RS9nUSS2S4R%SUS5nSX'U$) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://arxiv.org/abs/1904.08779). apply_spec_augmentTNr)r3r4r-r5r)r3r4r5r!)getattrrzrrUrrCrRr:rcmask_time_lengthmask_time_min_masksrr rrNrSmask_feature_lengthmask_feature_min_masksexpand)ryrr^r-rZr?rmask_feature_indicess r(ra/SpeechT5SpeechEncoderPrenet._mask_hidden_states]st{{$8$?? 4A3E3E3G0 [  (/3/E/E/H/HI\I\/]M , [[ ' '! + 5-++44 KK88-++99 ! !& ->G[G[chcmcm n /3/E/E/H/HI\I\/]M , ;; ( (1 ,#8)++77 KK;;++<< $ $)<<0DMaMainisis#t #74#@#G#GO]_#` 23M /r*)rzrPrQrUrVrXNN)rrrrrmr\rrr LongTensor FloatTensorrr:r`rrjrarrrs@r(rNrNs "2 6:9= -ll -!!1!12 -$E$5$56 -F  ]b]m]m  eEDTDTVYDY>Z &:>59 ,((,$E$5$56,!!1!12 ,,r*rNct^\rSrSrU4SjrSrSS\RS\\R4Sjjr Sr U=r $) SpeechT5SpeechDecoderPrenetic f>[TU]5 Xl[R"[ UR 5Vs/sH@n[R"US:Xa URO URUR5PMB sn5Ul [R"URUR5Ul [URURUR5Ul[R"UR"UR-UR5Ulgs snfr!)rlrmrzrr+rLspeech_decoder_prenet_layersrE num_mel_binsspeech_decoder_prenet_unitslayersr final_layerrpositional_dropoutrWencode_positionsspeaker_embedding_dimspeaker_embeds_layerryrzr/r|s r(rm$SpeechT5SpeechDecoderPrenet.__init__s  mm vBBC   DA  +,6F''v7Y7Y66D    99V%G%GI[I[\ @  % %     ' '!  %'IIf.J.JVM_M_._agasas$t! sAD.c[R"USUS9nURS5RUR S5SS5n[R "US:HUS5S-SU- - $)Nrrr)r bernoullirrepeatrwhere)ry inputs_embedsrr all_maskss r(_consistent_dropout/SpeechT5SpeechDecoderPrenet._consistent_dropoutsd}Q/15NN1%,,]-?-?-BAqI {{9>=![TU]5 US:Xa URnO URnX!RS- :Xa URnO URn[ R "UUURSURS- S-SS9Ul[ R"U5Ul X!RS- :a[ R"5Ul OSUl [ R"UR5Ulg)NrrrF)rirjrrk)rlrmrspeech_decoder_postnet_unitsspeech_decoder_postnet_layersrrqspeech_decoder_postnet_kernelru BatchNorm1d batch_normTanhrwrspeech_decoder_postnet_dropoutr)ryrzr{rorpr|s r(rm#SpeechT5BatchNormConvLayer.__init__s  q= --K ==K ;;a? ?!..L!>>LII  <<99A=!C   ..6 ::Q> > ggiDO"DOzz&"G"GH r*cURU5nURU5nURbURU5nURU5nU$r)rurrwrrs r(r"SpeechT5BatchNormConvLayer.forwardsJ -0  6 ?? & OOM:M ]3 r*)rwrrurrrrs@r(rrsI<r*rcl^\rSrSrU4SjrS\R 4SjrS\R 4SjrSr U=r $)SpeechT5SpeechDecoderPostnetic >[TU]5 Xl[R"UR UR UR-5Ul[R"UR UR5Ul [R"[UR5Vs/sHn[X5PM sn5Ulgs snfr)rlrmrzrrErrr,feat_outprob_outr+rLrrrrs r(rm%SpeechT5SpeechDecoderPostnet.__init__s   &"4"4f6I6IFLcLc6cd  &"4"4f6M6MN mm[TU]5 Xl[R"UR UR UR5Ul[URUR UR5Ul gr) rlrmrzrr vocab_sizerr embed_tokensrrmax_text_positionsrrHs r(rm"SpeechT5TextEncoderPrenet.__init__sc  LL):):F[TU]5 Xl[R"UR 5UlUR(a [R"UR5OSUl [R"URURUR5Ul[!UR"UR-S-URUR5Ulg)Nrr)rlrmrzrrrrscale_embeddingrsqrtr embed_scalerrrrrrembed_positionsrHs r(rm"SpeechT5TextDecoderPrenet.__init__s  zz&";";< [TU]5 Xl[R"UR UR SS9Ulg)NFrk)rlrmrzrrErrlm_headrHs r(rm#SpeechT5TextDecoderPostnet.__init__Ds3  yy!3!3V5F5FUS r*rc$URU5$rrrs r(r"SpeechT5TextDecoderPostnet.forwardIs||M**r*cUR$rrr[s r(get_output_embeddings0SpeechT5TextDecoderPostnet.get_output_embeddingsL ||r*cXlgrrrynew_embeddingss r(set_output_embeddings0SpeechT5TextDecoderPostnet.set_output_embeddingsOs% r*)rzr) rrrrrmrrrrrrrrs@r(rrCs(T +U\\+&&r*rc^\rSrSrSrSS\S\S\S\S\4 U4SjjjrS \ RS \S \4S jr SS \ RS\ \ RS\ \ \ RS\ \ RS\ \ RS\ \ RS\S\ \ R\ \ R\ \ \ R44SjjrSrU=r$)SpeechT5AttentioniSz Multi-headed attention from 'Attention Is All You Need' paper with relative position bias (see https://aclanthology.org/N18-2074.pdf) embed_dim num_headsr is_decoderrkc>[TU]5 XlX lX0lX-UlUR U-UR:wa[ SURSUS35eUR S-UlX@l[R"XUS9Ul [R"XUS9Ul [R"XUS9Ul [R"XUS9Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gr)rlrmrrrhead_dimr%scalingrrrEk_projv_projq_projout_proj)ryrrrrrkr|s r(rmSpeechT5Attention.__init__Ys "" !. MMI %$.. 8MdnnM]$YKr3 }}d* $ii 4@ ii 4@ ii 4@  )TB r*r rrcURX2URUR5RSS5R 5$r)rrrr contiguous)ryr rrs r(_shapeSpeechT5Attention._shapets5{{3GQQRSUVWbbddr*rkey_value_statespast_key_valuer-layer_head_mask position_biasoutput_attentionsr6cD USLnUR5upn URU5UR-n U(aUb USn USnGO U(aEURUR U5SU 5n URUR U5SU 5nOUbyURUR U5SU 5n URUR U5SU 5n[ R"USU /SS9n [ R"USU/SS9nODURUR U5SU 5n URUR U5SU 5nUR(aX4nXR-SUR4nURXU 5R"U6n U R"U6n UR"U6nU RS5n[ R"XRSS55nUR5XR-U U4:wa.[SXR-U U4SUR535eUbU R5RXR-SUR5RSS5n[ R "UURS S55nURSS5RXR-URS5URS55nUU- nUbzUR5U SU U4:wa#[S U SU U4SUR535eURXRU U5U-nURXR-U U5n["R$R'USS9nUbUR5UR4:wa*[S UR4SUR535eURSSSS5URXRU U5-nURXR-U U5nU(a=URXRU U5nURXR-U U5nOSn["R$R)UUR(UR*S 9n[ R"UU5nUR5XR-XR4:wa5[S XRXR4SUR535eURXRXR5nURSS5nUR-XUR.5nUR1U5nUUU4$)z#Input shape: Batch x Time x ChannelNrrr!rrz$Attention weights should be of size z , but is rz!Attention mask should be of size z/Head mask for a single layer should be of size )rr:z `attn_output` should be of size )rrrrrrrrrrrrbmmrr%rmatmulrrsoftmaxrr:rXrr)ryrrrr-rr r is_cross_attentionrtgt_lenr[ query_states key_states value_states proj_shapesrc_len attn_weights reshape_q rel_pos_biasattn_weights_reshaped attn_probs attn_outputs r(rSpeechT5Attention.forwardwsP.T9',,.a{{=1DLL@ ."<'*J)!,L T[[1A%BBLJ;;t{{3C'Db#NL  'T[[%?SIJ;;t{{='A2sKLN1$5z#BJJ 99nQ&7%FANLT[[%?SIJ;;t{{='A2sKL ??)7NNN*B > {{<#>CCZP __j1 #((*5 //!$yy/C/CAq/IJ    3#7'"J J6nn8LgW^7_6`a %%'(*   $$//166s^^7KRQUQ^Q^_iijkmnoI << =3J3J2r3RSL'11!Q7<<nn$m&8&8&;]=O=OPQ=RL L (L  %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S..'7SVddL',,S>>-A7GTL}},,\r,B  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVdVdfmov?wwL',,S>>-A7GTL  %1$5$5c>>7T[$\ !055cNN6JGU\]L$( !]]**<4<AAr*) rrrrrrrrrr)r0FT)NNNNNF)rrrrrr:rrNrmrrrrrrrrrs@r(rrSsS  CCC C  C  CC6eU\\eCece 488<152604"'yB||yB#5<<0yB!u||!45 yB !. yB "%,,/ yB  -yB yB u||Xell3XeELL>Q5RR SyByBr*rc.^\rSrSrU4SjrSrSrU=r$)SpeechT5FeedForwardic>[TU]5 [R"UR5Ul[R "URU5Ul[UR[5(a[URUl OURUl [R "X!R5Ul[R"UR5Ulgr)rlrmrractivation_dropoutintermediate_dropoutrErintermediate_dense isinstance hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropout)ryrzintermediate_sizer|s r(rmSpeechT5FeedForward.__init__s $&JJv/H/H$I!"$))F,>,>@Q"R f'' - -'-f.?.?'@D $'-'8'8D $II&79K9KL jj)>)>?r*cURU5nURU5nURU5nURU5nUR U5nU$r)r"r&r!r'r)rs r(rSpeechT5FeedForward.forwardsX// > 00? 11-@ ))-8 ++M: r*)r&r"r!r'r)rrs@r(rrs @r*rc ^\rSrSrS\4U4SjjrS S\RS\\RS\\RS\\RS\ 4 S jjr S r U=r $) SpeechT5EncoderLayeri rzc>[TU]5 [URURUR SS9Ul[R"UR5Ul [R"URURS9Ul [XR5Ul[R"URURS9Ulg)NFrrrrrB)rlrmrrencoder_attention_headsattention_dropout attentionrrr(rrrDrrencoder_ffn_dim feed_forwardfinal_layer_normrHs r(rmSpeechT5EncoderLayer.__init__ s *((44,,   zz&"7"78 ,,v'9'9v?T?TU/8N8NO " V-?-?VEZEZ [r*rr-rr r cUnURUUUUUS9upnURU5nXa-nURU5nXRU5-nUR U5nU4n U(aX4- n U $)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(config.encoder_attention_heads,)`. position_bias (`torch.FloatTensor`): relative position embeddings of size `(seq_len, seq_len, hidden_size // encoder_attention_heads)` output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rr-rr r )r4rrr6r7) ryrr-rr r residualrr[outputss r(rSpeechT5EncoderLayer.forwards.!)-')+'/ *8* & Q ]3  0  6 %(9(9-(HH --m<  "   &Gr*)r4rr6r7r)NNNF) rrrrrrmrrrrNrrrrs@r(r/r/ sx \~ \ 262604"' ,||,!.,"%,,/ ,   - ,  ,,r*r/c4^\rSrSrS\4U4SjjrSS\RS\\RS\\RS\\RS\\RS \\RS \\ \RS \\ S \\ 4S jjr Sr U=r $)SpeechT5DecoderLayeriHrzcp>[TU]5 [URURUR SS9Ul[R"UR5Ul [R"URURS9Ul [URURUR SS9Ul[R"URURS9Ul[!XR"5Ul[R"URURS9Ulg)NTr1rB)rr)rlrmrrdecoder_attention_headsr3 self_attnrrr(rrrDself_attn_layer_norm encoder_attnencoder_attn_layer_normrdecoder_ffn_dimr6r7rHs r(rmSpeechT5DecoderLayer.__init__Is *((44,,   zz&"7"78 $&LL1C1CI^I^$_!-     * *,,   (*||F4F4FFLaLa'b$/8N8NO " V-?-?VEZEZ [r*rr-encoder_hidden_statesencoder_attention_maskrcross_attn_layer_head_maskrr  use_cachec Un UbUSSOSn URUU UUUS9upn URU5nX-nURU5nSnSnUbMUn UbUSSOSnURUUUUUUS9upnURU5nX-nUR U5nX-n XR U5-nUR U5nU4nU(aUX4- nU (aUU 4- nU$)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, hidden_size)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. Nr)rrr-rr r)rrr-rrr )rArrBrCrDr6r7)ryrr-rGrHrrIrr rJr:self_attn_past_key_valueself_attn_weightspresent_key_valuecross_attn_present_key_valuecross_attn_weightscross_attn_past_key_valuer;s r(rSpeechT5DecoderLayer.forward_s[<!:H9S>"1#5Y] >Bnn'3)+/ ?M? ; *; ]3  0 11-@ (,$! ,$H@N?Yrs(;_c %NRN_N_+!65 :8"3 O`O KM/K!LL7M$4M 88GM!2 P &(9(9-(HH --m<  "  )> >G  )+ +Gr*)rrCrDr6r7rArB)NNNNNNFT)rrrrrrmrrrrrNrrrrs@r(r>r>Hs\~\2268<9=26=A8<,1$(R||R!.R ( 5 R !) 6 R "%,,/ R%-U\\$:R!u||!45R$D>RD>RRr*r>c*\rSrSr\rSrSrSrSr Sr g)SpeechT5PreTrainedModelispeecht5r+Tc [U[5(a[RR UR R SS[R"SUR RSUR R-- 5-S9 [RRUR RS5 g[U[5(a[R"SURR- 5n[RR!URR U*US9 [RR!URRU*US9 g[U[R"5(akUR R$R SUR&R(S9 URb%URR$R+5 gg[U[R,[R.45(aJURR$R+5 UR R$R1S5 g[U[R25(a[RR5UR 5 URbh[R"UR6URURS-- 5n[RR!URU*US9 gg[U[R85(axUR R$R SUR&R(S9 UR:b2UR R$UR:R+5 ggg) zInitialize the weightsrrrmeanstd)abr0Nr)r#rrinitnormal_rurrrri in_channels constant_rkr@rF in_featuresrTrEdatarzinitializer_rangezero_rrfill_rqkaiming_normal_rrr)rymoduleks r( _init_weights%SpeechT5PreTrainedModel._init_weightss f= > > GGOO "" !v{{'>'>q'AFKKD[D['["\]]   GG  fkk.. 2  9 : : !f//;;;..r*N) rrrrr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointingrhrrjr*r(rTrTs!L"$O&*#?r*rTc^\rSrSrSrS\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \ S \ \ \44S jjrS rU=r$)SpeechT5Encoderizm Transformer encoder consisting of *config.encoder_layers* layers. Each layer is a [`SpeechT5EncoderLayer`]. rzc>[TU]U5 [R"URUR S9Ul[R"UR5Ul URUl [R"[UR5Vs/sHn[U5PM sn5Ul[#URUR$-UR&5UlSUlUR-5 gs snf)NrBF)rlrmrrrrDrrr(rencoder_layerdrop layerdropr+rLencoder_layersr/rrr2encoder_max_relative_positionrr- post_initryrzr[r|s r(rmSpeechT5Encoder.__init__s  ,,v'9'9v?T?TUzz&"7"78 11mm5QWQfQfKg$hKga%9&%AKg$hi A   &"@"@ @&BfBf ',# %isDrr- head_maskr output_hidden_states return_dictr6c UbUOURRnUbUOURRnUbUOURRnUb[ X!R 5nUR U5nURU5nURU5n[5=(d [U5nU(aSOSn U(aSOSn Ub`UR5S[UR5:wa6[S[UR5SUR5SS35e[UR5HupU(aX4-n Sn UR (a$["R$"/5nXR&:n U (aU(a`UR((a9UR (a(UR+U R,UUUbX;OSUU5nOU "UUUUbX;OSUS9nUSnU (aS nU(dMU WS 4-n M U(aX4-n U(d[/S XU 455$[1UU U S 9$) aA Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, feature_size)`): Features extracted from the speech or text input by the encoder prenet. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nrjrz&The head_mask should be specified for  layers, but it is for .F)r-r rr rrc3.# UH ocMUv M g7frrj.0vs r( *SpeechT5Encoder.forward..Zsm$[q$[s last_hidden_stater attentions)rzr rzuse_return_dictrrCrrrrrrrQrr% enumerater:rrGrsr-r;r<tupler)ryrr-ryr rzr{r  synced_gpusall_hidden_statesall_self_attentionsidx encoder_layerskip_the_layerdropout_probability layer_outputss r(rSpeechT5Encoder.forwardsIH2C1N-TXT_T_TqTq$8$D $++JjJj &1% p 'tn p d^p  uo% &p p r*rpc^\rSrSrSrS\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \ S \ \ \44S jjrS rU=r$)SpeechT5EncoderWithSpeechPreneticz Wrapper around SpeechT5Encoder that applies SpeechT5SpeechEncoderPrenet to convert the audio waveform data to hidden features. rzc>[TU]U5 [U5Ul[ U5UlUR 5 gr)rlrmrNprenetrpwrapped_encoderrvrHs r(rm(SpeechT5EncoderWithSpeechPrenet.__init__i5  1&9 .v6 r*r+r-ryr rzr{r6c TURX5uprURUUUUUUS9nU$N)rr-ryr rzr{rr ryr+r-ryr rzr{rr;s r(r'SpeechT5EncoderWithSpeechPrenet.forwardqsC)- L(Q% &&')/!5# ' r*rrrrs@r(rrcs ~26,0,0/3&*''!.ELL)  $D>  'tn d^ uo% &r*rc^\rSrSrSrS\4U4SjjrSrSrSS\ RS\ \ RS \ \ RS \ \ S \ \ S \ \ S \\\44SjjrSrU=r$)SpeechT5EncoderWithTextPrenetizt Wrapper around SpeechT5Encoder that applies SpeechT5TextEncoderPrenet to convert the input_ids to hidden features. rzc>[TU]U5 [U5Ul[ U5UlUR 5 gr)rlrmrrrprrvrHs r(rm&SpeechT5EncoderWithTextPrenet.__init__5  /7 .v6 r*c6URR5$rrrr[s r(r2SpeechT5EncoderWithTextPrenet.get_input_embeddings{{//11r*c:URRU5 grrrrs r(r2SpeechT5EncoderWithTextPrenet.set_input_embeddings ((/r*r+r-ryr rzr{r6c PURU5nURUUUUUUS9nU$rrrs r(r%SpeechT5EncoderWithTextPrenet.forwards@ L1 &&')/!5# ' r*rr)rrrrrrrmrrrrrrrNrrrrrrrs@r(rrs~20 26,0,0/3&*''!.ELL)  $D>  'tn d^ uo% &r*rc^\rSrSrSrS\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \ S \ \ \44S jjrS rU=r$)SpeechT5EncoderWithoutPreneti This wrapper class is a helper class to correctly load pretrained checkpoints when used in combination with [`SpeechT5Model`]. rzcd>[TU]U5 [U5UlUR 5 gr)rlrmrprrvrHs r(rm%SpeechT5EncoderWithoutPrenet.__init__)  .v6 r*r+r-ryr rzr{r6c *URUUUUUUS9$rr)ryr+r-ryr rzr{s r(r$SpeechT5EncoderWithoutPrenet.forwards.##&)/!5# $  r*rrrrs@r(rrs ~26,0,0/3&* '' !. ELL)  $D>  'tn  d^  uo% &  r*rcf^\rSrSrSrS\4U4SjjrSS\\RS\\RS\\RS\\RS \\RS \\RS \\ \RS \\ S \\ S\\ S\\ S\\\44SjjrSrU=r$)SpeechT5Decoderizl Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`SpeechT5DecoderLayer`] rzc>[TU]U5 URUl[R "[ UR5Vs/sHn[U5PM sn5Ul SUl UR5 gs snfr2) rlrmdecoder_layerdroprsrr+rLdecoder_layersr>rr-rvrws r(rmSpeechT5Decoder.__init__sj  11mm5QWQfQfKg$hKga%9&%AKg$hi &+#  %is Brr-rGrHrycross_attn_head_maskrrJr rzr{r6c U bU OURRn U bU OURRn UbUOURRnU bU OURRn UR 5SSn UbUSSR SOSn [X,X5nUbUb[XARU SS9n[5=(d [U5nUR(a/UR(aU(a[RS5 SnU (aSOSnU (aSOSnU (aUbSOSnU(aSOSn[!XV/S S /5HnunnUcM UR 5S[#UR$5:wdM7['S US [#UR$5S UR 5SS35e [)UR$5GHunnU (aX4-nSnUR(a%[*R,"/5nUUR.:nU(a U(dM[UbUUOSnUR(aFUR(a5UR1UR2UUUUUbUUOSUbUUOSSU U5 nOU"UUUUUbUUOSUbUUOSUU US9 nUSnU(aUUU (aSOS4- nU (dGMUUS4-nUcGMUUS4-nGM U (aX4-nU(aUOSnU (d[5SUUUUU455$[7UUUUUS9$)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, feature_size)`): Features extracted from the speech or text input by the decoder prenet. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr!rr)rzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FrjryrzThe `z` should be specified for r}r~)r-rGrHrrIrr rJr rc30# UH nUcMUv M g7frrjrs r(r*SpeechT5Decoder.forward..srArs )rrrrcross_attentions)rzr rzrJrrr#rrrCrrr-r:logger warning_oncervrQrr%rrrGrsr;r<rr)ryrr-rGrHryrrrJr rzr{rrrrrall_cross_attentionsnext_decoder_cache attn_mask mask_namer decoder_layerrrrr next_caches r(rSpeechT5Decoder.forwardsR2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK>#A&3t{{+;<$ {*DSEUDVW%NN,Q/03%r#,DKK"8 C#$58H$H!#N}}&+jjn#!4t~~!Ek5D5P_S1VZN**t}} $ A A!**!")*&/&;IcN1E1Q(-W[% ! !.!#1*?+A7@7LYs^RV5I5U,S1[_#1&7' ! *!,M"}:KQQR'S&UU"  &9]1=M| $D>| 'tn| d^|  u?? @| | r*rc^\rSrSrSrS\4U4SjjrSS\\RS\\RS\\RS\\RS \\RS \\RS \\RS \\ \RS \\ S\\ S\\ S\\ S\\\44SjjrSrU=r$)SpeechT5DecoderWithSpeechPrenetiz| Wrapper around SpeechT5Decoder that applies SpeechT5SpeechDecoderPrenet to convert log-mel filterbanks to hidden features. rzc>[TU]U5 [U5Ul[ U5UlUR 5 gr)rlrmrrrwrapped_decoderrvrHs r(rm(SpeechT5DecoderWithSpeechPrenet.__init__rr*r+r-rGrHrryrrrJr rzr{r6c ZURX5n URU UUUUUUU U U U S9 nU$N) rr-rGrHryrrrJr rzr{rr)ryr+r-rGrHrryrrrJr rzr{decoder_hidden_statesr;s r(r'SpeechT5DecoderWithSpeechPrenet.forwardsP!% L M&&/)"7#9!5+/!5#'  r*r) NNNNNNNNNNNNrrs@r(rrs6 ~5959=A=A59,07;=A$(,0/3&*u001!!1!12 ((9(9:  !))9)9 :  %U\\2 ELL)'u||4"$u'8'8"9:D>$D>'tnd^ u?? @r*rcr^\rSrSrSrS\4U4SjjrSrSrSS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \\ RS\ \S\ \S\ \S\ \S\\\44SjjrSrU=r$)SpeechT5DecoderWithTextPrenetizs Wrapper around SpeechT5Decoder that applies SpeechT5TextDecoderPrenet to convert input tokens to hidden features. rzc>[TU]U5 [U5Ul[ U5UlUR 5 gr)rlrmrrrrrvrHs r(rm&SpeechT5DecoderWithTextPrenet.__init__rr*c6URR5$rrr[s r(r2SpeechT5DecoderWithTextPrenet.get_input_embeddingsrr*c:URRU5 grrrs r(r2SpeechT5DecoderWithTextPrenet.set_input_embeddingsrr*r+r-rGrHryrrrJr rzr{r6c `URXU5upURU UUUUUUUU U U S9 n U $rr)ryr+r-rGrHryrrrJr rzr{rr;s r(r%SpeechT5DecoderWithTextPrenet.forwardsV15 LZi0j-&&/)"7#9!5+/!5#'  r*rr)rrrrrrrmrrrrrrrrrNrrrrrrrs@r(rrs*~20 5959=A=A,07;=A$(,0/3&*u001!!1!12 ((9(9:  !))9)9 :  ELL) 'u||4"$u'8'8"9:D>$D>'tnd^ u?? @r*rcf^\rSrSrSrS\4U4SjjrSS\\RS\\RS\\RS\\RS \\RS \\RS \\ \RS \\ S \\ S\\ S\\ S\\\44SjjrSrU=r$)SpeechT5DecoderWithoutPrenetirrzcd>[TU]U5 [U5UlUR 5 gr)rlrmrrrvrHs r(rm%SpeechT5DecoderWithoutPrenet.__init__ rr*r+r-rGrHryrrrJr rzr{r6c 8URUUUUUUUUU U U S9 n U $rr) ryr+r-rGrHryrrrJr rzr{r;s r(r$SpeechT5DecoderWithoutPrenet.forwardsA&&&)"7#9!5+/!5#'  r*rrrrs@r(rrs  ~5959=A=A,07;=A$(,0/3&*u001!!1!12 ((9(9:  !))9)9 :  ELL) 'u||4"$u'8'8"9:D>$D>'tnd^ u?? @r*rc^\rSrSrSrS\4U4SjjrS\RS\RS\RS\R4S jr S r \ S 5rS rU=r$) $SpeechT5GuidedMultiheadAttentionLossi0z Guided attention loss from the paper [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://arxiv.org/abs/1710.08969), adapted for multi-head attention. rzcf>[TU]5 URUlURUlgr)rlrmguided_attention_loss_sigmasigmaguided_attention_loss_scalescalerHs r(rm-SpeechT5GuidedMultiheadAttentionLoss.__init__6s( 77 77 r*r input_masks output_masksr6cDURX#UR5nURS5URS5-nURUR5RS5nXA-n[R "UR U55nURU-$)a Compute the attention loss. Args: attentions (`torch.FloatTensor` of shape `(batch_size, layers * heads, output_sequence_length, input_sequence_length)`): Batch of multi-head attention weights input_masks (`torch.BoolTensor` of shape `(batch_size, input_sequence_length)`): Input attention mask as booleans. output_masks (`torch.BoolTensor` of shape `(batch_size, output_sequence_length)`): Target attention mask as booleans. Returns: `torch.Tensor` with the loss value r!rr)_make_guided_attention_masksrrrrrX masked_selectr)ryrrrguided_attn_masksmaskslosseslosss r(r,SpeechT5GuidedMultiheadAttentionLoss.forward;s"!==kYcYjYjk&&r*[-B-B2-FF**+55a8"/zz&..u56zzD  r*cjURS5nURS5n[R"[U5URSURS4US9n[ [ XE55H.unupURXURU5XgSU 2SU24'M0 URS5$)Nr!rri) rJrrMrQr#rrv_make_guided_attention_maskrr) ryrrrr\rmrrilenolens r(rASpeechT5GuidedMultiheadAttentionLoss._make_guided_attention_masksTs#+ %))"-!KK[)9<;M;Ma;PR]RcRcdeRf(gpvw!*3}+M!N C$373S3STX`d`j`jlr3s 5D5%4%/ 0"O!**1--r*c[R"[R"XS9[R"XS9SS9upEUR5U- nUR5U- nS[R"XE- S-*SUS--- 5- $)Nrixy)indexingrr)rmeshgridrPrr)r< output_lengthrrgrid_ygrid_xs r(r@SpeechT5GuidedMultiheadAttentionLoss._make_guided_attention_mask_sz LL 5 LL 6  -/,.UYY&/a!78ANKLLLr*)rr)rrrrrrrmrr BoolTensorrrrrrrrrs@r(rr0sj 8~8 !++!:?:J:J!Z_ZjZj! !2 .MMr*rc^\rSrSrSrS\4U4SjjrSS\RS\RS\RS\RS \RS \ \RS \R4S jjr S r U=r$)SpeechT5SpectrogramLossikz3 Loss computation used by SpeechT5ForTextToSpeech. rzc.>[TU]5 URUlURUlURUl[ 5Ul[[R"S5S9Ul UR(a[U5Ul gg)Ng@) pos_weight) rlrmuse_guided_attention_lossguided_attention_loss_num_headsr,r l1_criterionrrr  bce_criterionrattn_criterionrHs r(rm SpeechT5SpectrogramLoss.__init__pss )/)I)I&/5/U/U, & 7 7"H.%,,s:KL  ) )"Fv"ND  *r*r-rrrlabelsrr6c VUS:gnURU5nURU5nURU5nURX55URX%5-nUSS2SS2S4n [R"U )S-[R"U R S5S5R U R5/SS9n U SS2SS24RU 5n URU 5nURXJ5n X-n UR(a[R"UV s/sHoSS2SUR24PM sn SS9nUS:HnUSS2SS2S4nURS:a#USS2URS- SUR24nURXU5nU U- n U $s sn f)Nr/rrrr) rrrrrSrrrrrrr,r)ryr-rrrrrrdl1_lossr stop_labelsbce_lossrxattnrr attn_losss r(rSpeechT5SpectrogramLoss.forward|s' %%l3!7!E!El!S 5 C CL Q##$9BTEVEVWmEvvQ1W%ii%#uzz%**Q-/K/N/Nu||/\ ]cde !!QR%(66u= %%e,%%f:!  ) )99TdeTdq#IT%I%I#I IJTdeklmD(A-K'1a0L$$q(+At/D/Dq/H/aDLaLa/a,ab ++D|LI I D fs%F&)rrrrr,rr)rrrrrrrmrrrrrrrrrs@r(r r ks O~ O&9=)(()!& 1 1) %00 ) !! ) !! )#5#4#45) ))r*r zv The bare SpeechT5 Encoder-Decoder Model outputting raw hidden-states without any specific pre- or post-nets.  custom_introc"Z^\rSrSrSS\S\\RS\\R4U4SjjjrSr Sr Sr S r S r \SS \\R S \\R"S \\R S\\R"S\\R$S\\R$S\\R S\\\\R$S\\\\R$S\\S\\R$S\\S\\S\\S\\\R$\44Sjj5rSrU=r$) SpeechT5Modelirzencoderdecoderc>[TU]U5 XlUc [U5OUUlUc [ U5OUUlUR5 g)z encoder (`PreTrainedModel`, *optional*): The encoder model to use. decoder (`PreTrainedModel`, *optional*): The decoder model to use. N)rlrmrzrr"rr#rv)ryrzr"r#r|s r(rmSpeechT5Model.__init__sK   ?F3F;T[ ?F3F;T[  r*c[UR[5(aURR5$[UR[ 5(aURR5$[ er)r#r"rrr#rNotImplementedErrorr[s r(r"SpeechT5Model.get_input_embeddingssR dll$A B B<<446 6 dll$A B B<<446 6!!r*c[UR[5(aURRU5 [UR[ 5(aURRU5 ggr)r#r"rrr#rrs r(r"SpeechT5Model.set_input_embeddingssP dll$A B B LL - -e 4 dll$A B B LL - -e 4 Cr*cUR$r)r"r[s r( get_encoderSpeechT5Model.get_encoderrr*cUR$r)r#r[s r( get_decoderSpeechT5Model.get_decoderrr*c[UR[5(a%URRR 5 ggz Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)r#r"rrr\r[s r(r\$SpeechT5Model.freeze_feature_encoders2 dll$C D D LL   6 6 8 Er*r+r-decoder_input_valuesdecoder_attention_maskrydecoder_head_maskrencoder_outputsrrJrr rzr{r6cU bU OURRn U bU OURRn U bU OURRn UbUOURRnUcUR UUUU U US9nORU(aK[ U[5(d6[US[U5S:aUSOS[U5S:aUSOSS9nUbV[ UR [5(a7UR RRUSRSU5nOUn[ UR[5(aSU 0nO0nUR"S UUUSUUUU U U U US. UD6nU(dUU-$[UR UR"UR$UR&UR(UR UR$UR&S 9$) a) input_values (`torch.Tensor` of shape `(batch_size, sequence_length)`): Depending on which encoder is being used, the `input_values` are either: float values of the input raw speech waveform, or indices of input sequence tokens in the vocabulary, or hidden states. decoder_input_values (`torch.Tensor` of shape `(batch_size, target_sequence_length)`, *optional*): Depending on which decoder is being used, the `decoder_input_values` are either: float values of log-mel filterbank features extracted from the raw speech waveform, or indices of decoder input sequence tokens in the vocabulary, or hidden states. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. N)r+r-ryr rzr{rrrrr) r+r-rGrHryrrrJr rzr{)rrrdecoder_attentionsrencoder_last_hidden_staterGencoder_attentionsrj)rzr rzrJrr"r#rrQrrr`r#r#rrrrrrr)ryr+r-r4r5ryr6rr7rrJrr rzr{rH decoder_argsdecoder_outputss r(rSpeechT5Model.forwardsR2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK,0/3&*i u||,i !!1!12i 'u||4 i !))9)9 : i E--. i $E$5$56i 'u||4i "%e.?.?(@"ABi "%e.?.?(@"ABi D>i %U%6%67i $D>i 'tni d^i uU&&');; [TU]U5 URc[SURS35e[ U5n[ U5n[XU5Ul[U5Ul UR5 g)NYou are trying to instantiate a with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SpeechT5ForSpeechToText.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.) rlrmrr%r|rrr!rUrtext_decoder_postnetrv)ryrzspeech_encoder text_decoderr|s r(rm SpeechT5ForSpeechToText.__init__Rs}     $00@A// 9@4V< %flK $>v$F! r*c6URR5$rrUr,r[s r(r,#SpeechT5ForSpeechToText.get_encoderf}}((**r*c6URR5$rrUr/r[s r(r/#SpeechT5ForSpeechToText.get_decoderirLr*cTUR5RR5 gr2r,rr\r[s r(r\.SpeechT5ForSpeechToText.freeze_feature_encoderl !!88:r*c6URR5$r)rErr[s r(r-SpeechT5ForSpeechToText.get_output_embeddingsss((>>@@r*c:URRU5 gr)rErrs r(r-SpeechT5ForSpeechToText.set_output_embeddingsvs !!77Gr*r+r-decoder_input_idsr5ryr6rr7rrJr rzr{rr6cU bU OURRn Ub7Uc4[XRRURR5nUR UUUUUUUUU U U U SS9 nUR US5nSnUbF[5nU"URSURR5URS55nU (dU4USS-nUbU4U-$U$[UUURURURURURUR UR"S9 $)a% input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (*pip install soundfile*). To prepare the array into `input_values`, the [`SpeechT5Processor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SpeechT5Processor.__call__`] for details. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) SpeechT5 uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Label indices can be obtained using [`SpeechT5Tokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForSpeechToText >>> from datasets import load_dataset >>> dataset = load_dataset( ... "hf-internal-testing/librispeech_asr_demo", "clean", split="validation", trust_remote_code=True ... ) # doctest: +IGNORE_RESULT >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_asr") >>> model = SpeechT5ForSpeechToText.from_pretrained("microsoft/speecht5_asr") >>> # audio file is decoded on the fly >>> inputs = processor(audio=dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt") >>> predicted_ids = model.generate(**inputs, max_length=100) >>> # transcribe speech >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True) >>> transcription[0] 'mister quilter is the apostle of the middle classes and we are glad to welcome his gospel' ``` ```python >>> inputs["labels"] = processor(text_target=dataset[0]["text"], return_tensors="pt").input_ids >>> # compute loss >>> loss = model(**inputs).loss >>> round(loss.item(), 2) 19.68 ``` NT) r+r-r4r5ryr6rr7rrJr rzr{rr!r) rrrrr9rr:rGr;)rzrr)rrrUrEr rrrrrr9rr:rGr;)ryr+r-rXr5ryr6rr7rrJr rzr{rr;rrloss_fctoutputs r(rSpeechT5ForSpeechToText.forwardysXr&1%# UH/oRSTRUR55v M1 g7f)rN)rrr)r past_statebeam_idxs r(r9SpeechT5ForSpeechToText._reorder_cache.. s1ncmU_--aZ=N=N1OPPcms7:)r)rr`reordered_past layer_pasts ` r(_reorder_cache&SpeechT5ForSpeechToText._reorder_cache s8)J ncmnn N*r*)rUrEr?)rrrr_tied_weights_keysrrmr,r/r\rrrrrrrrrrNrrrrrdrrrs@r(rBrBJs @@~(++;AH59598<=A159=7;EIEI$(,0/3&*-1E u001E !!1!12E $E$4$45 E !))9)9 : E E--. E $E$5$56E 'u||4E "%e.?.?(@"ABE "%e.?.?(@"ABE D>E $D>E 'tnE d^E ))*E uo% &!E E Nr*rBmodelr threshold minlenratio maxlenratiovocoderoutput_cross_attentionsreturn_output_lengthsc  Uc [S5eUc*SXRR:HR5- n OUn UR S5n UR R UU SS9n U Rn [UR R [5(a@UR R RRU SRSU 5n [U R S5U-URR- 5n[U R S5U-URR- 5nU RU SURR5n/n/nSnSn0nUS- nUR R RUU5nUR R R#USS2SS24SU U USUSS9nU(a.UR%[&R("UR*SS95 URR-S5nUR.nUR0R3U5nUR5XRRURR5nUR%U5 USS2SSS24R5U SURR5n[&R("UU4SS9n[&R6"UR0R9U55nUU:aGMUU:a@[&R:"USS9U:n[&R<"U5SR?5nO[A[CU55nUVs/sH nUU;dM UPM nn[CU5S:ad[&RD"U5nURGSS5RISS 5nUR0RKU5nUH n UU UU 'M [CU5U :aOGM[A[CU55Vs/sHnUUPM nnU (dU S:XaUSO1[&RLRNRPRSUSS 9nUb U"U5n!OUn!U(a_[&R("US S9nU S:a@UR4"U [UR S5U - 5/UR 5S SQ76nU!U4n!U!$/n"[AU 5H&nU"R%UUR S55 M( Uc7[&RLRNRPRSUSS 9nUU"4n!Oy/n#[&RLRNRPRSUSS 9nU"U5n#U"Vs/sH,n[U#R S5[UU"5- 5U-PM. n$nU#U$4n!U(a\[&R("US S9nUR4"U [UR S5U - 5/UR 5S SQ76n/U!QUP7n!U!$s snfs snfs snf) Na`speaker_embeddings` must be specified. For example, you can use a speaker embeddings by following the code snippet provided in this link: https://huggingface.co/datasets/Matthijs/cmu-arctic-xvectors rrT)r+r-r{r!)rr-rGrHrrJr r{rr) batch_first)+r%rzrr:rrUr"rr#rrr`r#r,r"rr#rrUrrrsqueezerspeech_decoder_postnetrrsigmoidrrJrrKrLrQstackrflattenrrrrnn pad_sequencer;)%rgr+rr-rhrirjrkrlrmrHr encoder_outr:maxlenminlenoutput_sequence spectrogramrrrresult_spectrogramr decoder_outlast_decoder_outputspectrumnew_spectrogramprobmeet_thresholds meet_indexesr/ spectrograms meet_indexr;spectrogram_lengths waveformswaveform_lengthss% r(_generate_speechr s!   !"lll6O6O&O%T%T%V!V!/   A C..((!-)K !, = =%..((*IJJ!&!7!7!>!>!a!a N  #%;" *//2[@5<EEG $SY/ '3S|!q@R7RA|LS< 1$${{;7 +55a;CCAqI $;;CCLQ ".J5A*5M&z2#/%&#-i j49=O9P3QR3Qa&q)3QLR ),l1ouxx~~7I7I7V7VWcqu7V7v  k*G!G "$yy)9qA Qw#3#8#8-2215;<$?O?T?T?VWYWZ?[$  01G* N%!sA  & &|A';';A'> ? ? 88>>--::>--::[TU]U5 URc[SURS35e[ U5n[ U5n[XU5Ul[U5Ul UR5 g)NrDa with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SpeechT5ForTextToSpeech.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.) rlrmrr%r|rrr!rUrrrrv)ryrz text_encoderspeech_decoderr|s r(rm SpeechT5ForTextToSpeech.__init__ s}     $00@A// 5V< 8@%fNK &B6&J# r*r6cgr9rj)clss r( can_generate$SpeechT5ForTextToSpeech.can_generate s r*c6URR5$rrJr[s r(r,#SpeechT5ForTextToSpeech.get_encoder rLr*c6URR5$rrNr[s r(r/#SpeechT5ForTextToSpeech.get_decoder rLr*r-r4r5ryr6rr7rrJr rzr{rrrcpU bU OURRn UbBUc"[XRRU5up4URR(aSn UR UUUUUUUUU U UU U SS9nUR US5unnnSnUb,[UR5nU"UUUUUUR5nU (dU4USS-nUbU4U-$U$[UUURURURURURURURS9 $)a@ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`~PreTrainedTokenizer.encode`] and [`~PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`): Float values of input mel spectrogram. SpeechT5 uses an all-zero spectrum as the starting token for `decoder_input_values` generation. If `past_key_values` is used, optionally only the last `decoder_input_values` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. labels (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`, *optional*): Float values of target mel spectrogram. Timesteps set to `-100.0` are ignored (masked) for the loss computation. Spectrograms can be obtained using [`SpeechT5Processor`]. See [`SpeechT5Processor.__call__`] for details. stop_labels (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Binary tensor indicating the position of the stop token in the sequence. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan, set_seed >>> import torch >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_tts") >>> model = SpeechT5ForTextToSpeech.from_pretrained("microsoft/speecht5_tts") >>> vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan") >>> inputs = processor(text="Hello, my dog is cute", return_tensors="pt") >>> speaker_embeddings = torch.zeros((1, 512)) # or load xvectors from a file >>> set_seed(555) # make deterministic >>> # generate speech >>> speech = model.generate(inputs["input_ids"], speaker_embeddings=speaker_embeddings, vocoder=vocoder) >>> speech.shape torch.Size([15872]) ``` NTr+r-r4r5ryr6rr7rrJrr rzr{rr rr|rrr9rr:rGr;)rzrr2r,rrUrrr rrrrr9r:rGr;)ryrr-r4r5ryr6rr7rrJr rzr{rrrr;rrrr criterionr[s r(rSpeechT5ForTextToSpeech.forward sxX&1%.2.2#B E,,-B !!1!12B 'u'8'89 B !))9)9 : B E--. B $E$5$56B 'u||4B "%e.?.?(@"ABB "%e.?.?(@"ABB D>B $D>B 'tnB d^B %U%6%67B **+!B "ell+#B $ u.. /%B B H ]]_6::> !'+(-&+Y ##Y !!1!12Y %U%6%67 Y  Y  Y Y "))$Y "&Y  $Y  u  %(9(95;L;L(L"MM NY Y v ]]_;?59 !'+(-&+] ##] %U%6%67] !!1!12 ]  ]  ] ] "))$] "&]  $]  u  %(9(95;L;L(L"MM N] ] r*rzD SpeechT5 Model with a speech encoder and a speech decoder. c&^\rSrSrS\4U4SjjrSrSrSr\ S!S\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \\\ RS\ \\\ RS\ \S\ \S\ \S\ \S\ \ RS\ \ RS\ \ RS\\\44"Sjj5r\ R("5S"S\ RS\ \ RS\ \ RS\S\S\S\ \R.S\S\S\ R4Sjj5rS rU=r$)#SpeechT5ForSpeechToSpeechi rzc>[TU]U5 [U5n[U5n[ XU5Ul[ U5UlUR5 gr) rlrmrrr!rUrrrrv)ryrzrFrr|s r(rm"SpeechT5ForSpeechToSpeech.__init__ sK  8@8@%fnM &B6&J# r*c6URR5$rrJr[s r(r,%SpeechT5ForSpeechToSpeech.get_encoder rLr*c6URR5$rrNr[s r(r/%SpeechT5ForSpeechToSpeech.get_decoder rLr*cTUR5RR5 gr2rQr[s r(r\0SpeechT5ForSpeechToSpeech.freeze_feature_encoder rSr*r+r-r4r5ryr6rr7rrJr rzr{rrrr6cU bU OURRn Ub%Uc"[XRRU5up4UR UUUUUUUUU U UU U SS9nUR US5unnnSnU (dU4USS-nUbU4U-$U$[ UUURURURURURURURS9 $)a input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (*pip install soundfile*). To prepare the array into `input_values`, the [`SpeechT5Processor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SpeechT5Processor.__call__`] for details. decoder_input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`): Float values of input mel spectrogram. SpeechT5 uses an all-zero spectrum as the starting token for `decoder_input_values` generation. If `past_key_values` is used, optionally only the last `decoder_input_values` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. labels (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`, *optional*): Float values of target mel spectrogram. Spectrograms can be obtained using [`SpeechT5Processor`]. See [`SpeechT5Processor.__call__`] for details. stop_labels (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Binary tensor indicating the position of the stop token in the sequence. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForSpeechToSpeech, SpeechT5HifiGan, set_seed >>> from datasets import load_dataset >>> import torch >>> dataset = load_dataset( ... "hf-internal-testing/librispeech_asr_demo", "clean", split="validation", trust_remote_code=True ... ) # doctest: +IGNORE_RESULT >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_vc") >>> model = SpeechT5ForSpeechToSpeech.from_pretrained("microsoft/speecht5_vc") >>> vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan") >>> # audio file is decoded on the fly >>> inputs = processor(audio=dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt") >>> speaker_embeddings = torch.zeros((1, 512)) # or load xvectors from a file >>> set_seed(555) # make deterministic >>> # generate speech >>> speech = model.generate_speech(inputs["input_values"], speaker_embeddings, vocoder=vocoder) >>> speech.shape torch.Size([77824]) ``` NTrrrr)rzrr2r,rUrrrrrr9rr:rGr;)ryr+r-r4r5ryr6rr7rrJr rzr{rrrr;r[r|rrr[s r(r!SpeechT5ForSpeechToSpeech.forward$ s%d&1%Q>Q!R          # !  r*rrr)rrrrrrmr,r/r\rrrrrrrrNrrrrrrr@rrrrs@r(rr s ~ ++;5959<@=A159=7;EIEI$(,0/3&*:>.2.2#| u001| !!1!12| 'u'8'89 | !))9)9 : | E--. | $E$5$56| 'u||4| "%e.?.?(@"AB| "%e.?.?(@"AB| D>| $D>| 'tn| d^| %U%6%67| **+!| "ell+#| $ u.. /%| | | ]]_;?59 !'+(-&+V ''V %U%6%67V !!1!12 V  V  V V "))$V "&V  $V    V V r*rcH^\rSrSrSU4SjjrS SjrSrSrSrSr U=r $) HifiGanResidualBlocki c>[TU]5 X@l[R"[ [ U55Vs/sH0n[R"UUUSX5URX#U5S9PM2 sn5Ul [R"[ [ U55Vs/sH,n[R"UUUSSURUS5S9PM. sn5Ul gs snfs snf)Nr)rjdilationr) rlrmleaky_relu_sloperr+rLrQrq get_paddingconvs1convs2)rychannelsrirrr/r[r|s r(rmHifiGanResidualBlock.__init__ s  0mms8}- .A %[ ,,[1+F .    mms8}- .A  ,,[!< .      s 7C%%3C*cX-U- S-$rrj)ryrirs r(r HifiGanResidualBlock.get_padding s&1a77r*c>[RRn[[RRS5(a$[RRRnUR H nU"U5 M UR H nU"U5 M gNr)rrrrrrrryrr(s r(apply_weight_norm&HifiGanResidualBlock.apply_weight_norm sihh** 288,,m < <((33??K[[E  ![[E  !r*cURH"n[RRU5 M$ URH"n[RRU5 M$ gr)rrrremove_weight_normrryr(s r(r'HifiGanResidualBlock.remove_weight_norm* sB[[E HH ' ' .![[E HH ' ' .!r*c([URUR5Hmup#Un[RR XR 5nU"U5n[RR XR 5nU"U5nX-nMo U$r)rvrrrr leaky_relur)ryrconv1conv2r:s r(rHifiGanResidualBlock.forward0 sz T[[9LE$HMM44]DYDYZM!-0MMM44]DYDYZM!-0M)4M :r*)rrr)r )rr g?)r) rrrrrmrrrrrrrs@r(rr s! >8/ r*rz HiFi-GAN vocoder. c^\rSrSr\rSrS\4U4SjjrSrSr Sr \ "SS 9S\ RS \ R4S j5rS rU=r$) SpeechT5HifiGani; r|rzc >[TU]U5 [UR5Ul[UR 5Ul[R"URURSSSS9Ul [R"5Ul [[UR UR 55Haunup4URR#[R$"URSU--URSUS---UUXC- S-S95 Mc [R"5Ul[)[UR55HpnURSUS---n[URUR*5H4upFUR&R#[-XTXaR.55 M6 Mr [R"WSSSSS9UlUR3S[4R6"UR55 UR3S[4R8"UR55 UR;5 g)Nrr )rirjrrrXr)rlrmrQresblock_kernel_sizes num_kernelsupsample_rates num_upsamplesrrq model_in_dimupsample_initial_channelconv_prer+ upsamplerrrvupsample_kernel_sizesrUConvTranspose1d resblocksrLresblock_dilation_sizesrr conv_postrrrMrSrv)ryrzr/ upsample_raterirrr|s r(rmSpeechT5HifiGan.__init__D s  v;;< !6!67     + +   /8V=R=RTZTpTp9q/r +A+ NN ! !""331=33a!e E +((8Q>  0ss4>>*+A661Q<HH),V-I-I6KiKi)j% %%&:8RZ\s\s&tu*k, 8QAaQRS VU[[1D1D%EF Wejj1D1D&EF r*c8[U[R[R45(akURR R SURRS9 URb%URR R5 ggg)zInitialize the weights.r0rWN) r#rrErqrrar]rzrbrkrc)ryrfs r(rhSpeechT5HifiGan._init_weightsj sm fryy"))4 5 5 MM   & &CT[[5R5R & S{{&   &&(' 6r*c[RRn[[RRS5(a$[RRRnU"UR 5 UR H nU"U5 M URHnUR5 M U"UR5 gr) rrrrrrrrrrrs r(r!SpeechT5HifiGan.apply_weight_normq shh** 288,,m < <((33??KDMM"^^E  $^^E  # # %$DNN#r*cR[RRUR5 URH"n[RRU5 M$ UR HnUR5 M [RRUR 5 gr)rrrrrrrrs r(r"SpeechT5HifiGan.remove_weight_norm} sh ##DMM2^^E HH ' ' .$^^E  $ $ &$ ##DNN3r*a Converts a log-mel spectrogram into a speech waveform. Passing a batch of log-mel spectrograms returns a batch of speech waveforms. Passing a single, un-batched log-mel spectrogram returns a single, un-batched speech waveform. rr6cURR(aXR- UR- nUR 5S:HnU(dUR S5nUR SS5nURU5n[UR5Hn[RRX0RR5nURU"U5nURX@R -"U5n[SUR 5H)nXPRX@R -U-"U5- nM+ XPR - nM [RRU5nUR#U5n[$R&"U5nU(d2UR)S5R SS5R+S5nU$UR)S5nU$)a spectrogram (`torch.FloatTensor`): Tensor containing the log-mel spectrograms. Can be batched and of shape `(batch_size, sequence_length, config.model_in_dim)`, or un-batched and of shape `(sequence_length, config.model_in_dim)`. Returns: `torch.FloatTensor`: Tensor containing the speech waveform. If the input spectrogram is batched, will be of shape `(batch_size, num_frames,)`. If un-batched, will be of shape `(num_frames,)`. r rrrr!)rznormalize_beforerXrrrrrrLrrrrrrrrrrtanhrqr)ryr| is_batchedrr/ res_statejwaveforms r(rSpeechT5HifiGan.forward s" ;; ' '&2djj@K __&!+ %//2K#--a3  m4 t))*AMM44]KKD`D`aM NN1-mrTrprrrrrrrrr r!rBrrNrrrrr__all__rjr*r(rs // @@!)@7e.,I   H %!" %,, c [^ "ei0,,0250KSTYT`T`Ka0426 t c?tttU--. t  t ZZ tp299,82A8BIIA8J*bii*Zryy0""(299+RYY+^ 1 1D"))DN1"))1h%%P<299<2 .(- (-V & & ]B ]B@"))0:299:zi299iX"?o"?"?JF -F R"&="J'$;'T #: @L -L ^-&=-`1$;1h(#:(V8M2998Mv:bii:z Z +Z  Z z y5y  y~7;15#'$)"'L "L##L!!2!23LU--. L  L  LLbii L"L L 5  eE$5$5u7H7H$HI IJL^ c 5c  c L  p 7p  p f;299;| tot  tn r*