o ZŽh¶nã@s€dZddlZddlmZddlmZddlZddlZddl m Z ddl m m Z ddlmZddlmZmZdd lmZeGd d „d eƒƒZeGd d „d eƒƒZeGdd„deƒƒZGdd„de jƒZGdd„de jƒZGdd„de jƒZGdd„de jƒZGdd„de jƒZGdd„de jƒZGdd„de jƒZGdd„de jƒZ eGd d!„d!eƒƒZ!ed"d#Gd$d%„d%e!ƒƒZ"d%d!gZ#dS)&zTransformers DAC model.éN)Ú dataclass)ÚOptionalé)ÚPreTrainedModel)Ú ModelOutputÚauto_docstringé)Ú DacConfigc@sleZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eej ed<dZ eejed<dS)Ú DacOutputa. Args: loss (`torch.Tensor`): Loss from the encoder model, comprising the weighted combination of the commitment and codebook losses. audio_values (`torch.Tensor` of shape `(batch_size, input_length)`): Reconstructed audio data. quantized_representation (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`): Quantized continuous representation of input. audio_codes (`torch.LongTensor` of shape `(batch_size, num_codebooks, time_steps)`): Codebook indices for each codebook (quantized discrete representation of input). projected_latents (`torch.Tensor` of shape `(batch_size, num_codebooks * dimension, time_steps)`): Projected latents (continuous representation of input before quantization). NÚlossÚ audio_valuesÚquantized_representationÚ audio_codesÚprojected_latents)Ú__name__Ú __module__Ú __qualname__Ú__doc__r rÚtorchÚ FloatTensorÚ__annotations__r r rZ LongTensorr©rrúS/var/www/auris/lib/python3.10/site-packages/transformers/models/dac/modeling_dac.pyr s r c@sZeZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dS)ÚDacEncoderOutputaÛ Args: loss (`torch.Tensor`): Loss from the encoder model, comprising the weighted combination of the commitment and codebook losses. quantized_representation (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`, *optional*): Quantized continuous representation of input. audio_codes (`torch.Tensor` of shape `(batch_size, num_codebooks, time_steps)`, *optional*): Codebook indices for each codebook (quantized discrete representation of input). projected_latents (`torch.Tensor` of shape `(batch_size, num_codebooks * dimension, time_steps)`, *optional*): Projected latents (continuous representation of input before quantization). Nr r rr) rrrrr rrrrr rrrrrrr6s  rc@s$eZdZUdZdZeejed<dS)ÚDacDecoderOutputz¸ Args: audio_values (`torch.FloatTensor` of shape `(batch_size, input_length)`, *optional*): Decoded audio values, obtained using the decoder part of Dac. Nr ) rrrrr rrrrrrrrrJs rcs(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚSnake1dz; A 1-dimensional Snake activation function module. cs$tƒ ¡t t d|d¡¡|_dS)Nr)ÚsuperÚ__init__ÚnnÚ ParameterrÚonesÚalpha)ÚselfÚ hidden_dim©Ú __class__rrr[s zSnake1d.__init__cCsR|j}| |d|dd¡}||jd ¡t |j|¡ d¡}| |¡}|S)Nrréÿÿÿÿg•Ö&è .>é)ÚshapeÚreshaper!Z reciprocalrÚsinÚpow)r"Ú hidden_statesr(rrrÚforward_s ( zSnake1d.forward)rrrrrr-Ú __classcell__rrr$rrVs rcs6eZdZdZdef‡fdd„ Zdd„Zdd„Z‡ZS) ÚDacVectorQuantizeaÕ Implementation of VQ similar to Karpathy's repo (https://github.com/karpathy/deep-vector-quantization) Additionally uses following tricks from improved VQGAN (https://arxiv.org/pdf/2110.04627.pdf): 1. Factorized codes: Perform nearest neighbor lookup in low-dimensional space for improved codebook usage 2. l2-normalized codes: Converts euclidean distance to cosine similarity which improves training stability ÚconfigcsLtƒ ¡tj|j|jdd|_tj|j|jdd|_t |j |j¡|_ dS)Nr©Ú kernel_size) rrrÚConv1dÚ hidden_sizeÚ codebook_dimÚin_projÚout_projZ EmbeddingÚ codebook_sizeÚcodebook©r"r0r$rrrss zDacVectorQuantize.__init__cCsh| |¡}| |¡\}}tj|| ¡dd}tj|| ¡dd}||| ¡}| |¡}|||||fS)aJ Quantizes the input tensor using a fixed codebook and returns the corresponding codebook vectors. Args: hidden_state (`torch.FloatTensor` of shape `(batch_size, dimension, time_steps)`): Input tensor. Returns: quantized_representation (`torch.Tensor`of shape `(batch_size, dimension, time_steps)`): Quantized continuous representation of input. commitment_loss (`torch.FloatTensor`of shape `(1)`): Commitment loss to train encoder to predict vectors closer to codebook entries. codebook_loss (`torch.FloatTensor`of shape `(1)`): Codebook loss to update the codebook. audio_codes (`torch.LongTensor` of shape `(batch_size, time_steps)`): Codebook indices for each codebook, quantized discrete representation of input. projected_latents (torch.FloatTensor of shape `(batch_size, num_codebooks * dimension, time_steps)`): Projected latents (continuous representation of input before quantization). Úmean)Z reduction)r6Údecode_latentsÚFZmse_lossÚdetachr7)r"Ú hidden_staterr rÚcommitment_lossÚ codebook_lossrrrr-zs  zDacVectorQuantize.forwardc Cs¾|j\}}}| ddd¡ |||¡}|jj}t |¡}t |¡}| d¡jddd}|d||  ¡ | d¡jddd  ¡}|  d¡d} |  |  d¡d¡} | | ¡  dd¡} | | fS)Nrr'rT)Zkeepdimr&) r(Zpermuter)r9Úweightr=Ú normalizer+ÚsumÚtÚmaxÚsizeÚ transpose) r"r,Z batch_sizer#Zsequence_lengthÚ encodingsr9Zl2_normÚdistÚindicesr rrrr<šs   .z DacVectorQuantize.decode_latents) rrrrr rr-r<r.rrr$rr/gs   r/cs4eZdZdZd dedef‡fdd„ Zdd „Z‡ZS) ÚDacResidualUnitza A residual unit composed of Snake1d and weight-normalized Conv1d layers with dilations. érÚ dimensionÚdilationcsVtƒ ¡d|d}t|ƒ|_tj||d||d|_t|ƒ|_tj||dd|_dS)Nér'é)r2rOÚpaddingrr1) rrrÚsnake1rr3Úconv1Úsnake2Úconv2)r"rNrOÚpadr$rrr²s    zDacResidualUnit.__init__cCsb|}| | |¡¡}| | |¡¡}|jd|jdd}|dkr+|d|| …f}||}|S)ar Forward pass through the residual unit. Args: hidden_state (`torch.Tensor` of shape `(batch_size, channels, time_steps)`): Input tensor . Returns: output_tensor (`torch.Tensor` of shape `(batch_size, channels, time_steps)`): Input tensor after passing through the residual unit. r&r'r.)rTrSrVrUr()r"r?Z output_tensorrRrrrr-»s zDacResidualUnit.forward)rMr)rrrrÚintrr-r.rrr$rrL­s rLcó8eZdZdZd dededef‡fdd„ Zdd „Z‡ZS) ÚDacEncoderBlockz"Encoder block used in DAC encoder.rr0ÚstrideÚ stride_indexc sˆtƒ ¡|jd|}t|ddd|_t|ddd|_t|ddd|_t|dƒ|_t j |d|d||t   |d¡d|_ dS)Nr'r©rOré ©r2r[rR)rrÚencoder_hidden_sizerLÚ res_unit1Ú res_unit2Ú res_unit3rrSrr3ÚmathÚceilrT)r"r0r[r\rNr$rrrÕs  ÿzDacEncoderBlock.__init__cCs2| |¡}| |¡}| | |¡¡}| |¡}|S©N)rarbrSrcrT©r"r?rrrr-ás   zDacEncoderBlock.forward©rr© rrrrr rXrr-r.rrr$rrZÒs rZcrY) ÚDacDecoderBlockz"Decoder block used in DAC decoder.rr0r[r\c s†tƒ ¡|jd|}|jd|d}t|ƒ|_tj||d||t |d¡d|_ t |dd|_ t |dd|_ t |dd|_ dS)Nr'rr_r]rr^)rrÚdecoder_hidden_sizerrSrZConvTranspose1drdreÚconv_t1rLrarbrc)r"r0r[r\Z input_dimÚ output_dimr$rrrís   ûzDacDecoderBlock.__init__cCs6| |¡}| |¡}| |¡}| |¡}| |¡}|Srf)rSrlrarbrcrgrrrr-ÿs     zDacDecoderBlock.forwardrhrirrr$rrjêsrjcsZeZdZdZdef‡fdd„ Zddeefdd„Zd e j fd d „Z d e j fd d„Z ‡Z S)ÚDacResidualVectorQuantizez„ ResidualVectorQuantize block - Introduced in SoundStream: An end2end neural audio codec (https://arxiv.org/abs/2107.03312) r0csFtƒ ¡ˆj}ˆj}||_t ‡fdd„tˆjƒDƒ¡|_||_dS)Ncsg|]}tˆƒ‘qSr)r/)Ú.0Úi©r0rrÚ sz6DacResidualVectorQuantize.__init__..)rrÚ n_codebooksÚquantizer_dropoutrÚ ModuleListÚrangeÚ quantizers)r"r0rsrtr$rqrrs   z"DacResidualVectorQuantize.__init__NÚ n_quantizerscCsnd}|}d}d}g}g}|dur|n|j}|jrNt |jdf¡|jd}t d|jd|jdf¡} t|jd|jƒ} | d| …|d| …<| |j ¡}t |j ƒD]N\} } |jdurb| |krbn@| |ƒ\} }}}}tj |jdf| |j d|k}|| |dd…ddf}|| }|||7}|||7}|  |¡|  |¡qStj|dd}tj|dd}|||||fS)aQ Quantizes the input tensor using a fixed set of codebooks and returns corresponding codebook vectors. Args: hidden_state (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`): Input tensor to be quantized. n_quantizers (`int`, *optional*): Number of quantizers to use. If specified and `self.quantizer_dropout` is True, this argument is ignored during training, and a random number of quantizers is used. Returns: quantized_representation (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`): Quantized continuous representation of input. audio_codes (`torch.Tensor` of shape `(batch_size, num_codebooks, time_steps)`): Codebook indices for each codebook (quantized discrete representation of input). projected_latents (`torch.Tensor` of shape `(batch_size, num_codebooks * dimension, time_steps)`): Projected latents (continuous representation of input before quantization). commitment_loss (`torch.Tensor` of shape `(1)`): Commitment loss to train the encoder to predict vectors closer to codebook entries. codebook_loss (`torch.Tensor` of shape `(1)`): Codebook loss to update the codebook. rNrF)Z fill_valueÚdevice©Údim)rsZtrainingrr r(ÚrandintrXrtÚtoryÚ enumeraterwÚfullÚappendÚstackÚcat)r"r?rxr Zresidualr@rArrZdropoutZ n_dropoutrpÚ quantizerÚquantized_representation_iZcommitment_loss_iZcodebook_loss_iZ indices_iÚprojected_latents_iÚmaskrrrr-s: ÿ    z!DacResidualVectorQuantize.forwardrcCs|d}g}|jd}t|ƒD]&}|j| |dd…|dd…f¡ dd¡}| |¡||j| |¡7}q |tj|dd|fS)a– Reconstructs the continuous representation from quantized codes. Args: audio_codes (`torch.Tensor` of shape `(batch_size, num_codebooks, time_steps)`): Quantized discrete representation of input. Returns: quantized_representation (`torch.Tensor`): Quantized continuous representation of input. projected_latents (`torch.Tensor`): List of projected latents (continuous representations of input before quantization) for each codebook. audio_codes (`torch.Tensor`): Codebook indices for each codebook. grNr'rz) r(rvrwr9rHr€r7rr‚)r"rr rrsrpr…rrrÚ from_codesYs  * z$DacResidualVectorQuantize.from_codesÚlatentscCsèd}g}g}t dgdd„|jDƒ¡}tj|dd}t ||jdk¡djdddd}t|ƒD]8}||||d} } |j|  |dd…| | …dd…f¡\} } |  | ¡|  | ¡|j|  | ¡} || }q2|tj |ddfS) aReconstructs the quantized representation from unquantized latents. Args: latents (`torch.Tensor` of shape `(batch_size, total_latent_dimension, time_steps)`): Continuous representation of input after projection. Returns: quantized_representation (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`): Quantized representation of the full-projected space. quantized_latents (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`): Quantized representation of the latent space (continuous representation before quantization). rcSsg|]}|j‘qSr)r5)roÚqrrrrrƒsz:DacResidualVectorQuantize.from_latents..rzrT)ZaxisZkeepdimsN) rZtensorrwZcumsumÚnpÚwherer(rFrvr<r€r7r‚)r"rˆr Zquantized_latentsÚcodesZcodebook_dims_tensorÚdimsrsrpZ hidden_dim_jZ hidden_dim_kZquantized_latents_iZcodes_ir„rrrÚ from_latentsss & *   z&DacResidualVectorQuantize.from_latentsrf)rrrrr rrrXr-rÚTensorr‡rŽr.rrr$rrn s  @rncó.eZdZdZdef‡fdd„ Zdd„Z‡ZS)Ú DacDecoderz DAC Decoderr0c s¢tƒ ¡|j}|j}|j}tj||ddd|_g}t|ƒD] \}}|t |||ƒg7}qt  |¡|_ |jd|d}t |ƒ|_ tj|dddd|_t ¡|_dS)NrQr©r2rRr'r)rrr4rkZupsampling_ratiosrr3rTr~rjruÚblockrrSrVZTanhÚtanh) r"r0Z input_channelZchannelsÚstridesr“r\r[rmr$rrr–s   zDacDecoder.__init__cCs@| |¡}|jD]}||ƒ}q| |¡}| |¡}| |¡}|Srf)rTr“rSrVr”)r"r?Úlayerrrrr-«s      zDacDecoder.forward©rrrrr rr-r.rrr$rr‘“sr‘cr)Ú DacEncoderz DAC Encoderr0cs tƒ ¡|j}tjd|jddd|_g|_t|ƒD]\}}|d}|jt |||dg7_qt  |j¡|_|jd|}t |ƒ|_ tj||j ddd|_dS)NrrQrr’)r[r\r')rrZdownsampling_ratiosrr3r`rTr“r~rZrurrSr4rV)r"r0r•r\r[Zd_modelr$rrr»s  zDacEncoder.__init__cCs6| |¡}|jD]}||ƒ}q| |¡}| |¡}|Srf)rTr“rSrV)r"r?Úmodulerrrr-Ís     zDacEncoder.forwardr—rrr$rr˜¸sr˜c@s0eZdZeZdZdZdd„Zdd„Zdd„Z d S) ÚDacPreTrainedModelZdacÚ input_valuescCs6t|tjƒrtjj|jddtj |jd¡dSdS)Ng{®Gáz”?)Zstdr)Ú isinstancerr3ÚinitZ trunc_normal_rBZ constant_Zbias)r"r™rrrÚ _init_weightsßs þz DacPreTrainedModel._init_weightscCs6tjj}ttjjdƒrtjjj}|jjD] }||jƒ||jƒq||j j ƒ||j j ƒ|j j D]+}||j ƒ||j j ƒ||j j ƒ||jj ƒ||jj ƒ||jj ƒ||jj ƒq1||jj ƒ||jj ƒ|jj D]+}||jƒ||j j ƒ||j j ƒ||jj ƒ||jj ƒ||jj ƒ||jj ƒqmdS)NÚ weight_norm)rÚutilsrŸÚhasattrZparametrizationsrƒrwr6r7ÚencoderrTrVr“rarbrcÚdecoderrl)r"rŸr–rrrÚapply_weight_normäs6                      ùz$DacPreTrainedModel.apply_weight_normcCsf|jjD]}tj |j¡tj |j¡qtj |jj¡tj |jj ¡|jj D]9}tj |j¡tj |j j¡tj |j j ¡tj |j j¡tj |j j ¡tj |j j¡tj |j j ¡q)tj |jj¡tj |jj ¡|jj D]9}tj |j¡tj |j j¡tj |j j ¡tj |j j¡tj |j j ¡tj |j j¡tj |j j ¡qwdSrf)rƒrwrr Úremove_weight_normr6r7r¢rTrVr“rarbrcr£rl)r"r–rrrr¥s0   ùz%DacPreTrainedModel.remove_weight_normN) rrrr Z config_classZbase_model_prefixZmain_input_nameržr¤r¥rrrrršÙs !ršz/ The DAC (Descript Audio Codec) model. )Z custom_introc s¨eZdZdef‡fdd„ Ze  ddejdee dee fdd „ƒZ e   dd eejd eejdee fd d „ƒZ e  ddejdee dee fdd„ƒZ ‡ZS)ÚDacModelr0csjtƒ |¡||_t|ƒ|_t|ƒ|_t|ƒ|_t t   |jj ¡ƒ|_ d|j |jj kr/tdƒ‚| ¡dS)Nr'z'The codebook_size must be a power of 2.)rrr0r˜r¢r‘r£rnrƒrXrdÚlog2r8Zbits_per_codebookÚ ValueErrorZ post_initr:r$rrr)s     zDacModel.__init__Nr›rxÚ return_dictc Csj|dur|n|jj}| |¡}| ||¡\}}}}}|jj||jj|} |s.| |||fSt| |||ƒS)a input_values (`torch.Tensor of shape `(batch_size, 1, time_steps)`): Input audio data to encode, n_quantizers (int, *optional*): Number of quantizers to use. If None, all quantizers are used. Default is None. N)r0r©r¢rƒZcommitment_loss_weightZcodebook_loss_weightr) r"r›rxr©r rrr@rAr rrrÚencode9s ÿ zDacModel.encoder rcCsf|dur |dur tdƒ‚|dur|n|jj}|dur"|j |¡d}| |¡ d¡}|s/|fSt|ƒS)a% quantized_representation (torch.Tensor of shape `(batch_size, dimension, time_steps)`, *optional*): Quantized continuous representation of input. audio_codes (`torch.Tensor` of shape `(batch_size, num_codebooks, time_steps)`, *optional*): The codebook indices for each codebook, representing the quantized discrete representation of the input. This parameter should be provided if you want to decode directly from the audio codes (it will overwrite quantized_representation). NzDEither `quantized_representation` or `audio_codes` must be provided.rr)r¨r0r©rƒr‡r£Zsqueezer)r"r rr©r rrrÚdecodeTszDacModel.decodec Csv|dur|n|jj}|jd}|j||dd\}}}}|j|ddddd|…f} |s3|| |||fSt|| |||ƒS)a‹ input_values (`torch.Tensor` of shape `(batch_size, 1, time_steps)`): Audio data to encode. n_quantizers (`int`, *optional*): Number of quantizers to use. If `None`, all quantizers are used. Default is `None`. Examples: ```python >>> from datasets import load_dataset, Audio >>> from transformers import DacModel, AutoProcessor >>> librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> model = DacModel.from_pretrained("descript/dac_16khz") >>> processor = AutoProcessor.from_pretrained("descript/dac_16khz") >>> librispeech_dummy = librispeech_dummy.cast_column("audio", Audio(sampling_rate=processor.sampling_rate)) >>> audio_sample = librispeech_dummy[-1]["audio"]["array"] >>> inputs = processor(raw_audio=audio_sample, sampling_rate=processor.sampling_rate, return_tensors="pt") >>> encoder_outputs = model.encode(inputs["input_values"]) >>> # Get the intermediate audio codes >>> audio_codes = encoder_outputs.audio_codes >>> # Reconstruct the audio from its quantized representation >>> audio_values = model.decode(encoder_outputs.quantized_representation) >>> # or the equivalent with a forward pass >>> audio_values = model(inputs["input_values"]).audio_values ```Nr&F)r©r.)r0r©r(rªr«r ) r"r›rxr©Úlengthr r rrr rrrr-ss# ÿzDacModel.forward)NN)NNN)rrrr rrrrrrXÚboolrªr«r-r.rrr$rr¦#sBüþý üüþý üüþýür¦)$rrdÚ dataclassesrÚtypingrÚnumpyrŠrZtorch.nnrZtorch.nn.functionalZ functionalr=Zmodeling_utilsrr rrZconfiguration_dacr r rrÚModulerr/rLrZrjrnr‘r˜ršr¦Ú__all__rrrrÚsB      F% %!Iÿ {