o Zh*@sdZddlmZmZmZddlZddlmZm Z m Z ddl m Z ddl mZddlmZmZeeZGd d d e Zd gZdS) z" Feature extractor class for CLVP )ListOptionalUnionN)mel_filter_bank spectrogramwindow_function)SequenceFeatureExtractor) BatchFeature) TensorTypeloggingcseZdZdZddgZ      dfd d ZdejdejfddZ    dde eje e e eje e e fde edede ede e eefde ede ede edefddZZS) ClvpFeatureExtractora! Constructs a CLVP feature extractor. This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. This class extracts log-mel-spectrogram features from raw speech using a custom numpy implementation of the `Short Time Fourier Transform` which should match pytorch's `torch.stft` equivalent. Args: feature_size (`int`, *optional*, defaults to 80): The feature dimension of the extracted features. sampling_rate (`int`, *optional*, defaults to 22050): The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). default_audio_length (`int`, *optional*, defaults to 6): The default length of raw audio in seconds. If `max_length` is not set during `__call__` then it will automatically be set to default_audio_length * `self.sampling_rate`. hop_length (`int`, *optional*, defaults to 256): Length of the overlapping windows for the STFT used to obtain the Mel Frequency coefficients. chunk_length (`int`, *optional*, defaults to 30): The maximum number of chunks of `sampling_rate` samples used to trim and pad longer or shorter audio sequences. n_fft (`int`, *optional*, defaults to 1024): Size of the Fourier transform. padding_value (`float`, *optional*, defaults to 0.0): Padding value used to pad the audio. Should correspond to silences. mel_norms (`list` of length `feature_size`, *optional*): If `mel_norms` is provided then it will be used to normalize the log-mel spectrograms along each mel-filter. return_attention_mask (`bool`, *optional*, defaults to `False`): Whether to return the attention mask. If left to the default, it will return the attention mask. [What are attention masks?](../glossary#attention-mask) input_featuresZattention_maskP"VNFc sztjd |||| d| ||_||_||_|||_|j||_||_||_||_ t d|d|dd|ddd|_ dS) N) feature_size sampling_rate padding_valuereturn_attention_maskrg@@ZslaneyZhtk)Znum_frequency_binsZnum_mel_filtersZ min_frequencyZ max_frequencyrZnormZ mel_scale) super__init__n_fft hop_length chunk_lengthZ n_samplesZ nb_max_framesrdefault_audio_length mel_normsr mel_filters) selfrrr"r r!rrr#rkwargs __class__r_/var/www/auris/lib/python3.10/site-packages/transformers/models/clvp/feature_extraction_clvp.pyrGs2     zClvpFeatureExtractor.__init__waveformreturnc Csdt|t|jd|j|jd|jdd}ttj|ddd}|jdur0|t |jdddf}|S)z This method first computes the log-mel spectrogram of the provided audio then applies normalization along the each mel-filterbank, if `mel_norms` is provided. Zhanng@N)Z frame_lengthr powerr$Zlog_melgh㈵>)Za_minZa_max) rrrr r$nplogZclipr#array)r%r*Zlog_specrrr)_np_extract_fbank_featuresms  z/ClvpFeatureExtractor._np_extract_fbank_featuresT max_length raw_speechr truncationpad_to_multiple_ofreturn_tensorsrpaddingc  s|dur|jkrtdjjdjdjd|d n tdjjdt|tjo6t |j d k} | rGt |j d krGtd | pZt|t t foZt|d tjt t f} | red d|D}n&| svt|tjsvtj |tjd}nt|tjr|jttjur|tj}| st |gjg}td|i} |durjjn|}j| |||||d} | dd d d }fdd|d D}t|d trdd|D| d<n|| d<| |S)a `ClvpFeatureExtractor` is used to extract various voice specific properties such as the pitch and tone of the voice, speaking speed, and even speaking defects like a lisp or stuttering from a sample voice or `raw_speech`. First the voice is padded or truncated in a way such that it becomes a waveform of `self.default_audio_length` seconds long and then the log-mel spectrogram is extracted from it. Args: raw_speech (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`): The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep. sampling_rate (`int`, *optional*): The sampling rate at which the `raw_speech` input was sampled. It is strongly recommended to pass `sampling_rate` at the forward call to prevent silent errors and allow automatic speech recognition pipeline. truncation (`bool`, *optional*, default to `True`): Activates truncation to cut input sequences longer than *max_length* to *max_length*. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. return_attention_mask (`bool`, *optional*, defaults to `True`): Whether to return the attention mask. If left to the default, it will return the attention mask. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. padding_value (`float`, *optional*, defaults to 0.0): The value that is used to fill the padding values / vectors. max_length (`int`, *optional*): The maximum input length of the inputs. Nz3The model corresponding to this feature extractor: z& was trained using a sampling rate of zI. Please make sure that the provided `raw_speech` input was sampled with z and not .zDIt is strongly recommended to pass the `sampling_rate` argument to `zN()`. Failing to do so can result in silent errors that might be hard to debug.rrz2Only mono-channel audio is supported for input to rcSs g|] }tj|gtjdjqS)dtype)r-asarrayfloat32T).0Zspeechrrr) s z1ClvpFeatureExtractor.__call__..r8r)r6r1r3r4rcsg|] }|tjqSr)r0astyper-r;)r=r*r%rr)r>scSsg|]}t|qSr)r-r:)r=featurerrr)r>s)r ValueErrorr(__name__loggerwarning isinstancer-ndarraylenshapelisttupler:r;r9Zfloat64r?r<r r"padgetZ transposerZconvert_to_tensors)r%r2rr3r4r5rr6r1r&Zis_batched_numpyZ is_batchedZbatched_speechZ padded_inputsrrr@r)__call__s^3  "    zClvpFeatureExtractor.__call__) rrrrrrrNF)NTNNTr1N)rC __module__ __qualname____doc__Zmodel_input_namesrr-r/rGr0rrfloatrintboolstrr r rN __classcell__rrr'r)r !sP#&"  r )rQtypingrrrnumpyr-Z audio_utilsrrrZ!feature_extraction_sequence_utilsr Zfeature_extraction_utilsr utilsr r Z get_loggerrCrDr __all__rrrr)s    Q