o ZhE@sdZddlZddlmZmZmZmZmZddlZ ddl m Z m Z m Z mZddlmZddlmZddlmZmZmZeeZGd d d eZd gZdS) z%Feature extractor class for SpeechT5.N)AnyDictListOptionalUnion)mel_filter_bankoptimal_fft_length spectrogramwindow_function)SequenceFeatureExtractor) BatchFeature)PaddingStrategy TensorTypeloggingcseZdZdZddgZ         d7dedededededededededededededeffdd Z e d8de e j de e j ded!e e j fd"d#Zd$e j d!e j fd%d&Z ' '  '  ' ' ' 'd9d(eee j e ee e j e e efd)eee j e ee e j e e efd*eeeefd+eed,ed-eedeed.eeeefdeed!efd/d0Z   '  ' ' 'd:d1ee j e ee e j e e efd2ed*eeeefd+eed,ed-eedeed.eeeefd!efd3d4Zd!eeefffd5d6 ZZS);SpeechT5FeatureExtractora Constructs a SpeechT5 feature extractor. This class can pre-process a raw speech signal by (optionally) normalizing to zero-mean unit-variance, for use by the SpeechT5 speech encoder prenet. This class can also extract log-mel filter bank features from raw speech, for use by the SpeechT5 speech decoder prenet. This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: feature_size (`int`, *optional*, defaults to 1): The feature dimension of the extracted features. sampling_rate (`int`, *optional*, defaults to 16000): The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). padding_value (`float`, *optional*, defaults to 0.0): The value that is used to fill the padding values. do_normalize (`bool`, *optional*, defaults to `False`): Whether or not to zero-mean unit-variance normalize the input. Normalizing can help to significantly improve the performance for some models. num_mel_bins (`int`, *optional*, defaults to 80): The number of mel-frequency bins in the extracted spectrogram features. hop_length (`int`, *optional*, defaults to 16): Number of ms between windows. Otherwise referred to as "shift" in many papers. win_length (`int`, *optional*, defaults to 64): Number of ms per window. win_function (`str`, *optional*, defaults to `"hann_window"`): Name for the window function used for windowing, must be accessible via `torch.{win_function}` frame_signal_scale (`float`, *optional*, defaults to 1.0): Constant multiplied in creating the frames before applying DFT. This argument is deprecated. fmin (`float`, *optional*, defaults to 80): Minimum mel frequency in Hz. fmax (`float`, *optional*, defaults to 7600): Maximum mel frequency in Hz. mel_floor (`float`, *optional*, defaults to 1e-10): Minimum value of mel frequency banks. reduction_factor (`int`, *optional*, defaults to 2): Spectrogram length reduction factor. This argument is deprecated. return_attention_mask (`bool`, *optional*, defaults to `True`): Whether or not [`~SpeechT5FeatureExtractor.__call__`] should return `attention_mask`. input_valuesattention_mask>FP@ hann_window?绽|=T feature_size sampling_rate padding_value do_normalize num_mel_bins hop_length win_length win_functionframe_signal_scalefminfmax mel_floorreduction_factorreturn_attention_maskc stjd |||d|||_||_||_||_||_||_| |_| |_ | |_ | |_ | |_ ||d|_ ||d|_t|j |_|jdd|_t|j |jdd|_t|j|j|j |j |jddd|_| d krotd t| d kr{td tdSdS)N)rr r!irrT)Z window_lengthnameZperiodicZslaney)Znum_frequency_binsZnum_mel_filtersZ min_frequencyZ max_frequencyr ZnormZ mel_scalerzeThe argument `frame_signal_scale` is deprecated and will be removed in version 4.30.0 of Transformersg@zcThe argument `reduction_factor` is deprecated and will be removed in version 4.30.0 of Transformers)super__init__r"r,r#r$r%r&r'r(r)r*r+ sample_size sample_strider n_fftn_freqsr windowrr mel_filterswarningswarn FutureWarning)selfrr r!r"r#r$r%r&r'r(r)r*r+r,kwargs __class__r.g/var/www/auris/lib/python3.10/site-packages/transformers/models/speecht5/feature_extraction_speecht5.pyr0NsJ  z!SpeechT5FeatureExtractor.__init__returncCs|durEt|tj}g}t||dD]-\}}||d|t|d|d}||jdkr=|||d<| |q|Sdd|D}|S)z[ Every array in the list is normalized to have zero mean and unit variance NHz>rcSs*g|]}||t|dqS)rA)meannpsqrtvar).0xr.r.r> s*zDSpeechT5FeatureExtractor.zero_mean_unit_var_norm..) rCarrayint32zipsumrBrDrEshapeappend)rrr!Znormed_input_valuesZvectorlengthZ normed_slicer.r.r>zero_mean_unit_var_norms.  z0SpeechT5FeatureExtractor.zero_mean_unit_var_norm one_waveformc Cs*t||j|j|j|j|j|jdd}|jS)zZ Extracts log-mel filterbank features for one waveform array (unbatched). log10)r5Z frame_lengthr$Z fft_lengthr6r*Zlog_mel)r r5r1r2r3r6r*T)r:rQZ log_mel_specr.r.r>_extract_mel_featuress z.SpeechT5FeatureExtractor._extract_mel_featuresNaudio audio_targetpadding max_length truncationpad_to_multiple_ofreturn_tensorsc  Ks|dur |dur td| dur)| |jkr(td|d|jd|jd| d n td|jjd |durI|j|d ||||||fi| } nd} |durx|j|d ||||||fi| } | dure| S| d | d <| d} | durx| | d<| S)aA Main method to featurize and prepare for the model one or several sequence(s). Pass in a value for `audio` to extract waveform features. Pass in a value for `audio_target` to extract log-mel spectrogram features. Args: audio (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`, *optional*): The sequence or batch of sequences to be processed. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. This outputs waveform features. Must be mono channel audio, not stereo, i.e. single float per timestep. audio_target (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`, *optional*): The sequence or batch of sequences to be processed as targets. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. This outputs log-mel spectrogram features. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). truncation (`bool`): Activates truncation to cut input sequences longer than *max_length* to *max_length*. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific feature_extractor's default. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. sampling_rate (`int`, *optional*): The sampling rate at which the `audio` or `audio_target` input was sampled. It is strongly recommended to pass `sampling_rate` at the forward call to prevent silent errors. Nz9You must provide either `audio` or `audio_target` values.z3The model corresponding to this feature extractor: z& was trained using a sampling rate of zB. Please make sure that the provided audio input was sampled with z and not .zDIt is strongly recommended to pass the `sampling_rate` argument to `zN()`. Failing to do so can result in silent errors that might be hard to debug.FTrlabelsrdecoder_attention_mask) ValueErrorr loggerwarningr=__name___process_audioget)r:rUrVrWrXrYrZr,r[r r;ZinputsZ inputs_targetr^r.r.r>__call__sf?       z!SpeechT5FeatureExtractor.__call__speech is_targetc  sPt|tjo t|jdk} | rt|jdkrtd| p0t|ttfo0t|dtjttf} | r;dd|D}n&| sLt|tjsLtj|tj d}nt|tjra|j t tj ura| tj }| sf|g}j } |rfdd|D} td | i}j_ ntd |i}j|f|||||d | }| _ |d }t|dtjsd d|D|d <n:t|tjst|dtjr|dj t tj urd d|D|d <nt|tjr|j t tj ur| tj |d <|d }|durdd|D|d <|sjrj||dtjur |nd}j|d |jd|d <|dur&||}|S)Nrrz2Only mono-channel audio is supported for input to rcSg|] }tj|tjdqSdtyperCasarrayfloat32)rFrfr.r.r>rH=z;SpeechT5FeatureExtractor._process_audio..rjcsg|]}|qSr.)rT)rFZwaveformr:r.r>rHLsr)rWrXrYrZr,cSrhrirlrFrIr.r.r>rHarocSsg|]}|tjqSr.)astyperCrnrqr.r.r>rHgsrcSrhri)rCrmrJrqr.r.r>rHnro)rX)rr!) isinstancerCndarraylenrMr_listtuplermrnrkZfloat64rrrr r#padrdr"Z_get_padding_strategiesrZ DO_NOT_PADrPr!Zconvert_to_tensors)r:rfrgrWrXrYrZr,r[r;Zis_batched_numpyZ is_batchedZfeature_size_hackfeaturesZencoded_inputsZ padded_inputsrrr.rpr>rc)sr "          z'SpeechT5FeatureExtractor._process_audiocs.t}gd}|D] }||vr||=q |S)N)r5r6r1r2r3r4)r/to_dict)r:outputnamesr-r<r.r>rzs z SpeechT5FeatureExtractor.to_dict)rrrFrrrrrrrrrT)r) NNFNFNNNN)FFNFNNN)rb __module__ __qualname____doc__Zmodel_input_namesintfloatboolstrr0 staticmethodrrCrtrPrTrrrrr rercrrrz __classcell__r.r.r<r>rs,     <  &&    x"   "Wr)rr7typingrrrrrnumpyrCZ audio_utilsrr r r Z!feature_extraction_sequence_utilsr Zfeature_extraction_utilsr utilsrrrZ get_loggerrbr`r__all__r.r.r.r>s    o