o ZhN@sdZddlZddlmZmZmZddlZddlZddlm Z m Z ddl m Z ddl mZddlmZmZmZmZmZmZdd lmZerMddlZddlZerTddlZer[ddlZeeZed d Gd d d e Zd gZ dS)z%Feature extractor class for Pop2PianoN)ListOptionalUnion)mel_filter_bank spectrogram)SequenceFeatureExtractor) BatchFeature) TensorTypeis_essentia_availableis_librosa_availableis_scipy_availableloggingrequires_backends)requires)essentialibrosascipyZtorch)backendscsTeZdZdZgdZ       d1d ed ed ed edededeffdd Zdej fddZ dej fddZ de j de j de j fddZ dej dej fdd Zd2d"ej fd#d$Z %d3d&ed'ed(ed)eeeeffd*d+Z ! , %d4deej eeeej eeefd eeeefded-eed(eed)eeeefd.efd/d0ZZS)5Pop2PianoFeatureExtractora  Constructs a Pop2Piano feature extractor. This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. This class extracts rhythm and preprocesses the audio before it is passed to the model. First the audio is passed to `RhythmExtractor2013` algorithm which extracts the beat_times, beat positions and estimates their confidence as well as tempo in bpm, then beat_times is interpolated and to get beatsteps. Later we calculate extrapolated_beatsteps from it to be used in tokenizer. On the other hand audio is resampled to self.sampling_rate and preprocessed and then log mel spectogram is computed from that to be used in our transformer model. Args: sampling_rate (`int`, *optional*, defaults to 22050): Target Sampling rate of audio signal. It's the sampling rate that we forward to the model. padding_value (`int`, *optional*, defaults to 0): Padding value used to pad the audio. Should correspond to silences. window_size (`int`, *optional*, defaults to 4096): Length of the window in samples to which the Fourier transform is applied. hop_length (`int`, *optional*, defaults to 1024): Step size between each window of the waveform, in samples. min_frequency (`float`, *optional*, defaults to 10.0): Lowest frequency that will be used in the log-mel spectrogram. feature_size (`int`, *optional*, defaults to 512): The feature dimension of the extracted features. num_bars (`int`, *optional*, defaults to 2): Determines interval between each sequence. input_features beatstepsextrapolated_beatstep"Vr$@ sampling_rate padding_value window_size hop_length min_frequency feature_sizenum_barsc sztjd|||d|||_||_||_||_||_||_||_t |jdd|j|jt |jd|jddd|_ dS)N)r%r r!rZhtk)Znum_frequency_binsZnum_mel_filtersr$Z max_frequencyr ZnormZ mel_scale) super__init__r r!r"r#r$r%r&rfloat mel_filters) selfr r!r"r#r$r%r&kwargs __class__r(i/var/www/auris/lib/python3.10/site-packages/transformers/models/pop2piano/feature_extraction_pop2piano.pyr*Ts.    z"Pop2PianoFeatureExtractor.__init__sequencec CsTg}|D]}t|jddd}|t|||j|jd|jdqt|}|S)z Generates MelSpectrogram. Args: sequence (`numpy.ndarray`): The sequence of which the mel-spectrogram will be computed. r'Ng@)ZwaveformwindowZ frame_lengthr#powerr,)npZhanningr"appendrr#r,array)r-r2 mel_specsseqr4r(r(r1mel_spectrogramvs z)Pop2PianoFeatureExtractor.mel_spectrogramaudiocCs:t|dgtjjdd}||\}}}}}|||||fS)a This algorithm(`RhythmExtractor2013`) extracts the beat positions and estimates their confidence as well as tempo in bpm for an audio signal. For more information please visit https://essentia.upf.edu/reference/std_RhythmExtractor2013.html . Args: audio(`numpy.ndarray`): raw audio waveform which is passed to the Rhythm Extractor. rZ multifeature)method)rrstandardZRhythmExtractor2013)r-r<Zessentia_trackerbpm beat_times confidence estimatesessentia_beat_intervalsr(r(r1extract_rhythms z(Pop2PianoFeatureExtractor.extract_rhythmr@steps_per_beatn_extendcCsRt|dgtjjt|j|ddd}|td|j|d|j||}|S)a This method takes beat_times and then interpolates that using `scipy.interpolate.interp1d` and the output is then used to convert raw audio to log-mel-spectrogram. Args: beat_times (`numpy.ndarray`): beat_times is passed into `scipy.interpolate.interp1d` for processing. steps_per_beat (`int`): used as an parameter to control the interpolation. n_extend (`int`): used as an parameter to control the interpolation. rFZ extrapolate)Z bounds_errorZ fill_valuerr')rrZ interpolateZinterp1dr6ZarangesizeZlinspace)r-r@rErFZbeat_times_functionZ ext_beatsr(r(r1interpolate_beat_timess   z0Pop2PianoFeatureExtractor.interpolate_beat_timesbeatstepcCs8|durt|jdkrtd|jd|ddkr ||d}|jd}t|}|j|d|jdddd}g}d}td||D]+}|} t|||} t|| |j} t|| |j} | | | ft || | }qBg} |D] \} } || | }t j |d||jdffd dd }| |qrt | } | |fS) a Preprocessing for log-mel-spectrogram Args: audio (`numpy.ndarray` of shape `(audio_length, )` ): Raw audio waveform to be processed. beatstep (`numpy.ndarray`): Interpolated values of the raw audio. If beatstep[0] is greater than 0.0, then it will be shifted by the value at beatstep[0]. Nr'zUExpected `audio` to be a single channel audio input of shape `(n, )` but found shape .rgr@rErFconstantZconstant_values)lenshape ValueErrorr&rHrangeminintr r7maxr6padZasarray)r-r<rIZ num_stepsZnum_target_stepsrZsample_indicesZmax_feature_lengthiZ start_idxZend_idx start_sample end_sampleZ padded_batchfeatureZpadded_featurer(r(r1preprocess_mels@         z(Pop2PianoFeatureExtractor.preprocess_melTfeaturescCsdd|D}gg}}t|D]\}}t|jdkrItgt|d||d}tj||ddtjd} dd|fdf} | d| df} n,|dd }tgt|d||d}tj||tjddd } dd|ff} } tj || d |j d } tj | | d |j d } |rtgt|d} tj | t d| |j ggdd } tj | tj d| g| jdgdd } || || qtj |dd tj}tj |dd tj}||fS) NcSsg|]}|jqSr()rP).0 each_featurer(r(r1 sz2Pop2PianoFeatureExtractor._pad..rr'r)dtype)rrrr3rMrN)Zaxis) enumeraterOrPrUzipr6ZonesZint64ZreshaperVr!Z concatenateZzerosr%r`r7astypefloat32)r-r\ add_zero_lineZfeatures_shapesZattention_masksZpadded_featuresrWr^Zfeatures_pad_valueattention_maskZfeature_paddingZattention_mask_paddingZeach_padded_featureZzero_array_lenr(r(r1_pads:       zPop2PianoFeatureExtractor._padNinputs is_batchedreturn_attention_maskreturn_tensorsc Csi}|D]2\}}|dkr"|j|dd\}} |||<|r!| |d<q|j|dd\}} |||<|r8| |d|<q|sI|sI|dddd f|d<t||d } | S) a Pads the inputs to same length and returns attention_mask. Args: inputs (`BatchFeature`): Processed audio features. is_batched (`bool`): Whether inputs are batched or not. return_attention_mask (`bool`): Whether to return attention mask or not. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. If nothing is specified, it will return list of `np.ndarray` arrays. Return: `BatchFeature` with attention_mask, attention_mask_beatsteps and attention_mask_extrapolated_beatstep added to it: - **attention_mask** numpy.ndarray of shape `(batch_size, max_input_features_seq_length)` -- Example : 1, 1, 1, 0, 0 (audio 1, also here it is padded to max length of 5 that's why there are 2 zeros at the end indicating they are padded) 0, 0, 0, 0, 0 (zero pad to separate audio 1 and 2) 1, 1, 1, 1, 1 (audio 2) 0, 0, 0, 0, 0 (zero pad to separate audio 2 and 3) 1, 1, 1, 1, 1 (audio 3) - **attention_mask_beatsteps** numpy.ndarray of shape `(batch_size, max_beatsteps_seq_length)` - **attention_mask_extrapolated_beatstep** numpy.ndarray of shape `(batch_size, max_extrapolated_beatstep_seq_length)` rT)rerfFZattention_mask_Nr3.)Z tensor_type)itemsrgr ) r-rhrirjrkZprocessed_features_dictZ feature_nameZ feature_valueZpadded_feature_valuesrfZoutputsr(r(r1rVs"* zPop2PianoFeatureExtractor.padFresamplereturncKst|dgtt|ttfot|dtjttf}|r2t|ts)td|d|dur/dn|}n|g}|g}|dur>dn|}ggg} } } t||D]\} } |j | d\}}}}}|j ||d d }|j | kr|j dur|r|t j j| | |j d d } n td |j d| d|j } t|d| }t|d| }|| ||||d\}}||tj}ttj|ddd}t|d}| || || |qMt| | | d}|j||||d}|S)a Main method to featurize and prepare for the model. Args: audio (`np.ndarray`, `List`): The audio or batch of audio to be processed. Each audio can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. sampling_rate (`int`): The sampling rate at which the `audio` input was sampled. It is strongly recommended to pass `sampling_rate` at the forward call to prevent silent errors. steps_per_beat (`int`, *optional*, defaults to 2): This is used in interpolating `beat_times`. resample (`bool`, *optional*, defaults to `True`): Determines whether to resample the audio to `sampling_rate` or not before processing. Must be True during inference. return_attention_mask (`bool` *optional*, defaults to `False`): Denotes if attention_mask for input_features, beatsteps and extrapolated_beatstep will be given as output or not. Automatically set to True for batched inputs. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. If nothing is specified, it will return list of `np.ndarray` arrays. rrzwPlease give sampling_rate of each audio separately when you are passing multiple raw_audios at the same time. Received z), expected [audio_1_sr, ..., audio_n_sr].NTF)r<r'rLZ kaiser_best)Zorig_srZ target_srZres_typezmThe sampling_rate of the provided audio is different from the target sampling_rate of the Feature Extractor, z vs zp. In these cases it is recommended to use `resample=True` in the `__call__` method to get the optimal behaviour.r3gư>)Za_minZa_max)rr3r)rirjrk)rbool isinstancelisttupler6ndarrayrQrbrDrHr rcorermwarningswarnrTr[r;rcrdlogZclipZ transposer7r rV)r-r<r rErmrjrkr.riZbatch_input_featuresZbatch_beatstepsZbatch_ext_beatstepZsingle_raw_audioZsingle_sampling_rater?r@rArBrCrrXrYrrr9Z log_mel_specsoutputr(r(r1__call__Xsv #(     z"Pop2PianoFeatureExtractor.__call__)rrrrrrr)T)N)rTFN)__name__ __module__ __qualname____doc__Zmodel_input_namesrTr+r*r6rtr;rDnumpyrHr[rgr rprrstrr rVrrz __classcell__r(r(r/r1r3s" 00 D" r)!r~rvtypingrrrrr6Z audio_utilsrrZ!feature_extraction_sequence_utilsrZfeature_extraction_utilsr utilsr r r r rrZutils.import_utilsrrZessentia.standardrrZ get_loggerr{loggerr__all__r(r(r(r1s0