o ZhW>@sddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z e r+ddlZe r2ddlZddlmZmZddlmZddlmZmZmZmZdd lmZmZGd d d ed d ZGddded d ZGdddeZ dgZ!dS)N)Path)AnyDictListOptionalUnion)is_soundfile_availableis_torch_available) AudioInputmake_list_of_audio) BatchFeature) AudioKwargsProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInputc@s"eZdZUeeeefed<dS)CsmAudioKwargsencoded_length_kwargsN)__name__ __module__ __qualname__rrstrr__annotations__rrU/var/www/auris/lib/python3.10/site-packages/transformers/models/csm/processing_csm.pyr(s rF)totalc@sJeZdZUeed<ddddgdgdgddd d d d d idZdS)CsmProcessorKwargs audio_kwargsTleftF)paddingZ padding_sideZadd_special_tokens)rrr# rr# rr#r)r#r#r#r(r#r#r#r#r#r#r$r#)r#r#r#r#r#r#r#r#r#r#r#r#r#r#r#) kernel_sizesstrides dilationsuse_causal_convi])r sampling_ratereturn_tensorspt) text_kwargsr common_kwargsN)rrrrr _defaultsrrrrr,s  rc seZdZdZddgZdgZdZdZ dfdd Ze dd d Z d e d e e eee e effdeefddZ   ddee eeeeeefd ee deedeedeef ddZZS) CsmProcessora  Constructs a Csm processor which wraps [`EncodecFeatureExtractor`] and [`PretrainedTokenizerFast`] into a single processor that inherits both the audio feature extraction and tokenizer functionalities. See the [`~CsmProcessor.__call__`] for more information. The preferred way of passing kwargs is as a dictionary per modality, see usage example below. ```python from transformers import CsmProcessor from datasets import load_dataset ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") audio = ds[0]["audio"]["array"] processor = CsmProcessor.from_pretrained("eustlb/csm-1b") processor( text=["<|begin_of_text|>[0]What are you working on?<|end_of_text|><|AUDIO|><|audio_eos|><|begin_of_text|>[1]I'm figuring out my budget.<|end_of_text|>"], audio=audio, text_kwargs = {"padding": False}, audio_kwargs = {"sampling_rate": 16000}, common_kwargs = {"return_tensors": "pt"}, ) # this should error out because EncodecFeatureExtractor expects a 24kHz audio :) ``` Args: feature_extractor ([`EncodecFeatureExtractor`]): The feature extractor is a required input. tokenizer ([`PreTrainedTokenizer`, `PreTrainedTokenizerFast`]): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. feature_extractor tokenizer chat_templateZEncodecFeatureExtractorZPreTrainedTokenizerFastNcsvt|dsd|_||j|_n|j|_|j|_t|ds(d|_||j|_n|j|_|j|_tj|||ddS)N audio_tokenz <|AUDIO|>audio_eos_tokenz <|audio_eos|>)r9)hasattrr:Zconvert_tokens_to_idsaudio_token_idr;Zaudio_eos_token_idsuper__init__)selfr7r8r9 __class__rrr?js  zCsmProcessor.__init__cCs|}|dus|dus|dus|dur|St|||D]W\}}}|d|d} ||} | d} | | } || | |d} t| d} | ||| }||}|rW| } |} n| } | |} || | }|||dd|d}q|S)a| Compute the length of the encoded audio sequence. Args: audio_length (int): The length of the audio sequence. kernel_sizes (List[int]): The kernel sizes for the convolutional layers. strides (List[int]): The strides for the convolutional layers. use_causal_conv (bool): Whether to use causal convolutions. Nr#r+)zipmathceil)Z audio_lengthr,r-r.r/Z cur_lengthZ kernel_sizeZstrideZdilationZeffective_kernel_sizeZ padding_total padding_right padding_leftZn_framesZ ideal_lengthZ extra_paddingrrr_get_encoded_lengths(  z CsmProcessor._get_encoded_lengthaudio saving_pathkwargsc Kststdt|}t|ttfr|g}nt|ttfr&tdd|Ds*t dt |t |kr6t d|j t fi|}|d}|d}t ||D]\}}t|tjr^|}t|||qLdS)Nz/Please install `soundfile` to save audio files.css|] }t|ttfVqdSN) isinstancerr).0prrr sz*CsmProcessor.save_audio..zAInvalid input path. Please provide a string, or a list of stringsz5The number of audio and saving paths must be the samerr0)r ImportErrorr rMrrlisttupleall ValueErrorlen _merge_kwargsrrCtorchTensorcpufloatnumpysfwrite) r@rIrJrK output_kwargsrr0Z audio_valuerOrrr save_audios*  zCsmProcessor.save_audioF?text output_labelsdepth_decoder_labels_ratioc sjtfdjji|}|d}|d}|d} | dd} | dkr,tjjdt|t r5|g}nt|t t frEt d d |DsItd fd d |D} d} |dur`t |}t|} t| dkr| t| kr|durttdtd| d| d|dur|difdd |D} | }g}|D]<}g}j|vr|d}j|}|||jdd}j|vsd|vr|d|dd}d|vs||q|}j|fi|}i}|||durw|ddgg}}d}| D]I} | dkr|td|tdgq|tjdd |||| Ddd|tdd |||| Djdd|| 7}qj|fi|}|dd||tdd |Dfdd |D}tj|dd|d <|r|d!jk}|j d}|d"krt!|dt"|d|}||}n|}t#|d!jk|d!d#}d$||dddf|dddff<||d%<t$|| d&S)'a Main method to prepare text(s) and audio to be fed as input to the model. This method forwards the `text` arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizerFast.__call__`] to encode the text. To prepare the audio, this method forwards the `audio` arguments to EncodecFeatureExtractor's [`~EncodecFeatureExtractor.__call__`]. Please refer to the docstring of the above two methods for more information. Args: audio (`np.ndarray`, `torch.Tensor`, `List[np.ndarray]`, `List[torch.Tensor]`): The audio or batch of audio to be prepared. Each audio can be a NumPy array or PyTorch tensor. text (`str`, `List[str]`, `List[List[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). output_labels (bool, *optional*, default=False): Whether to return labels for training. Indices will be in `[config.audio_token_id, -100, -101]`. - `config.audio_token_id` indicates an audio frame (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) depth_decoder_labels_ratio (float, *optional*, default=1.0): The ratio of audio frames to keep for the depth decoder labels. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **input_values** -- List of audio values to be fed to a model. Returned when `audio` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **labels** -- List of labels for the audio frames. Returned when `output_labels=True`. Ztokenizer_init_kwargsr3rr4r1Nr2z% only supports `return_tensors='pt'`.css|]}t|tVqdSrL)rMrrNtrrrrPz(CsmProcessor.__call__..zAInvalid input text. Please provide a string, or a list of stringscsg|]}|jqSr)countr:re)r@rr sz)CsmProcessor.__call__..rz@No audio were provided, but there are audio tokens in the promptz)The number of audio tokens in each text (z7) should be the same as the number of provided audios (z).rcs$g|]}j|jdfiqS)rHshape)rNZ audio_array)rr@rrrisz r#Zreturn_attention_maskrkcSs(g|]}t|tjr|n|qSr)rMrXrYrZr\rNelrrrriBs)ZaxiscSsg|]}|jdqSrjrlrmrrrriJs)dimZ padding_maskcss|]}|jdVqdS)rkNrorNZcut_idxsrrrrPSrgcs.g|]}tjjj|d|jdfddqS)rrk)value)rXnnZ functionalpadrlrq)max_lenrrriTs input_values_cutoffsZ input_idsraiilabels)dataZ tensor_type)%rWrr8Z init_kwargspoprUrBrrMrrRrSrTr rVsumcopyr:appendreplaceupdatenpZzerosrXZtensorZ concatenateZcumsumr7maxstackr=ZnonzerorlZrandpermintwherer )r@rbrIrcrdrKr_r3rr4r1Zn_audio_in_textZn_audioZnum_audio_tokens_listZnum_audio_tokens_list_copyZ expanded_textsampleZ replace_strZnum_audio_tokensZexpanded_audio_tokenencodingrxZconcatenated_audiorvoffsetZ audio_inputsZaudio_frame_idxsZn_audio_framesZ rand_idxsZskip_frames_idxsrwr)rrur@r__call__s/                 &       $ zCsmProcessor.__call__rL)NNNN)NFra)rrr__doc__ attributesZ valid_kwargsZfeature_extractor_classZtokenizer_classr? staticmethodrHr rrrrrrr`rrrboolr[r __classcell__rrrArr6As># & %r6)"rDpathlibrtypingrrrrrr\rutilsr r rXZ soundfiler]Z audio_utilsr r Zfeature_extraction_utilsr Zprocessing_utilsrrrrZtokenization_utils_baserrrrr6__all__rrrrs&   -