o
Zhz$ @ sr d dl mZmZ ddlmZ ddlmZ ddlmZ e r-d dl Z ddl
m Z dd l m
Z
d
ZG d d d eZdS )
)ListUnion )GenerationConfig)is_torch_available )PipelineN)%MODEL_FOR_TEXT_TO_SPECTROGRAM_MAPPING)SpeechT5HifiGanzmicrosoft/speecht5_hifiganc s| e Zd ZdZdZeddZddd fdd
Zd d
Zd d Z d
e
e e e f f fdd Z
dddZdd Z ZS )TextToAudioPipelinea
Text-to-audio generation pipeline using any `AutoModelForTextToWaveform` or `AutoModelForTextToSpectrogram`. This
pipeline generates an audio file from an input text and optional other conditional inputs.
Unless the model you're using explicitly sets these generation parameters in its configuration files
(`generation_config.json`), the following default values will be used:
- max_new_tokens: 256
Example:
```python
>>> from transformers import pipeline
>>> pipe = pipeline(model="suno/bark-small")
>>> output = pipe("Hey it's HuggingFace on the phone!")
>>> audio = output["audio"]
>>> sampling_rate = output["sampling_rate"]
```
Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial)
You can specify parameters passed to the model by using [`TextToAudioPipeline.__call__.forward_params`] or
[`TextToAudioPipeline.__call__.generate_kwargs`].
Example:
```python
>>> from transformers import pipeline
>>> music_generator = pipeline(task="text-to-audio", model="facebook/musicgen-small", framework="pt")
>>> # diversify the music generation by adding randomness with a high temperature and set a maximum music length
>>> generate_kwargs = {
... "do_sample": True,
... "temperature": 0.7,
... "max_new_tokens": 35,
... }
>>> outputs = music_generator("Techno music with high melodic riffs", generate_kwargs=generate_kwargs)
```
This pipeline can currently be loaded from [`pipeline`] using the following task identifiers: `"text-to-speech"` or
`"text-to-audio"`.
See the list of available models on [huggingface.co/models](https://huggingface.co/models?filter=text-to-speech).
T )Zmax_new_tokensN)vocoder
sampling_ratec s t j|i | | jdkrtdd | _| jjt v r.|d u r+t
t | jj
n|| _|| _| jd ur<| jjj| _| jd u rj| jj}| jjdd }|d urX|| dD ]}t||d }|d uri|| _qZd S d S )Ntfz5The TextToAudioPipeline is only available in PyTorch.generation_config)Z sample_rater )super__init__Z framework
ValueErrorr
model __class__r valuesr
Zfrom_pretrainedDEFAULT_VOCODER_IDtodevicer config__dict__getupdateto_dictgetattr)selfr
r argskwargsr Z
gen_configZsampling_rate_namer S/var/www/auris/lib/python3.10/site-packages/transformers/pipelines/text_to_audio.pyr Y s0
z TextToAudioPipeline.__init__c K sf t |tr|g}| jjjdkr$| jjddddddd}|| |}| j |fi |dd i}|S )
NZbarkZmax_input_semantic_lengthr FT
max_length)r& Zadd_special_tokensZreturn_attention_maskZreturn_token_type_idspaddingZreturn_tensorspt)
isinstancestrr r Z
model_typer Zsemantic_configr r tokenizer)r textr" Z
new_kwargsoutputr$ r$ r%
preprocessx s
z TextToAudioPipeline.preprocessc K s | j || jd}|d }|d }| j r7| j || jd}d|vr&| j|d< || | jjdi ||}nt|rDtd| | jdi ||d }| j
d urZ|
|}|S )N)r forward_paramsgenerate_kwargsr zYou're using the `TextToAudioPipeline` with a forward-only model, but `generate_kwargs` is non empty. For forward-only TTA models, please use `forward_params` instead of `generate_kwargs`. For reference, the `generate_kwargs` used here are: r r$ ) Z_ensure_tensor_on_devicer r Z can_generater r generatelenr keysr
)r Z model_inputsr" r/ r0 r- r$ r$ r% _forward s&
z TextToAudioPipeline._forward text_inputsc s t j|fi |S )a
Generates speech/audio from the inputs. See the [`TextToAudioPipeline`] documentation for more information.
Args:
text_inputs (`str` or `List[str]`):
The text(s) to generate.
forward_params (`dict`, *optional*):
Parameters passed to the model generation/forward method. `forward_params` are always passed to the
underlying model.
generate_kwargs (`dict`, *optional*):
The dictionary of ad-hoc parametrization of `generate_config` to be used for the generation call. For a
complete overview of generate, check the [following
guide](https://huggingface.co/docs/transformers/en/main_classes/text_generation). `generate_kwargs` are
only passed to the underlying model if the latter is a generative model.
Return:
A `dict` or a list of `dict`: The dictionaries have two keys:
- **audio** (`np.ndarray` of shape `(nb_channels, audio_length)`) -- The generated audio waveform.
- **sampling_rate** (`int`) -- The sampling rate of the generated audio waveform.
)r __call__)r r5 r/ r# r$ r% r6 s z TextToAudioPipeline.__call__c C sr t | dd d ur
| j|d< t | dd d ur | j|d< | j|d< |r#|ni |r(|ni d}|d u r2i }i }|||fS )Nassistant_modelassistant_tokenizerr+ )r/ r0 )r r7 r+ r8 )r Zpreprocess_paramsr/ r0 paramsZpostprocess_paramsr$ r$ r% _sanitize_parameters s
z(TextToAudioPipeline._sanitize_parametersc C sP i }t |tr |d }n t |tr|d }|jdtjd |d< | j|d< |S )Nwaveformr cpu)r ZdtypeZaudior )r) dicttupler torchfloatnumpyr )r r; Z output_dictr$ r$ r% postprocess s
z TextToAudioPipeline.postprocess)NNN)__name__
__module__ __qualname____doc__Z_pipeline_calls_generater Z_default_generation_configr r. r4 r r* r r6 r: rB
__classcell__r$ r$ r# r% r s 4 !
r )typingr r Z
generationr utilsr baser r? Zmodels.auto.modeling_autor Z!models.speecht5.modeling_speecht5r
r r r$ r$ r$ r% s