o
Zh @ s d dl mZ d dlmZ d dlZd dlZddlmZm Z ddl
m Z ddl m
Z
mZ e eZeed d d
G d d d e
ZdS )
)UserDict)UnionN )add_end_docstringslogging ) ffmpeg_read)Pipelinebuild_pipeline_init_argsT)Zhas_feature_extractorZ
has_tokenizerc s` e Zd ZdZ fddZdeejee f f fdd Z
dd Z dd d Z d
d Z
dd Z ZS )#ZeroShotAudioClassificationPipelinea
Zero shot audio classification pipeline using `ClapModel`. This pipeline predicts the class of an audio when you
provide an audio and a set of `candidate_labels`.
The default `hypothesis_template` is : `"This is a sound of {}."`. Make sure you update it for your usage.
Example:
```python
>>> from transformers import pipeline
>>> from datasets import load_dataset
>>> dataset = load_dataset("ashraq/esc50")
>>> audio = next(iter(dataset["train"]["audio"]))["array"]
>>> classifier = pipeline(task="zero-shot-audio-classification", model="laion/clap-htsat-unfused")
>>> classifier(audio, candidate_labels=["Sound of a dog", "Sound of vaccum cleaner"])
[{'score': 0.9996, 'label': 'Sound of a dog'}, {'score': 0.0004, 'label': 'Sound of vaccum cleaner'}]
```
Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This audio
classification pipeline can currently be loaded from [`pipeline`] using the following task identifier:
`"zero-shot-audio-classification"`. See the list of available models on
[huggingface.co/models](https://huggingface.co/models?filter=zero-shot-audio-classification).
c s2 t jdi | | jdkrtd| j dd S )NptzThe z is only available in PyTorch. )super__init__ framework
ValueError __class__)selfkwargsr r
d/var/www/auris/lib/python3.10/site-packages/transformers/pipelines/zero_shot_audio_classification.pyr ? s
z,ZeroShotAudioClassificationPipeline.__init__audiosc s t j|fi |S )a
Assign labels to the audio(s) passed as inputs.
Args:
audios (`str`, `List[str]`, `np.array` or `List[np.array]`):
The pipeline handles three types of inputs:
- A string containing a http link pointing to an audio
- A string containing a local path to an audio
- An audio loaded in numpy
candidate_labels (`List[str]`):
The candidate labels for this audio. They will be formatted using *hypothesis_template*.
hypothesis_template (`str`, *optional*, defaults to `"This is a sound of {}"`):
The format used in conjunction with *candidate_labels* to attempt the audio classification by
replacing the placeholder with the candidate_labels. Pass "{}" if *candidate_labels* are
already formatted.
Return:
A list of dictionaries containing one entry per proposed label. Each dictionary contains the
following keys:
- **label** (`str`) -- One of the suggested *candidate_labels*.
- **score** (`float`) -- The score attributed by the model to that label. It is a value between
0 and 1, computed as the `softmax` of `logits_per_audio`.
)r __call__)r r r r r
r r F s z,ZeroShotAudioClassificationPipeline.__call__c K s6 i }d|v r |d |d< d|v r|d |d< |i i fS )Ncandidate_labelshypothesis_templater
)r r Zpreprocess_paramsr
r
r _sanitize_parameters_ s
z8ZeroShotAudioClassificationPipeline._sanitize_parametersNThis is a sound of {}.c s t |tr/|ds|drt|j}nt|d }| }W d n1 s*w Y t |tr;t || j
j }t |t j
sEtdt|jdkrPtd| j
|g| j
j dd}| jdkrf|| j}||d < fd
d |D }| j|| jd d
}|g|d< |S )Nzhttp://zhttps://rbz"We expect a numpy ndarray as inputr zNWe expect a single channel audio input for ZeroShotAudioClassificationPipeliner )
sampling_ratereturn_tensorsr c s g | ]} |qS r
)format).0xr r
r
s zBZeroShotAudioClassificationPipeline.preprocess..T)r padding text_inputs)
isinstancestr
startswithrequestsgetcontentopenreadbytesr Zfeature_extractorr npndarray TypeErrorlenshaper r toZ torch_dtypeZ tokenizer)r Zaudior r fZinputs sequencesr& r
r# r
preprocessh s,
z.ZeroShotAudioClassificationPipeline.preprocessc C s\ | d}| d}t|d tr|d }n|d d }| jdi ||}||jd}|S )Nr r& r )r logitsr
)popr' r modelZlogits_per_audio)r Z model_inputsr r& Zoutputs
model_outputsr
r
r _forward s
z,ZeroShotAudioClassificationPipeline._forwardc C sb | d}|d d }| jdkr|jdd}| }ntddd tt||d d
d D }|S ) Nr r9 r r )dimz `tf` framework not supported.c S s g | ] \}}||d qS ))scorelabelr
)r! r? Zcandidate_labelr
r
r r$ s zCZeroShotAudioClassificationPipeline.postprocess..c S s
| d S )Nr r
)r" r
r
r s
zAZeroShotAudioClassificationPipeline.postprocess..)key)r: r Zsoftmaxtolistr sortedzip)r r< r r9 ZprobsZscoresresultr
r
r postprocess s
z/ZeroShotAudioClassificationPipeline.postprocess)Nr )__name__
__module__ __qualname____doc__r r r0 r1 r/ r( r r r8 r= rG
__classcell__r
r
r r r s
r ) collectionsr typingr numpyr0 r* utilsr r Zaudio_classificationr baser r
Z
get_loggerrH loggerr r
r
r
r s