o
ZhK @ s d dl Z d dlZd dlZd dlZd dlmZmZmZmZm Z m
Z
d dl Z ddl
mZ ddlmZmZ ddlmZ ddlmZmZmZ ddlmZmZ dd lm Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z* dd
l+m,Z, dd l-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3 e) rdd lm4Z4 e& rd dl5Z5e' rdd
lm6Z6 e( rd dl7m8Z9 nd dl:m8Z9 e*;e<Z=dZ>e de>e,ddG dd deZ?e"e?j@e?_@e?j@jAdure?j@jAjBdddde?j@_AdS dS ) N)AnyDictListOptionalTupleUnion )custom_object_save) BatchFeature
get_size_dict)BaseImageProcessorFast)ChannelDimensionSizeDictvalidate_kwargs)Unpack VideosKwargs)VIDEO_PROCESSOR_NAME
TensorTypeadd_model_info_to_auto_map"add_model_info_to_custom_pipelinesadd_start_docstrings cached_file copy_func download_urlis_offline_mode
is_remote_urlis_torch_availableis_torchvision_availableis_torchvision_v2_availableis_vision_availablelogging)requires)
VideoInputgroup_videos_by_shape
load_videomake_batched_videosreorder_videosto_channel_dimension_format)PILImageResampling) pil_torch_interpolation_mapping)
functionalac
Args:
do_resize (`bool`, *optional*, defaults to `self.do_resize`):
Whether to resize the video's (height, width) dimensions to the specified `size`. Can be overridden by the
`do_resize` parameter in the `preprocess` method.
size (`dict`, *optional*, defaults to `self.size`):
Size of the output video after resizing. Can be overridden by the `size` parameter in the `preprocess`
method.
size_divisor (`int`, *optional*, defaults to `self.size_divisor`):
The size by which to make sure both the height and width can be divided.
default_to_square (`bool`, *optional*, defaults to `self.default_to_square`):
Whether to default to a square video when resizing, if size is an int.
resample (`PILImageResampling`, *optional*, defaults to `self.resample`):
Resampling filter to use if resizing the video. Only has an effect if `do_resize` is set to `True`. Can be
overridden by the `resample` parameter in the `preprocess` method.
do_center_crop (`bool`, *optional*, defaults to `self.do_center_crop`):
Whether to center crop the video to the specified `crop_size`. Can be overridden by `do_center_crop` in the
`preprocess` method.
do_pad (`bool`, *optional*):
Whether to pad the video to the `(max_height, max_width)` of the videos in the batch.
crop_size (`Dict[str, int]` *optional*, defaults to `self.crop_size`):
Size of the output video after applying `center_crop`. Can be overridden by `crop_size` in the `preprocess`
method.
do_rescale (`bool`, *optional*, defaults to `self.do_rescale`):
Whether to rescale the video by the specified scale `rescale_factor`. Can be overridden by the
`do_rescale` parameter in the `preprocess` method.
rescale_factor (`int` or `float`, *optional*, defaults to `self.rescale_factor`):
Scale factor to use if rescaling the video. Only has an effect if `do_rescale` is set to `True`. Can be
overridden by the `rescale_factor` parameter in the `preprocess` method.
do_normalize (`bool`, *optional*, defaults to `self.do_normalize`):
Whether to normalize the video. Can be overridden by the `do_normalize` parameter in the `preprocess`
method. Can be overridden by the `do_normalize` parameter in the `preprocess` method.
image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`):
Mean to use if normalizing the video. This is a float or list of floats the length of the number of
channels in the video. Can be overridden by the `image_mean` parameter in the `preprocess` method. Can be
overridden by the `image_mean` parameter in the `preprocess` method.
image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`):
Standard deviation to use if normalizing the video. This is a float or list of floats the length of the
number of channels in the video. Can be overridden by the `image_std` parameter in the `preprocess` method.
Can be overridden by the `image_std` parameter in the `preprocess` method.
do_convert_rgb (`bool`, *optional*, defaults to `self.image_std`):
Whether to convert the video to RGB.
return_tensors (`str` or `TensorType`, *optional*):
Returns stacked tensors if set to `pt, otherwise returns a list of tensors.
data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`):
The channel dimension format for the output video. Can be one of:
- `"channels_first"` or `ChannelDimension.FIRST`: video in (num_channels, height, width) format.
- `"channels_last"` or `ChannelDimension.LAST`: video in (height, width, num_channels) format.
- Unset: Use the channel dimension format of the input video.
input_data_format (`ChannelDimension` or `str`, *optional*):
The channel dimension format for the input video. If unset, the channel dimension format is inferred
from the input video. Can be one of:
- `"channels_first"` or `ChannelDimension.FIRST`: video in (num_channels, height, width) format.
- `"channels_last"` or `ChannelDimension.LAST`: video in (height, width, num_channels) format.
- `"none"` or `ChannelDimension.NONE`: video in (height, width) format.
device (`torch.device`, *optional*):
The device to process the videos on. If unset, the device is inferred from the input videos.z!Constructs a base VideoProcessor.)ZvisionZ torchvision)backendsc " s e Zd ZdZdZdZdZdZdZdZ dZ
dZ dZ dZ
dZdZdZdZeZdgZdee ddf fdd Zdefd d
Zd d defd
dZ dMdedee e e f ded de d fddZ e!e"dedee defddZ# dNde d de$de$de%dee& ded d e$d e%d e$d e$d!e'd"e$d#ee e'e e' f d$ee e'e e' f d%ee e e(f def d&d'Z)e* ( ( )dOd*e e e+j,f d+ee e e+j,f d,e$d-e$d.ee e e$f d/e f d0d1Z-dPd2e e e+j,f d3e$fd4d5Z.e*d*e e e+j,f de/e0e e1f e0e e1f f fd6d7Z2e*d8e0e e1f fd9d:Z3de0e e1f fd;d<Z4de fd=d>Z5d?e e e+j,f fd@dAZ6dBdC Z7e*dDe e e+j,f fdEdFZ8e*dQdHdIZ9dJe e e e f fdKdLZ: Z;S )RBaseVideoProcessorNTgp?pixel_values_videoskwargsreturnc s" t |dd | _| D ](\}}zt| || W q ty8 } ztd| d| d| |d }~ww |d| j }|d urOt
||d| j dnd | _ |d| j }|d urct
|dd nd | _ t
| jj | _| jD ] }||d urt| ||| qrt| |t| |d qrd S )
Nprocessor_classz
Can't set z with value z for sizedefault_to_square)r1 r2 crop_size)
param_name)super__init__pop_processor_classitemssetattrAttributeErrorloggererrorr1 r r2 r3 list valid_kwargs__annotations__keysZmodel_valid_processing_keysgetgetattr)selfr. keyvalueerrr1 r3 __class__ R/var/www/auris/lib/python3.10/site-packages/transformers/video_processing_utils.pyr6 s.
zBaseVideoProcessor.__init__c K s | j |fi |S N)
preprocess)rD videosr. rJ rJ rK __call__ s zBaseVideoProcessor.__call__videoz torch.Tensorc C s t |}|jd dks |ddddddf dk s |S |ddddddf d }d|ddddddf d |ddddddf |dddddddf }|S )z
Converts a video to RGB format.
Args:
video (`"torch.Tensor"`):
The video to convert.
Returns:
`torch.Tensor`: The converted video.
.N g o@r )FZgrayscale_to_rgbshapeany)rD rP alpharJ rJ rK convert_to_rgb s
. Tz!BaseVideoProcessor.convert_to_rgbrN input_data_formatdevicez torch.devicec C s^ t |}g }|D ]$}t|tjr t|tj|}t| }|dur'|
|}| | q|S )z:
Prepare the input videos for processing.
N) r%
isinstancenpndarrayr' r
ZFIRSTtorchZ
from_numpy
contiguoustoappend)rD rN rY rZ processed_videosrP rJ rJ rK _prepare_input_videos s
z(BaseVideoProcessor._prepare_input_videosc K s t | | jj d | jjD ] }||t| |d q|d}|d}| j|||d}| jd
i |}| j d
i | |d}t
|t t frOt
| n||d< |d |d | jd
d |i|S ) N)Zcaptured_kwargsZvalid_processor_keysrY rZ )rN rY rZ resample
interpolationr2 Z data_formatrN rJ )r rA r? r@
setdefaultrC r7 rc Z_further_process_kwargsZ_validate_preprocess_kwargsr[ r( intr) _preprocess)rD rN r. Z
kwarg_namerY rZ rd rJ rJ rK rM s
z BaseVideoProcessor.preprocessdo_convert_rgb do_resizer1 size_divisorre zF.InterpolationModedo_center_cropr3
do_rescaledo_padrescale_factor do_normalize
image_mean image_stdreturn_tensorsc C s t |\}}i }| D ]\}}|r| |}|r"| j||||d}|||< q t||}t |\}}i }| D ]\}}|rD| ||}| || | | |
|}|||< q8t||}|ratj|ddn|}t d|i|dS )N)r1 rk re r )dimr- )dataZ tensor_type)
r# r9 rX resizer& Z center_cropZrescale_and_normalizer^ stackr
)rD rN ri rj r1 rk re rl r3 rm rn ro rp rq rr rs Zgrouped_videosZgrouped_videos_indexZresized_videos_groupedrU Zstacked_videosZresized_videosZprocessed_videos_groupedrb rJ rJ rK rh s.
z BaseVideoProcessor._preprocessFmain pretrained_model_name_or_path cache_dirforce_downloadlocal_files_onlytokenrevisionc
K s ||d< ||d< ||d< ||d< | dd}|dur*tdt |dur(td|}|dur2||d < | j|fi |\} }| j| fi |S )
a
Instantiate a type of [`~video_processing_utils.VideoProcessorBase`] from an video processor.
Args:
pretrained_model_name_or_path (`str` or `os.PathLike`):
This can be either:
- a string, the *model id* of a pretrained video hosted inside a model repo on
huggingface.co.
- a path to a *directory* containing a video processor file saved using the
[`~video_processing_utils.VideoProcessorBase.save_pretrained`] method, e.g.,
`./my_model_directory/`.
- a path or url to a saved video processor JSON *file*, e.g.,
`./my_model_directory/preprocessor_config.json`.
cache_dir (`str` or `os.PathLike`, *optional*):
Path to a directory in which a downloaded pretrained model video processor should be cached if the
standard cache should not be used.
force_download (`bool`, *optional*, defaults to `False`):
Whether or not to force to (re-)download the video processor files and override the cached versions if
they exist.
resume_download:
Deprecated and ignored. All downloads are now resumed by default when possible.
Will be removed in v5 of Transformers.
proxies (`Dict[str, str]`, *optional*):
A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',
'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request.
token (`str` or `bool`, *optional*):
The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use
the token generated when running `huggingface-cli login` (stored in `~/.huggingface`).
revision (`str`, *optional*, defaults to `"main"`):
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
identifier allowed by git.
To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`.
return_unused_kwargs (`bool`, *optional*, defaults to `False`):
If `False`, then this function returns just the final video processor object. If `True`, then this
functions returns a `Tuple(video_processor, unused_kwargs)` where *unused_kwargs* is a dictionary
consisting of the key/value pairs whose keys are not video processor attributes: i.e., the part of
`kwargs` which has not been used to update `video_processor` and is otherwise ignored.
subfolder (`str`, *optional*, defaults to `""`):
In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can
specify the folder name here.
kwargs (`Dict[str, Any]`, *optional*):
The values in kwargs of any keys which are video processor attributes will be used to override the
loaded values. Behavior concerning key/value pairs whose keys are *not* video processor attributes is
controlled by the `return_unused_kwargs` keyword parameter.
Returns:
A video processor of type [`~video_processing_utils.ImagVideoProcessorBase`].
Examples:
```python
# We can't instantiate directly the base class *VideoProcessorBase* so let's show the examples on a
# derived class: *LlavaOnevisionVideoProcessor*
video_processor = LlavaOnevisionVideoProcessor.from_pretrained(
"llava-hf/llava-onevision-qwen2-0.5b-ov-hf"
) # Download video_processing_config from huggingface.co and cache.
video_processor = LlavaOnevisionVideoProcessor.from_pretrained(
"./test/saved_model/"
) # E.g. video processor (or model) was saved using *save_pretrained('./test/saved_model/')*
video_processor = LlavaOnevisionVideoProcessor.from_pretrained("./test/saved_model/preprocessor_config.json")
video_processor = LlavaOnevisionVideoProcessor.from_pretrained(
"llava-hf/llava-onevision-qwen2-0.5b-ov-hf", do_normalize=False, foo=False
)
assert video_processor.do_normalize is False
video_processor, unused_kwargs = LlavaOnevisionVideoProcessor.from_pretrained(
"llava-hf/llava-onevision-qwen2-0.5b-ov-hf", do_normalize=False, foo=False, return_unused_kwargs=True
)
assert video_processor.do_normalize is False
assert unused_kwargs == {"foo": False}
```rz r{ r| r~ use_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`.r} )r7 warningswarn
FutureWarning
ValueErrorget_video_processor_dict from_dict)
clsry rz r{ r| r} r~ r. r video_processor_dictrJ rJ rK from_pretrainedI s& Z z"BaseVideoProcessor.from_pretrainedsave_directory push_to_hubc K s | dd}|dur tdt |dddur td||d< tj|r.t d| dtj
|dd |rX| d
d}| d | tjj d }| j
|fi |}| |}| jdurdt| || d
tj|t}| | td| |r| j|||||dd |gS )aq
Save an video processor object to the directory `save_directory`, so that it can be re-loaded using the
[`~video_processing_utils.VideoProcessorBase.from_pretrained`] class method.
Args:
save_directory (`str` or `os.PathLike`):
Directory where the video processor JSON file will be saved (will be created if it does not exist).
push_to_hub (`bool`, *optional*, defaults to `False`):
Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the
repository you want to push to with `repo_id` (will default to the name of `save_directory` in your
namespace).
kwargs (`Dict[str, Any]`, *optional*):
Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method.
r Nr r} r zProvided path (z#) should be a directory, not a fileT)exist_okcommit_messagerepo_id)configzVideo processor saved in )r r} )r7 r r r rB r ospathisfileAssertionErrormakedirssplitsepZ _create_repoZ_get_files_timestamps _auto_classr joinr to_json_filer< infoZ_upload_modified_files) rD r r r. r r r Zfiles_timestampsZoutput_video_processor_filerJ rJ rK save_pretrained sB
z"BaseVideoProcessor.save_pretrainedc K s | dd}| dd}| dd}| dd}| dd}| dd}| d d} | d
d}
| d d } | d
d} | dd}
|durVtdt |durTtd|}d|
d}| durc| |d< t ro| sotd d} t|}t j
|}t j
|r|}d}nYt
|r|}t|}nNzt}t||||||| |||
| d }W n: ty d}t||||||| |||
| d }td Y n ty ty td| d| dt d w z"t|d d d }| }W d n1 sw Y t|}W n tjy td | d!w |r td"| n td"| d#| |sGd$|v r9t|d$ ||d$< d%|v rGt|d% ||d%< ||fS )&a
From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a
video processor of type [`~video_processing_utils.VideoProcessorBase`] using `from_dict`.
Parameters:
pretrained_model_name_or_path (`str` or `os.PathLike`):
The identifier of the pre-trained checkpoint from which we want the dictionary of parameters.
subfolder (`str`, *optional*, defaults to `""`):
In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can
specify the folder name here.
Returns:
`Tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the video processor object.
rz Nr{ Fresume_downloadproxiesr} r r| r~ subfolder Z_from_pipelineZ
_from_autor r video processor) file_typefrom_auto_classZusing_pipelinez+Offline mode: forcing local_files_only=TrueT) rz r{ r r r| r}
user_agentr~ r zpreprocessor_config.jsonaA You have video processor config saved in `preprocessor.json` file which is deprecated. Video processor configs should be saved in their own `video_preprocessor.json` file. You can rename the file or load and save the processor back which renames it automatically. Loading from `preprocessor.json` will be removed in v5.0.z Can't load video processor for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z filerutf-8encodingz"It looks like the config file at 'z' is not a valid JSON file.zloading configuration file z from cache at Zauto_mapZcustom_pipelines)r7 r r r r r r< r strr r isdirr r r r r EnvironmentErrorZ warning_once ExceptionopenreadjsonloadsJSONDecodeErrorr r )r ry r. rz r{ r r r} r r| r~ r Z
from_pipeliner r is_localZ resolved_video_processor_fileZvideo_processor_filereadertextr rJ rJ rK r s
z+BaseVideoProcessor.get_video_processor_dictr c K s | }|dd}d|v rd|v r|d|d< d|v r(d|v r(|d|d< | di |}g }| D ]\}}t||rIt||| || q5|D ]}||d qLtd| |rc||fS |S )a
Instantiates a type of [`~video_processing_utils.VideoProcessorBase`] from a Python dictionary of parameters.
Args:
video_processor_dict (`Dict[str, Any]`):
Dictionary that will be used to instantiate the video processor object. Such a dictionary can be
retrieved from a pretrained checkpoint by leveraging the
[`~video_processing_utils.VideoProcessorBase.to_dict`] method.
kwargs (`Dict[str, Any]`):
Additional parameters from which to initialize the video processor object.
Returns:
[`~video_processing_utils.VideoProcessorBase`]: The video processor object instantiated from those
parameters.
return_unused_kwargsFr1 r3 NzVideo processor rJ )copyr7 r9 hasattrr: ra r< r )r r r. r Zvideo_processor to_removerE rF rJ rJ rK r s&
z BaseVideoProcessor.from_dictc C s t | j}| jj|d< |S )z
Serializes this instance to a Python dictionary.
Returns:
`Dict[str, Any]`: Dictionary of all the attributes that make up this video processor instance.
Zvideo_processor_type)r deepcopy__dict__rI __name__)rD outputrJ rJ rK to_dict s zBaseVideoProcessor.to_dictc C sb | }| D ]\}}t|tjr| ||< q|dd}|dur'||d< tj|dddd S )z
Serializes this instance to a JSON string.
Returns:
`str`: String containing all the attributes that make up this feature_extractor instance in JSON format.
r8 Nr0 T)indent sort_keys
) r r9 r[ r\ r] tolistr7 r dumps)rD
dictionaryrE rF r8 rJ rJ rK to_json_string s z!BaseVideoProcessor.to_json_stringjson_file_pathc C sB t |ddd}|| W d dS 1 sw Y dS )z
Save this instance to a JSON file.
Args:
json_file_path (`str` or `os.PathLike`):
Path to the JSON file in which this image_processor instance's parameters will be saved.
wr r N)r writer )rD r writerrJ rJ rK r s "z BaseVideoProcessor.to_json_filec C s | j j d| S )N )rI r r rD rJ rJ rK __repr__ s zBaseVideoProcessor.__repr__ json_filec C sN t |ddd }| }W d n1 sw Y t|}| di |S )a
Instantiates a video processor of type [`~video_processing_utils.VideoProcessorBase`] from the path to a JSON
file of parameters.
Args:
json_file (`str` or `os.PathLike`):
Path to the JSON file containing the parameters.
Returns:
A video processor of type [`~video_processing_utils.VideoProcessorBase`]: The video_processor object
instantiated from that JSON file.
r r r NrJ )r r r r )r r r r r rJ rJ rK from_json_file s
z!BaseVideoProcessor.from_json_fileAutoVideoProcessorc C sD t |ts|j}ddlm m} t||s t| d|| _dS )a
Register this class with a given auto class. This should only be used for custom video processors as the ones
in the library are already mapped with `AutoVideoProcessor `.
This API is experimental and may have some slight breaking changes in the next releases.
Args:
auto_class (`str` or `type`, *optional*, defaults to `"AutoVideoProcessor "`):
The auto class to register this new video processor with.
r Nz is not a valid auto class.) r[ r r Ztransformers.models.automodelsautor r r )r Z
auto_classZ auto_modulerJ rJ rK register_for_auto_class s
z*BaseVideoProcessor.register_for_auto_classvideo_url_or_urlsc s@ t |tr fdd|D S t |trt|S tdt| )z
Convert a single or a list of urls into the corresponding `np.array` objects.
If a single url is passed, the return value will be a single object. If a list is passed a list of objects is
returned.
c s g | ]} |qS rJ ) fetch_videos).0xr rJ rK
s z3BaseVideoProcessor.fetch_videos..z=only a single or a list of entries is supported but got type=)r[ r> r r$ TypeErrortype)rD r rJ r rK r
s
z BaseVideoProcessor.fetch_videos)NNrL )NFFNrx )F)r ) sT D
;