o Zhm@sddlZddlmZddlmZddlmZddlmZm Z m Z m Z m Z m Z mZddlmZddlZddlZddlmZmZdd lmZmZmZdd lmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&e#rvddl'Z(ddl)Z(e"rvdd l*mZ+e r}ddl,Z,e%-e.Z/ee d d de d e de e d e e de e dfZ0eGdddZ1ddZ2ddZ3ddZ4ddZ5dej6de7fddZ8de e0de edfd d!Z9de edfd"d#Z:dOdej6d$ede e;e;ffd%d&Zd.e?d/efd0d1Z@ dOd.e?d/e efd2d3ZAd.e?d/efd4d5ZBd.e?d/efd6d7ZCeAe@eBeCd8ZD   9 dQdee?d:fd(e e;d;e e;dZF  dPdejEd?e ed@e ee?efdejEfdAdBZGejHdCddfdej6dDee;e e;e;fe e e;e;ffdEedFeeIe eIfd?e ee?efd@e ee?efdej6fdGdHZJde dde e e e;e;fe dfe e;e e e;e;fe;ffffdIdJZKdKe e e;e;fdfdLe e;e e;e;ffde dfdMdNZLdS)RN)redirect_stdout) dataclass)BytesIO)CallableDictIterableListOptionalTupleUnion)urlparse) PaddingModeto_channel_dimension_format)ChannelDimensioninfer_channel_dimension_formatis_valid_image) is_av_availableis_cv2_availableis_decord_availableis_numpy_arrayis_torch_availableis_torch_tensoris_torchvision_availableis_vision_availableis_yt_dlp_availableloggingrequires_backends)iozPIL.Image.Image np.ndarray torch.Tensorz np.ndarrrayc@s.eZdZUeed<eed<eed<eed<dS) VideoMetadatatotal_num_framesfpsduration video_backendN)__name__ __module__ __qualname__int__annotations__floatstrr-r-G/var/www/auris/lib/python3.10/site-packages/transformers/video_utils.pyr!Es  r!cCs(t|tjjpt|st|o|jdkS)N) isinstancePILZImagerrndim)framer-r-r.is_valid_video_frameMsr4cCs:t|ttfst|st|o|jdkStdd|DS)Ncss|]}t|VqdSN)r4.0r3r-r-r. Vsz!is_valid_video..)r0listtuplerrr2allvideor-r-r.is_valid_videoSsr?cCsLt|ttfr|D] }t|st|sdSq dSt|r"|jdkr$dSdS)NFT)r0r:r;r?r4r2)videosZvideo_or_framer-r-r. valid_videosYsrBcCs<t|ttfr t|dSt|st|r|jdkrdSdS)Nrr@TF)r0r:r;r?rrr2rAr-r-r.is_batched_videoes  rDr>returncCst|dko t|dkS)zV Checks to see whether the pixel values have already been rescaled to [0, 1]. rr )npminmaxr=r-r-r.is_scaled_videomsrIrA)rr cCsJt|dttfs |Sg}|D]}dd|D}t|}||q|S)aK Given a batch of videos, converts each video to a 4D array. If video is already in array type, it is simply returned. We assume that all inputs in the list are in the same format, based on the type of the first element. Args: videos (`VideoInput`): Video inputs to turn into a list of videos. rcSsg|]}t|qSr-)rFarrayr7r-r-r. sz/convert_pil_frames_to_video..)r0r:r;rFstackappend)rAZvideo_convertedr>r-r-r.convert_pil_frames_to_videous   rNcCsts tdt|dt|r t |St|r |g}t |St|r0t|dg}t |St|dt t frHt|ddrHdd|DSt |S)a Ensure that the input is a list of videos. If the input is a single video, it is converted to a list of length 1. If the input is a batch of videos, it is converted to a list of 4D video arrays. Videos passed as list `PIL.Image` frames are converted to 4D arrays. We assume that all inputs in the list are in the same format, based on the type of the first element. Args: videos (`VideoInput`): Video inputs to turn into a list of videos. zkInvalid video input. Expected either a list of video frames or an input of 4 or 5 dimensions, but got type .)N.rcSsg|] }|D]}|qqSr-r-)r8Zsublistr>r-r-r.rKsz'make_batched_videos..) rB ValueErrortyperDr?rrFrJr0r:r;rNrCr-r-r.make_batched_videoss$  "rR channel_dimcCsZ|durt|}|tjkr|jd|jdfS|tjkr&|jd|jdfStd|)a Returns the (height, width) dimensions of the video. Args: video (`np.ndarray`): The video to get the dimensions of. channel_dim (`ChannelDimension`, *optional*): Which dimension the channel dimension is in. If `None`, will infer the channel dimension from the video. Returns: A tuple of the video's height and width. NzUnsupported data format: )rrFIRSTshapeZLASTrP)r>rSr-r-r.get_video_sizes   rYr" num_framescCs:|durtd|||t}|Std|t}|S)a Creates a numpy array for uniform sampling of `num_frame` frames from `total_num_frames` when loading a video. Args: total_num_frames (`int`): Total number of frames that a video has. num_frames (`int`, *optional*): Number of frames to sample uniformly. If not specified, all frames are sampled. Returns: np.ndarray: np array of frame indices that will be sampled. Nr)rFarangeZastyper))r"rZindicesr-r-r.get_uniform_frame_indicess r]metadatacKs|j}|j}|dur(|dur(t|||}||kr(td|d|d|d|dur9tjd|||td}|Stjd|td}|S)a` A default sampling function that replicates the logic used in get_uniform_frame_indices, while optionally handling `fps` if `num_frames` is not provided. Args: metadata (`VideoMetadata`): `VideoMetadata` object containing metadata about the video, such as "total_num_frames" or "fps". num_frames (`int`, *optional*): Number of frames to sample uniformly. fps (`int`, *optional*): Desired frames per second. Takes priority over num_frames if both are provided. Returns: `np.ndarray`: Array of frame indices to sample. Nz When loading the video with fps=z, we computed num_frames=z which exceeds total_num_frames=z. Check fps or video metadata.r)Zdtype)r"r#r)rPrFr[)r^rZr#kwargsr" video_fpsr\r-r-r.default_sample_indices_fnsra video_pathsample_indices_fncKsttdgddl}||}t||j}||j}|r#||nd}tt|t |t |dd}|dd|i|} d} g} | r}| \} } | sMn0| | vrn| j \}}}| | |j} | | d|d|d|f| rt| d7} | |kryn| sD|| |_t| |fS) av Decode a video using the OpenCV backend. Args: video_path (`str`): Path to the video file. sample_indices_fn (`Callable`): A callable function that will return indices at which the video should be sampled. If the video has to be loaded using by a different sampling technique than provided by `num_frames` or `fps` arguments, one should provide their own `sample_indices_fn`. If not provided, simple uniform sampling with fps is performed. Example: def sample_indices_fn(metadata, **kwargs): return np.linspace(0, metadata.total_num_frames - 1, num_frames, dtype=int) Returns: Tuple[`np.array`, `VideoMetadata`]: A tuple containing: - Numpy array of frames in RGB (shape: [num_frames, height, width, 3]). - `VideoMetadata` object. cv2rNopencvr"r#r$r%r^r r-)rread_video_opencvrdZ VideoCapturer)getZCAP_PROP_FRAME_COUNTZ CAP_PROP_FPSr!r+ZisOpenedreadrXZcvtColorZ COLOR_BGR2RGBrMreleaseframes_indicesrFrL)rbrcr_rdr>r"r`r$r^r\indexframessuccessr3heightwidthZchannelr-r-r.rgs8       rgc Ksttdgddlm}m}|||dd}|}t|}|r$||nd}tt|t |t |dd} |dd| i|} | |  } | | _ | | fS) a Decode a video using the Decord backend. Args: video_path (`str`): Path to the video file. sample_indices_fn (`Callable`, *optional*): A callable function that will return indices at which the video should be sampled. If the video has to be loaded using by a different sampling technique than provided by `num_frames` or `fps` arguments, one should provide their own `sample_indices_fn`. If not provided, simple uniform sampling with fps is performed. Example: def sample_indices_fn(metadata, **kwargs): return np.linspace(0, metadata.total_num_frames - 1, num_frames, dtype=int) Returns: Tuple[`np.array`, `VideoMetadata`]: A tuple containing: - Numpy array of frames in RGB (shape: [num_frames, height, width, 3]). - `VideoMetadata` object. decordr) VideoReadercpu)urictxrfr^Nr-) rread_video_decordrqrrrsZ get_avg_fpslenr!r)r+Z get_batchZasnumpyrk) rbrcr_rrrsZvrr`r"r$r^r\rmr-r-r.rv2s rvcKsttdgddl}||}|jjdj}|jjdj}|r#||nd}tt |t |t |dd}|d d|i|} g} | d| d} t |j ddD]\} } | | krYn| dkrf| | vrf| | qOtd d | D}| |_||fS) a} Decode the video with PyAV decoder. Args: video_path (`str`): Path to the video file. sample_indices_fn (`Callable`, *optional*): A callable function that will return indices at which the video should be sampled. If the video has to be loaded using by a different sampling technique than provided by `num_frames` or `fps` arguments, one should provide their own `sample_indices_fn`. If not provided, simple uniform sampling with fps is performed. Example: def sample_indices_fn(metadata, **kwargs): return np.linspace(0, metadata.total_num_frames - 1, num_frames, dtype=int) Returns: Tuple[`np.array`, `VideoMetadata`]: A tuple containing: - Numpy array of frames in RGB (shape: [num_frames, height, width, 3]). - `VideoMetadata` object. avrNpyavrfr^rUr=cSsg|]}|jddqS)Zrgb24)format)Z to_ndarray)r8xr-r-r.rKsz#read_video_pyav..r-)rread_video_pyavrxopenstreamsr>rmZ average_rater!r)r+seek enumeratedecoderMrFrLrk)rbrcr_rx containerr"r`r$r^r\rmZ end_indexir3r>r-r-r.r|]s,    r|c Kstj|ddddd\}}}|d}|d}|r||nd}tt|t|t|dd } |d d | i|} || }| | _|| fS) a Decode the video with torchvision decoder. Args: video_path (`str`): Path to the video file. sample_indices_fn (`Callable`, *optional*): A callable function that will return indices at which the video should be sampled. If the video has to be loaded using by a different sampling technique than provided by `num_frames` or `fps` arguments, one should provide their own `sample_indices_fn`. If not provided, simple uniform sampling with fps is performed. Example: def sample_indices_fn(metadata, **kwargs): return np.linspace(0, metadata.total_num_frames - 1, num_frames, dtype=int) Returns: Tuple[`np.array`, `VideoMetadata`]: A tuple containing: - Numpy array of frames in RGB (shape: [num_frames, height, width, 3]). - `VideoMetadata` object. NsecZTHWC)Z start_ptsZend_ptsZpts_unitZ output_formatr`r torchvisionrfr^r-) torchvision_ioZ read_videosizer!r)r+ contiguousnumpyrk) rbrcr_r>_infor`r"r$r^r\r-r-r.read_video_torchvisions(  r)rqreryrry VideoInputr#backendc sdurdur|durtd|durfdd}|}t|jdvrpts+tdttdgdd lm}t }t |!|} | |gWdn1sSwYWdn1sbwY| } t | } n4| d sz| d rt t|j} n!tj|r|} nt|st|ttfrt|drd} ntd | d p| d } | r|d vrtd| dur|Sts|dksts|dksts|dksts|dkrtd|d|dt|} | | |fi|\}}||fS)a Loads `video` to a numpy array. Args: video (`str` or `VideoInput`): The video to convert to the numpy array format. Can be a link to video or local path. num_frames (`int`, *optional*): Number of frames to sample uniformly. If not passed, the whole video is loaded. fps (`int`, *optional*): Number of frames to sample per second. Should be passed only when `num_frames=None`. If not specified and `num_frames==None`, all frames are sampled. backend (`str`, *optional*, defaults to `"pyav"`): The backend to use when loading the video. Can be any of ["decord", "pyav", "opencv", "torchvision"]. Defaults to "pyav". sample_indices_fn (`Callable`, *optional*): A callable function that will return indices at which the video should be sampled. If the video has to be loaded using by a different sampling technique than provided by `num_frames` or `fps` arguments, one should provide their own `sample_indices_fn`. If not provided, simple uniformt sampling with fps is performed, otherwise `sample_indices_fn` has priority over other args. The function expects at input the all args along with all kwargs passed to `load_video` and should output valid indices at which the video should be sampled. For example: Example: def sample_indices_fn(metadata, **kwargs): return np.linspace(0, metadata.total_num_frames - 1, num_frames, dtype=int) Returns: Tuple[`np.array`, Dict]: A tuple containing: - Numpy array of frames in RGB (shape: [num_frames, height, width, 3]). - Metadata dictionary. Nzc`num_frames`, `fps`, and `sample_indices_fn` are mutually exclusive arguments, please use only one!cst|fd|S)N)rZr#)ra)r^Z fn_kwargsr#rZr-r.sample_indices_fn_funcsz*load_video..sample_indices_fn_func)zwww.youtube.comz youtube.comzETo load a video from YouTube url you have to install `yt_dlp` first.yt_dlpr) YoutubeDLzhttp://zhttps://zVIncorrect format used for video. Should be an url linking to an video or a local path.)rerzlIf you are trying to load a video from URL, you can decode the video only with `pyav` or `decord` as backendrqryrerzYou chose backend=zf for loading the video but the required library is not found in your environment Make sure to install z before loading the video.)rPr netlocr ImportErrorr load_videorrrrdownloadgetvalue startswithrequestsrhcontentospathisfilerr0r:r; TypeErrorrrrrVIDEO_DECODERS)r>rZr#rrcr_rrbufferfZ bytes_objZfile_objZ video_is_urlZ video_decoderr^r-rr.rsl'    " r data_formatinput_data_formatcCst|tjstdt||durt|}t|tj|d}|j ddkr(|S|j ddkr5| ddS|ddddddfdk sG|S|ddddddfd }d|ddddddfd|ddddddf|ddddddf}|S) a Convert video to RGB by blending the transparency layer if it's in RGBA format, otherwise simply returns it. Args: video (`np.array`): The video to convert. data_format (`ChannelDimension`, *optional*): The channel dimension format of the output video. If unset, will use the inferred format from the input. input_data_format (`ChannelDimension`, *optional*): The channel dimension format of the input video. If unset, will use the inferred format from the input. zBVideo has to be a numpy array to convert to RGB format, but found N)Zinput_channel_dimrVr/r .go@) r0rFndarrayrPrQrrrrWrXrepeatany)r>rralphar-r-r.convert_to_rgb,s   Prrpaddingmodeconstant_valuesc sdurtfdd}tjdtjdtjdtjdi}||}i}||vr.td||tjkr9|||d <tj|fd ||i||durSt |SS) a Pads the `video` with the specified (height, width) `padding` and `mode`. Args: video (`np.ndarray`): The video to pad. padding (`int` or `Tuple[int, int]` or `Iterable[Tuple[int, int]]`): Padding to apply to the edges of the height, width axes. Can be one of three formats: - `((before_height, after_height), (before_width, after_width))` unique pad widths for each axis. - `((before, after),)` yields same before and after pad for height and width. - `(pad,)` or int is a shortcut for before = after = pad width for all axes. mode (`PaddingMode`): The padding mode to use. Can be one of: - `"constant"`: pads with a constant value. - `"reflect"`: pads with the reflection of the vector mirrored on the first and last values of the vector along each axis. - `"replicate"`: pads with the replication of the last value on the edge of the array along each axis. - `"symmetric"`: pads with the reflection of the vector mirrored along the edge of the array. constant_values (`float` or `Iterable[float]`, *optional*): The value to use for the padding if `mode` is `"constant"`. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output video. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: video in (num_frames, num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: video in (num_frames, height, width, num_channels) format. If unset, will use same as the input video. input_data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the input video. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: video in (num_frames, num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: video in (num_frames, height, width, num_channels) format. If unset, will use the inferred format of the input video. Returns: `np.ndarray`: The padded video. Ncst|ttfr||f||ff}nOt|tr,t|dkr,|d|df|d|dff}n3t|trCt|dkrCt|dtrC||f}nt|trXt|dkrXt|dtrX|}ntd|tjkrkddg|Rndg|dR}jdkrdg|R}|S|}|S)za Convert values to be in the format expected by np.pad based on the data format. r rzUnsupported format: )rrr@) r0r)r+r;rwrPrrWr2)valuesrr>r-r._expand_for_data_formats"$ $&z$pad.._expand_for_data_formatZconstantZreflectZ replicateZ symmetriczInvalid padding mode: rr) rrCONSTANTZREFLECTZ REPLICATEZ SYMMETRICrPrFpadr) r>rrrrrrZ padding_mapZ pad_kwargsr-rr.rVs&+  rcCsxi}i}t|D]&\}}|jdd}||vrg||<||||t||df||<qdd|D}||fS)a Groups videos by shape. Returns a dictionary with the shape as key and a list of videos with that shape as value, and a dictionary with the index of the video in the original list as key and the shape and index in the grouped list as value. rTNr cSs i|] \}}|tj|ddqS)r)dim)torchrL)r8rXrAr-r-r. s z)group_videos_by_shape..)rrXrMrwitems)rAZgrouped_videosgrouped_videos_indexrr>rXr-r-r.group_videos_by_shapesrprocessed_videosrcsfddttDS)z> Reconstructs a list of videos in the original order. cs(g|]}|d|dqS)rr r-)r8rrrr-r.rKsz"reorder_videos..)rangerw)rrr-rr.reorder_videoss  rr6)NN)NNryN)Mr contextlibr dataclassesrrrtypingrrrrr r r urllib.parser rrFrZimage_transformsrrZ image_utilsrrrutilsrrrrrrrrrrrZ PIL.Imager1Z PIL.ImageOpsrrrZ get_loggerr&loggerrr!r4r?rBrDrboolrIrNrRr)rYr]rar,rgrvr|rrrJrrrr+rrrr-r-r-r.s   $ 4       "# ; + 3 1   f -  Y6